├── .editorconfig ├── .github └── ISSUE_TEMPLATE.md ├── .gitignore ├── .travis.yml ├── LICENSE ├── README.md ├── README_ZH.md ├── account.go ├── bin ├── csctl │ ├── cmd.go │ └── cmd │ │ ├── backup.go │ │ ├── import.go │ │ ├── import_test.go │ │ ├── ls.go │ │ ├── node.go │ │ ├── restore.go │ │ └── upgrade.go ├── node │ └── server.go └── web │ └── server.go ├── build.sh ├── client.go ├── common.go ├── conf ├── conf.go └── files │ ├── base.json.sample │ ├── db.json.sample │ ├── etcd.json.sample │ ├── mail.json.sample │ ├── security.json.sample │ └── web.json.sample ├── csctl.go ├── db ├── mgo.go └── mid │ └── auto_inc.go ├── doc └── img │ ├── Cronsun_dashboard_en.png │ ├── Cronsun_job_list_en.png │ ├── Cronsun_job_new_en.png │ ├── Cronsun_log_item_en.png │ ├── Cronsun_log_list_en.png │ ├── Cronsun_node_en.png │ ├── brief.png │ ├── job.png │ ├── log.png │ ├── new_job.png │ └── node.png ├── errors.go ├── event ├── event.go └── event_test.go ├── go.mod ├── go.sum ├── group.go ├── id.go ├── job.go ├── job_log.go ├── log └── log.go ├── mdb.go ├── node.go ├── node ├── cron │ ├── LICENSE │ ├── README.md │ ├── at.go │ ├── at_test.go │ ├── constantdelay.go │ ├── constantdelay_test.go │ ├── cron.go │ ├── cron_test.go │ ├── doc.go │ ├── parser.go │ ├── parser_test.go │ ├── spec.go │ └── spec_test.go ├── csctl.go ├── group.go ├── job.go └── node.go ├── noticer.go ├── once.go ├── proc.go ├── release.sh ├── utils ├── argument_parser.go ├── argument_parser_test.go ├── confutil.go ├── confutil_test.go ├── local_ip.go ├── string.go ├── test.json └── test1.json ├── version.go └── web ├── administrator.go ├── authentication.go ├── base.go ├── configuration.go ├── gen_bindata.sh ├── info.go ├── job.go ├── job_log.go ├── log_cleaner.go ├── node.go ├── routers.go ├── session └── session.go ├── static_assets.go └── ui ├── .babelrc ├── index.html ├── package.json ├── src ├── App.vue ├── components │ ├── Account.vue │ ├── AccountEdit.vue │ ├── Dash.vue │ ├── ExecuteJob.vue │ ├── Job.vue │ ├── JobEdit.vue │ ├── JobEditForm.vue │ ├── JobEditRule.vue │ ├── JobExecuting.vue │ ├── Log.vue │ ├── LogDetail.vue │ ├── Login.vue │ ├── Messager.vue │ ├── Node.vue │ ├── NodeGroup.vue │ ├── NodeGroupEdit.vue │ ├── Profile.vue │ └── basic │ │ ├── Dropdown.vue │ │ └── Pager.vue ├── i18n │ ├── language.js │ └── languages │ │ ├── en.js │ │ └── zh-CN.js ├── libraries │ ├── functions.js │ └── rest-client.js ├── main.js └── vuex │ └── store.js └── webpack.config.js /.editorconfig: -------------------------------------------------------------------------------- 1 | # for vscode extension[editorconfig] 2 | root = true 3 | 4 | [*.{js,json,html,vue}] 5 | indent_style = space 6 | indent_size = 2 -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE.md: -------------------------------------------------------------------------------- 1 | Please answer these questions before submitting your issue. Thanks! 2 | 在你提交 issue 前，请先回答以下问题，谢谢！ 3 | 4 | 1. What version of Go and cronsun version are you using? 5 | 你用的是哪个版本的 Go 和 哪个版本的 cronsun？ 6 | 7 | 2. What operating system and processor architecture are you using (`go env`)? 8 | 你用的是哪个操作系统，什么架构的？ 9 | 10 | 11 | 3. What did you do? 12 | If possible, provide a recipe for reproducing the error. 13 | A complete runnable program is good. 14 | 你做了什么，遇到了什么问题？尽可能描述清楚问题，最好把操作步骤写下来，按这些步骤操作后能重现你的问题。 15 | 16 | 17 | 4. What did you expect to see? 18 | 你期望得到什么样的结果？ 19 | 20 | 21 | 5. What did you see instead? 22 | 现在你得到的结果是什么样的？ 23 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | conf/files/*.json 2 | dist 3 | .tags 4 | .tags_sorted_by_file 5 | bin/*/*server 6 | .DS_Store 7 | web/ui/node_modules 8 | web/ui/package-lock.json 9 | web/ui/semantic.json 10 | web/ui/semantic 11 | web/ui/dist 12 | .vscode 13 | *npm-debug.log 14 | vendor 15 | -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | language: go 2 | go: 3 | - 1.11.x 4 | env: 5 | - GO111MODULE=on 6 | install: 7 | - go mod vendor 8 | 9 | before_script: 10 | - go vet -v $(go list ./... | grep -v vendor) 11 | 12 | matrix: 13 | include: 14 | - go: "1.11.x" 15 | script: go test -v -race -mod=vendor ./... 16 | 17 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # cronsun [![Build Status](https://travis-ci.org/shunfei/cronsun.svg?branch=master)](https://travis-ci.org/shunfei/cronsun) 2 | 3 | `cronsun` is a distributed cron-style job system. It's similar with `crontab` on stand-alone `*nix`. 4 | 5 | [简体中文](README_ZH.md) 6 | 7 | ## Purpose 8 | 9 | The goal of this project is to make it much easier to manage jobs on lots of machines and provides high availability. 10 | `cronsun` is different from [Azkaban](https://azkaban.github.io/), [Chronos](https://mesos.github.io/chronos/), [Airflow](https://airflow.incubator.apache.org/). 11 | 12 | ## Features 13 | 14 | - Easy manage jobs on multiple machines 15 | - Management panel 16 | - Mail service 17 | - Multi-language support 18 | - Simple authentication and accounts manager(default administrator email and password: admin@admin.com/admin) 19 | 20 | ## Status 21 | 22 | `cronsun` has been tested in production for years on hundreds of servers. 23 | Although the current version is not release as an stable version, but we think it is completely available for the production environment. 24 | We encourage you to try it, it's easy to use, see how it works for you. We believe you will like this tool. 25 | 26 | 27 | ## Architecture 28 | 29 | ``` 30 | [web] 31 | | 32 | -------------------------- 33 | (add/del/update/exec jobs)| |(query job exec result) 34 | [etcd] [mongodb] 35 | | ^ 36 | -------------------- | 37 | | | | | 38 | [node.1] [node.2] [node.n] | 39 | (job exec fail)| | | | 40 | [send mail]<-----------------------------------------(job exec result) 41 | 42 | ``` 43 | 44 | 45 | ## Security 46 | 47 | `cronsun` support security with `security.json` config. When `open=true`， job command is only allow local files with special extension on the node. 48 | 49 | ```json 50 | { 51 | "open": true, 52 | "#users": "allowed execution users", 53 | "users": [ 54 | "www", "db" 55 | ], 56 | "#ext": "allowed execution file extensions", 57 | "ext": [ 58 | ".cron.sh", ".cron.py" 59 | ] 60 | } 61 | ``` 62 | 63 | ## Getting started 64 | 65 | ### Setup / installation 66 | 67 | Install from binary [latest release](https://github.com/shunfei/cronsun/releases/latest) 68 | 69 | Or build from source, require `go >= 1.11+`. 70 | > NOTE: The branch `master` is not in stable, using Cronsun for production please checkout corresponding tags. 71 | 72 | ``` 73 | export GO111MODULE=on 74 | go get -u github.com/shunfei/cronsun 75 | cd $GOPATH/src/github.com/shunfei/cronsun 76 | go mod vendor 77 | sh build.sh 78 | ``` 79 | 80 | ### Run 81 | 82 | 1. Install [MongoDB](http://docs.mongodb.org/manual/installation/) 83 | 2. Install [etcd3](https://github.com/coreos/etcd) 84 | 3. Open and update Etcd(`conf/etcd.json`) and MongoDB(`conf/db.json`) configurations 85 | 4. Start cronnode: `./cronnode -conf conf/base.json`, start cronweb: `./cronweb -conf conf/base.json` 86 | 5. Open `http://127.0.0.1:7079` in browser 87 | 6. Login with username `admin@admin.com` and password `admin` 88 | 89 | ## Screenshot 90 | 91 | **Brief**: 92 | 93 | ![](doc/img/Cronsun_dashboard_en.png) 94 | 95 | **Exec result**: 96 | 97 | ![](doc/img/Cronsun_log_list_en.png) 98 | ![](doc/img/Cronsun_log_item_en.png) 99 | 100 | **Job**: 101 | 102 | ![](doc/img/Cronsun_job_list_en.png) 103 | 104 | ![](doc/img/Cronsun_job_new_en.png) 105 | 106 | **Node**: 107 | 108 | ![](doc/img/Cronsun_node_en.png) 109 | 110 | ## Credits 111 | 112 | cron is base on [robfig/cron](https://github.com/robfig/cron) 113 | -------------------------------------------------------------------------------- /README_ZH.md: -------------------------------------------------------------------------------- 1 | # cronsun [![Build Status](https://travis-ci.org/shunfei/cronsun.svg?branch=master)](https://travis-ci.org/shunfei/cronsun) 2 | 3 | `cronsun` 是一个分布式任务系统，单个节点和 `*nix` 机器上的 `crontab` 近似。支持界面管理机器上的任务，支持任务失败邮件提醒，安装简单，使用方便，是替换 `crontab` 一个不错的选择。 4 | 5 | `cronsun` 是为了解决多台 `*nix` 机器上`crontab` 任务管理不方便的问题，同时提供任务高可用的支持（当某个节点死机的时候可以自动调度到正常的节点执行）。`cronsun` 和 [Azkaban](https://azkaban.github.io/)、[Chronos](https://mesos.github.io/chronos/)、[Airflow](https://airflow.incubator.apache.org/) 这些不是同一类型的。 6 | 7 | > QQ交流群： 123731057 8 | 9 | ## 项目状态 10 | 11 | `cronsun`已经在线上几百台规模的服务器上面稳定运行了一年多了，虽然目前版本不是正式版，但是我们认为是完全可以用于生产环境的。强烈建议你试用下，因为它非常简单易用，同时感受下他的强大，相信你会喜欢上这个工具的。 12 | 13 | 14 | ## 特性 15 | 16 | - 方便对多台服务器上面的定时任务进行集中式管理 17 | - 任务调度时间粒度支持到`秒`级别 18 | - 任务失败自动重试 19 | - 任务可靠性保障（从N个节点里面挑一个可用节点来执行任务） 20 | - 简洁易用的管理后台，支持多语言 21 | - 任务日志查看 22 | - 任务失败邮件告警（也支持自定义http告警接口） 23 | - 用户验证与授权 (默认账号密码: admin@admin.com / admin) 24 | - [可靠性说明](https://github.com/shunfei/cronsun/wiki/%E5%8F%AF%E9%9D%A0%E6%80%A7%E8%AF%B4%E6%98%8E) 25 | 26 | 27 | ## 架构 28 | 29 | ``` 30 | [web] 31 | | 32 | -------------------------- 33 | (add/del/update/exec jobs)| |(query job exec result) 34 | [etcd] [mongodb] 35 | | ^ 36 | -------------------- | 37 | | | | | 38 | [node.1] [node.2] [node.n] | 39 | (job exec fail)| | | | 40 | [send mail]<-----------------------------------------(job exec result) 41 | 42 | ``` 43 | 44 | 45 | ## 安全性 46 | 47 | `cronsun`是在管理后台添加任务的，所以一旦管理后台泄露出去了，则存在一定的危险性，所以`cronsun`支持`security.json`的安全设置： 48 | 49 | ```json 50 | { 51 | "open": true, 52 | "#users": "允许选择运行脚本的用户", 53 | "users": [ 54 | "www", "db" 55 | ], 56 | "#ext": "允许添加以下扩展名结束的脚本", 57 | "ext": [ 58 | ".cron.sh", ".cron.py" 59 | ] 60 | } 61 | ``` 62 | 63 | 如上设置开启安全限制，则添加和执行任务的时候只允许选择配置里面指定的用户来执行脚本，并且脚本的扩展名要在配置的脚本扩展名限制列表里面。 64 | 65 | 66 | ## 开始 67 | 68 | ### 安装 69 | 70 | 直接下载执行文件 [latest release](https://github.com/shunfei/cronsun/releases/latest)。 71 | 72 | 如果你熟悉 `Go`，也可以从源码编译, 要求 `go >= 1.11+` 73 | 74 | ``` 75 | go get -u github.com/shunfei/cronsun 76 | cd $GOPATH/src/github.com/shunfei/cronsun 77 | go mod vendor 78 | # 如果 go mod vendor 下载失败，请尝试 https://goproxy.io 79 | sh build.sh 80 | ``` 81 | 82 | ### 运行 83 | 84 | 1. 安装 [MongoDB](http://docs.mongodb.org/manual/installation/) 85 | 2. 安装 [etcd3](https://github.com/coreos/etcd) 86 | 3. 修改 `conf` 相关的配置 87 | 4. 在任务结点启动 `./cronnode -conf conf/base.json`，在管理结点启动 `./cronweb -conf conf/base.json` 88 | 5. 访问管理界面 `http://127.0.0.1:7079/ui/` 89 | 6. 使用用户名 `admin@admin.com` 和密码 `admin` 进行登录 90 | 91 | ### 关于后台权限 92 | 93 | 当前实现了一个可选的简单登录认证和帐号管理的功能（首次启用之后默认管理员的邮箱密码是 admin@admin.com/admin），没有详细的权限管理功能。登录控制也可以考虑使用 [aproxy](https://github.com/shunfei/aproxy) ，相关介绍见 [aProxy: 带认证授权和权限控制的反向代理](http://www.cnblogs.com/QLeelulu/p/aproxy.html)。 94 | 95 | ## 截图 96 | 97 | **概要**: 98 | 99 | ![](doc/img/brief.png) 100 | 101 | **执行日志**: 102 | 103 | ![](doc/img/log.png) 104 | 105 | **任务管理**: 106 | 107 | ![](doc/img/job.png) 108 | 109 | ![](doc/img/new_job.png) 110 | 111 | **结点状态**: 112 | 113 | ![](doc/img/node.png) 114 | 115 | ## 致谢 116 | 117 | cron is base on [robfig/cron](https://github.com/robfig/cron) 118 | -------------------------------------------------------------------------------- /account.go: -------------------------------------------------------------------------------- 1 | package cronsun 2 | 3 | import ( 4 | "time" 5 | 6 | "gopkg.in/mgo.v2" 7 | "gopkg.in/mgo.v2/bson" 8 | ) 9 | 10 | const ( 11 | Coll_Account = "account" 12 | ) 13 | 14 | type Account struct { 15 | ID bson.ObjectId `bson:"_id" json:"id"` 16 | Role Role `bson:"role" json:"role"` 17 | Email string `bson:"email" json:"email"` 18 | Password string `bson:"password" json:"password"` 19 | Salt string `bson:"salt" json:"salt"` 20 | Status UserStatus `bson:"status" json:"status"` 21 | Session string `bson:"session" json:"-"` 22 | // If true, role and status are unchangeable, email and password can be change by it self only. 23 | Unchangeable bool `bson:"unchangeable" json:"-"` 24 | CreateTime time.Time `bson:"createTime" json:"createTime"` 25 | } 26 | 27 | type Role int 28 | 29 | const ( 30 | Administrator Role = 1 31 | Developer Role = 2 32 | Reporter Role = 3 33 | ) 34 | 35 | func (r Role) Defined() bool { 36 | switch r { 37 | case Administrator, Developer, Reporter: 38 | return true 39 | } 40 | return false 41 | } 42 | 43 | func (r Role) String() string { 44 | switch r { 45 | case Administrator: 46 | return "Administrator" 47 | case Developer: 48 | return "Developer" 49 | case Reporter: 50 | return "Reporter" 51 | } 52 | return "Undefined" 53 | } 54 | 55 | type UserStatus int 56 | 57 | const ( 58 | UserBanned UserStatus = -1 59 | UserActived UserStatus = 1 60 | ) 61 | 62 | func (s UserStatus) Defined() bool { 63 | switch s { 64 | case UserBanned, UserActived: 65 | return true 66 | } 67 | return false 68 | } 69 | 70 | func GetAccounts(query bson.M) (list []Account, err error) { 71 | err = mgoDB.WithC(Coll_Account, func(c *mgo.Collection) error { 72 | return c.Find(query).All(&list) 73 | }) 74 | return 75 | } 76 | 77 | func GetAccountByEmail(email string) (u *Account, err error) { 78 | err = mgoDB.FindOne(Coll_Account, bson.M{"email": email}, &u) 79 | return 80 | } 81 | 82 | func CreateAccount(u *Account) error { 83 | u.ID = bson.NewObjectId() 84 | u.CreateTime = time.Now() 85 | return mgoDB.Insert(Coll_Account, u) 86 | 87 | } 88 | 89 | func UpdateAccount(query bson.M, change bson.M) error { 90 | return mgoDB.WithC(Coll_Account, func(c *mgo.Collection) error { 91 | return c.Update(query, bson.M{"$set": change}) 92 | }) 93 | } 94 | 95 | func BanAccount(email string) error { 96 | return mgoDB.WithC(Coll_Account, func(c *mgo.Collection) error { 97 | return c.Update(bson.M{"email": email}, bson.M{"$set": bson.M{"status": UserBanned}}) 98 | }) 99 | } 100 | 101 | func EnsureAccountIndex() error { 102 | return mgoDB.WithC(Coll_Account, func(c *mgo.Collection) error { 103 | return c.EnsureIndex(mgo.Index{ 104 | Key: []string{"email"}, 105 | Unique: true, 106 | }) 107 | }) 108 | } 109 | -------------------------------------------------------------------------------- /bin/csctl/cmd.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "fmt" 5 | "os" 6 | 7 | "github.com/spf13/cobra" 8 | 9 | "github.com/shunfei/cronsun" 10 | subcmd "github.com/shunfei/cronsun/bin/csctl/cmd" 11 | ) 12 | 13 | var confFile string 14 | 15 | var rootCmd = &cobra.Command{ 16 | Use: "csctl", 17 | Short: "cronsun command tools for data manage", 18 | PersistentPreRun: func(cmd *cobra.Command, args []string) { 19 | if err := cronsun.Init(confFile, false); err != nil { 20 | fmt.Println(err) 21 | os.Exit(1) 22 | } 23 | }, 24 | Run: func(cmd *cobra.Command, args []string) {}, 25 | } 26 | 27 | func init() { 28 | rootCmd.PersistentFlags().StringVarP(&confFile, "conf", "c", "conf/files/base.json", "base.json file path.") 29 | rootCmd.AddCommand(subcmd.BackupCmd, subcmd.RestoreCmd, subcmd.UpgradeCmd, subcmd.NodeCmd, subcmd.ImportCmd, subcmd.LsCmd) 30 | } 31 | 32 | func main() { 33 | if err := rootCmd.Execute(); err != nil { 34 | fmt.Println(err) 35 | os.Exit(1) 36 | } 37 | } 38 | -------------------------------------------------------------------------------- /bin/csctl/cmd/backup.go: -------------------------------------------------------------------------------- 1 | package cmd 2 | 3 | import ( 4 | "archive/zip" 5 | "encoding/binary" 6 | "fmt" 7 | "io" 8 | "os" 9 | "path" 10 | "runtime" 11 | "strings" 12 | "time" 13 | 14 | "github.com/coreos/etcd/clientv3" 15 | "github.com/spf13/cobra" 16 | 17 | "github.com/shunfei/cronsun" 18 | "github.com/shunfei/cronsun/conf" 19 | ) 20 | 21 | var ( 22 | backupDir string 23 | backupFile string 24 | ) 25 | 26 | var BackupCmd = &cobra.Command{ 27 | Use: "backup", 28 | Short: "backup job & group data", 29 | Run: func(cmd *cobra.Command, args []string) { 30 | var err error 31 | var ea = NewExitAction() 32 | 33 | backupDir = strings.TrimSpace(backupDir) 34 | if len(backupDir) > 0 { 35 | err = os.MkdirAll(backupDir, 0755) 36 | if err != nil { 37 | ea.Exit("failed to make directory %s, err: %s", backupDir, err) 38 | } 39 | } 40 | 41 | backupFile = strings.TrimSpace(backupFile) 42 | if len(backupFile) == 0 { 43 | backupFile = time.Now().Format("20060102_150405") 44 | } 45 | backupFile += ".zip" 46 | 47 | name := path.Join(backupDir, backupFile) 48 | f, err := os.OpenFile(name, os.O_TRUNC|os.O_CREATE|os.O_WRONLY, 0600) 49 | ea.ExitOnErr(err) 50 | ea.Defer = func() { 51 | f.Close() 52 | if err != nil { 53 | os.Remove(name) 54 | } 55 | } 56 | 57 | var waitForStore = [][]string{ 58 | // [file name in ZIP archive, key prefix in etcd] 59 | []string{"job", conf.Config.Cmd}, 60 | []string{"node_group", conf.Config.Group}, 61 | } 62 | zw := zip.NewWriter(f) 63 | 64 | for i := range waitForStore { 65 | zf, err := zw.Create(waitForStore[i][0]) 66 | ea.ExitOnErr(err) 67 | storeKvs(zf, waitForStore[i][1]) 68 | } 69 | 70 | ea.ExitOnErr(zw.Close()) 71 | }, 72 | } 73 | 74 | func init() { 75 | BackupCmd.Flags().StringVarP(&backupDir, "dir", "d", "", "the directory to store backup file") 76 | BackupCmd.Flags().StringVarP(&backupFile, "file", "f", "", "the backup file name") 77 | } 78 | 79 | type ExitAction struct { 80 | Defer func() 81 | After func() 82 | } 83 | 84 | func NewExitAction() *ExitAction { 85 | return &ExitAction{} 86 | } 87 | 88 | func (ea *ExitAction) ExitOnErr(err error) { 89 | if err != nil { 90 | _, f, l, _ := runtime.Caller(1) 91 | ea.Exit("%s line %d: %s", f, l, err.Error()) 92 | } 93 | } 94 | 95 | func (ea *ExitAction) Exit(format string, v ...interface{}) { 96 | if ea.Defer != nil { 97 | ea.Defer() 98 | } 99 | 100 | fmt.Printf(format+"\n", v...) 101 | 102 | if ea.After != nil { 103 | ea.After() 104 | } 105 | os.Exit(1) 106 | } 107 | 108 | var ( 109 | sizeBuf = make([]byte, 2+4) // key length(uint16) + value length(uint32) 110 | ) 111 | 112 | func storeKvs(w io.Writer, keyPrefix string) error { 113 | gresp, err := cronsun.DefalutClient.Get(keyPrefix, clientv3.WithPrefix()) 114 | if err != nil { 115 | return fmt.Errorf("failed to fetch %s from etcd: %s", keyPrefix, err) 116 | } 117 | 118 | var prefixLen = len(keyPrefix) 119 | 120 | for i := range gresp.Kvs { 121 | key := gresp.Kvs[i].Key[prefixLen:] 122 | binary.LittleEndian.PutUint16(sizeBuf[:2], uint16(len(key))) 123 | binary.LittleEndian.PutUint32(sizeBuf[2:], uint32(len(gresp.Kvs[i].Value))) 124 | 125 | // length of key 126 | if _, err = w.Write(sizeBuf[:2]); err != nil { 127 | return err 128 | } 129 | if _, err = w.Write(key); err != nil { 130 | return err 131 | } 132 | 133 | // lenght of value 134 | if _, err = w.Write(sizeBuf[2:]); err != nil { 135 | return err 136 | } 137 | if _, err = w.Write(gresp.Kvs[i].Value); err != nil { 138 | return err 139 | } 140 | } 141 | 142 | return nil 143 | } 144 | -------------------------------------------------------------------------------- /bin/csctl/cmd/import.go: -------------------------------------------------------------------------------- 1 | package cmd 2 | 3 | import ( 4 | "bytes" 5 | "encoding/json" 6 | "fmt" 7 | "math/rand" 8 | "os/exec" 9 | "strings" 10 | "time" 11 | 12 | "github.com/shunfei/cronsun" 13 | cron2 "github.com/shunfei/cronsun/node/cron" 14 | "github.com/spf13/cobra" 15 | ) 16 | 17 | type cron struct { 18 | timer string 19 | cmd string 20 | } 21 | 22 | var ( 23 | importNodes string 24 | ) 25 | 26 | func init() { 27 | ImportCmd.Flags().StringVar(&importNodes, "nodes", "", `the node ids that needs to run these imported job, 28 | split by ',', e.g: '--nodes=aa,bb,cc', empty means no node will run`) 29 | } 30 | 31 | var ImportCmd = &cobra.Command{ 32 | Use: "import", 33 | Short: `it will load the job from the crontab, but you must to confirm you can execute 'crontab -l'`, 34 | Run: func(cmd *cobra.Command, args []string) { 35 | var nodeInclude []string 36 | if len(importNodes) > 0 { 37 | nodeInclude = strings.Split(importNodes, spliter) 38 | } 39 | 40 | ea := NewExitAction() 41 | crons, err := loadCrons() 42 | if err != nil { 43 | ea.Exit("load crontab failed,err:%s", err.Error()) 44 | } 45 | total := len(crons) 46 | var successCount int 47 | ea.After = func() { 48 | fmt.Printf("total:%d,success:%d,failed:%d\n", total, successCount, total-successCount) 49 | if err := cmd.Help(); err != nil { 50 | return 51 | } 52 | } 53 | rand.Seed(time.Now().Unix()) 54 | for _, cron := range crons { 55 | job := cronsun.Job{} 56 | job.ID = cronsun.NextID() 57 | job.Command = cron.cmd 58 | jr := &cronsun.JobRule{ 59 | Timer: "* " + cron.timer, 60 | } 61 | jr.NodeIDs = nodeInclude 62 | job.Name = fmt.Sprintf("crontab-%d", rand.Intn(1000)) 63 | job.Group = "crontab" 64 | job.Rules = append(job.Rules, jr) 65 | // 默认先暂停 66 | job.Pause = true 67 | if err := job.Check(); err != nil { 68 | ea.Exit("job check error:%s", err.Error()) 69 | } 70 | b, err := json.Marshal(job) 71 | if err != nil { 72 | ea.Exit("json marshal error:%s", err.Error()) 73 | } 74 | 75 | _, err = cronsun.DefalutClient.Put(job.Key(), string(b)) 76 | if err != nil { 77 | ea.Exit("etcd put error:%s", err.Error()) 78 | } 79 | successCount++ 80 | fmt.Printf("crontab-%s %s has import to the cronsun, the job id is:%s\n", cron.timer, cron.cmd, job.ID) 81 | } 82 | 83 | fmt.Printf("import fininsh,succes:%d\n", successCount) 84 | }, 85 | } 86 | 87 | func checkCrons(crons []string) (invalid []string) { 88 | for _, item := range crons { 89 | item = strings.TrimSpace(item) 90 | if item != "" && !strings.HasPrefix(item, "#") { 91 | expr := strings.Fields(item) 92 | expr = expr[:5] 93 | _, err := cron2.ParseStandard(strings.Join(expr, " ")) 94 | if err != nil { 95 | invalid = append(invalid, item) 96 | } 97 | } 98 | } 99 | return 100 | } 101 | 102 | func loadCrons() (crons []cron, err error) { 103 | var b bytes.Buffer 104 | cmd := exec.Command("crontab", "-l") 105 | cmd.Stdout = &b 106 | cmd.Stderr = &b 107 | err = cmd.Run() 108 | if err != nil { 109 | return 110 | } 111 | 112 | result := strings.Split(b.String(), "\n") 113 | invalid := checkCrons(result) 114 | if len(invalid) > 0 { 115 | title := fmt.Sprintf("There are %d invalid cron expression,please check them at first.\n", len(invalid)) 116 | err = fmt.Errorf(title + strings.Join(invalid, "\n")) 117 | return 118 | } 119 | 120 | for _, item := range result { 121 | item = strings.TrimSpace(item) 122 | if item != "" && !strings.HasPrefix(item, "#") { 123 | spec := strings.Split(item, " ") 124 | timer := strings.Join(spec[:5], " ") 125 | cmd := strings.Join(spec[5:], " ") 126 | crons = append(crons, cron{timer, cmd}) 127 | } 128 | } 129 | return 130 | } 131 | -------------------------------------------------------------------------------- /bin/csctl/cmd/import_test.go: -------------------------------------------------------------------------------- 1 | package cmd 2 | 3 | import ( 4 | "strings" 5 | "testing" 6 | ) 7 | 8 | func TestCheckCrons(t *testing.T) { 9 | crontab := ` 10 | */1 * * * * /usr/bine/echo hello 11 | * * * * * /usr/bin/ls 12 | * & * * * /usr/bin/php -v 13 | * * * * /usr/bin/go run main.go 14 | ` 15 | invalidCrons := checkCrons(strings.Split(crontab, "\n")) 16 | if len(invalidCrons) != 2 { 17 | t.Error("should have 2 cron expression,but get none.") 18 | } 19 | if invalidCrons[0] != "* & * * * /usr/bin/php -v" || 20 | invalidCrons[1] != "* * * * /usr/bin/go run main.go" { 21 | t.Error("invalid cron expression should * & * * * /usr/bin/php -v and * * * * /usr/bin/go run main.go.") 22 | } 23 | } 24 | -------------------------------------------------------------------------------- /bin/csctl/cmd/ls.go: -------------------------------------------------------------------------------- 1 | package cmd 2 | 3 | import ( 4 | "fmt" 5 | 6 | "github.com/shunfei/cronsun" 7 | "github.com/spf13/cobra" 8 | ) 9 | 10 | var all bool 11 | 12 | func init() { 13 | LsCmd.Flags().BoolVarP(&all, "all", "a", false, "list all nodes include not alive") 14 | } 15 | 16 | var LsCmd = &cobra.Command{ 17 | Use: "ls", 18 | Short: "list the nodes", 19 | Run: func(cmd *cobra.Command, args []string) { 20 | ea := NewExitAction() 21 | ea.After = func() { 22 | fmt.Println() 23 | cmd.Help() 24 | } 25 | 26 | nodes, err := cronsun.GetNodes() 27 | if err != nil { 28 | ea.Exit(err.Error()) 29 | } 30 | 31 | fmt.Print("ID") 32 | for i := 0; i < 5; i++ { 33 | fmt.Print("\t") 34 | } 35 | fmt.Print("ip\t\t\t") 36 | fmt.Print("pid\t\t") 37 | fmt.Print("hostname\t") 38 | fmt.Print("alived\t") 39 | fmt.Println() 40 | 41 | for _, item := range nodes { 42 | if !all && !item.Alived { 43 | continue 44 | } 45 | fmt.Print(item.ID + "\t") 46 | fmt.Print(item.IP + "\t\t") 47 | fmt.Print(item.PID + "\t\t") 48 | fmt.Print(item.Hostname + "\t\t") 49 | if item.Alived { 50 | fmt.Print("Yes" + "\t\t") 51 | } else { 52 | fmt.Print("No" + "\t\t") 53 | } 54 | 55 | } 56 | }, 57 | } 58 | -------------------------------------------------------------------------------- /bin/csctl/cmd/node.go: -------------------------------------------------------------------------------- 1 | package cmd 2 | 3 | import ( 4 | "fmt" 5 | "strings" 6 | 7 | "github.com/spf13/cobra" 8 | 9 | "github.com/shunfei/cronsun" 10 | ) 11 | 12 | var ( 13 | nodeCmd string 14 | nodeInclude string 15 | nodeExclude string 16 | 17 | spliter = "," 18 | ) 19 | 20 | func init() { 21 | NodeCmd.Flags().StringVar(&nodeCmd, "cmd", "", "the command send to node") 22 | NodeCmd.Flags().StringVar(&nodeInclude, "include", "", "the node ids that needs to execute the command, split by ',', e.g: '--include=aa,bb,cc', empty means all nodes") 23 | NodeCmd.Flags().StringVar(&nodeExclude, "exclude", "", "the node ids that doesn't need to execute the command, split by ',', e.g: '--exclude=aa,bb,cc', empty means none") 24 | } 25 | 26 | var NodeCmd = &cobra.Command{ 27 | Use: "node", 28 | Short: "Send some commands to nodes", 29 | Long: `Send a command to nodes and execute it. 30 | 31 | Available Commands: 32 | rmold: remove old version(< 0.3.0) node info from mongodb and etcd 33 | sync: sync node info to mongodb 34 | `, 35 | Run: func(cmd *cobra.Command, args []string) { 36 | ea := NewExitAction() 37 | ea.After = func() { 38 | fmt.Println() 39 | cmd.Help() 40 | } 41 | nc, err := cronsun.ToNodeCmd(nodeCmd) 42 | if err != nil { 43 | ea.Exit(err.Error() + ": " + nodeCmd) 44 | } 45 | 46 | var include, exclude []string 47 | if len(nodeInclude) > 0 { 48 | include = strings.Split(nodeInclude, spliter) 49 | } 50 | if len(nodeExclude) > 0 { 51 | exclude = strings.Split(nodeExclude, spliter) 52 | } 53 | 54 | err = cronsun.PutCsctl(&cronsun.CsctlCmd{ 55 | Cmd: nc, 56 | Include: include, 57 | Exclude: exclude, 58 | }) 59 | if err != nil { 60 | ea.ExitOnErr(err) 61 | } 62 | 63 | fmt.Printf("command[%s] send success\n", nodeCmd) 64 | }, 65 | } 66 | -------------------------------------------------------------------------------- /bin/csctl/cmd/restore.go: -------------------------------------------------------------------------------- 1 | package cmd 2 | 3 | import ( 4 | "archive/zip" 5 | "encoding/binary" 6 | "fmt" 7 | "io" 8 | "strings" 9 | "sync" 10 | 11 | "github.com/spf13/cobra" 12 | 13 | "github.com/shunfei/cronsun" 14 | "github.com/shunfei/cronsun/conf" 15 | ) 16 | 17 | var restoreFile string 18 | 19 | func init() { 20 | RestoreCmd.Flags().StringVarP(&restoreFile, "file", "f", "", "the backup zip file") 21 | } 22 | 23 | var RestoreCmd = &cobra.Command{ 24 | Use: "restore", 25 | Short: "restore job & group data", 26 | Run: func(cmd *cobra.Command, args []string) { 27 | var err error 28 | var ea = NewExitAction() 29 | 30 | restoreFile = strings.TrimSpace(restoreFile) 31 | if len(restoreFile) == 0 { 32 | ea.Exit("backup file is required") 33 | } 34 | 35 | r, err := zip.OpenReader(restoreFile) 36 | ea.ExitOnErr(err) 37 | ea.Defer = func() { 38 | r.Close() 39 | } 40 | 41 | restoreChan, wg := startRestoreProcess() 42 | for _, f := range r.File { 43 | var keyPrefix string 44 | switch f.Name { 45 | case "job": 46 | keyPrefix = conf.Config.Cmd 47 | case "node_group": 48 | keyPrefix = conf.Config.Group 49 | } 50 | 51 | rc, err := f.Open() 52 | ea.ExitOnErr(err) 53 | 54 | ea.ExitOnErr(restoreKvs(rc, keyPrefix, restoreChan, wg)) 55 | rc.Close() 56 | } 57 | 58 | wg.Wait() 59 | close(restoreChan) 60 | }, 61 | } 62 | 63 | type kv struct { 64 | k, v string 65 | } 66 | 67 | var ( 68 | keyLenBuf = make([]byte, 2) 69 | valLenBuf = make([]byte, 4) 70 | keyBuf = make([]byte, 256) 71 | valBuf = make([]byte, 1024) 72 | ) 73 | 74 | func fixRead(r io.Reader, p []byte) (int, error) { 75 | valLen, readLen := len(p), 0 76 | for readLen != valLen { 77 | n, err := r.Read(p[readLen:]) 78 | readLen += n 79 | if err != nil { 80 | return readLen, err 81 | } 82 | } 83 | return readLen, nil 84 | } 85 | 86 | 87 | func restoreKvs(r io.Reader, keyPrefix string, storeChan chan *kv, wg *sync.WaitGroup) error { 88 | for { 89 | // read length of key 90 | n, err := fixRead(r, keyLenBuf) 91 | if err == io.EOF && n != 0 { 92 | return fmt.Errorf("unexcepted data, the file may broken") 93 | } else if err == io.EOF && n == 0 { 94 | break 95 | } else if err != nil { 96 | return err 97 | } 98 | keyLen := binary.LittleEndian.Uint16(keyLenBuf) 99 | 100 | // read key 101 | if n, err = fixRead(r, keyBuf[:keyLen]); err != nil { 102 | return err 103 | } 104 | key := keyBuf[:keyLen] 105 | 106 | // read length of value 107 | if n, err = fixRead(r, valLenBuf); err != nil { 108 | return err 109 | } 110 | valLen := binary.LittleEndian.Uint32(valLenBuf) 111 | 112 | // read value 113 | if len(valBuf) < int(valLen) { 114 | valBuf = make([]byte, valLen*2) 115 | } 116 | if n, err = fixRead(r, valBuf[:valLen]); err != nil && err != io.EOF { 117 | return err 118 | } 119 | value := valBuf[:valLen] 120 | 121 | wg.Add(1) 122 | storeChan <- &kv{ 123 | k: keyPrefix + string(key), 124 | v: string(value), 125 | } 126 | } 127 | 128 | return nil 129 | } 130 | 131 | func startRestoreProcess() (chan *kv, *sync.WaitGroup) { 132 | c := make(chan *kv, 0) 133 | wg := &sync.WaitGroup{} 134 | 135 | const maxResries = 3 136 | go func() { 137 | for _kv := range c { 138 | for retries := 1; retries <= maxResries; retries++ { 139 | _, err := cronsun.DefalutClient.Put(_kv.k, _kv.v) 140 | if err != nil { 141 | if retries == maxResries { 142 | fmt.Println("[Error] restore err:", err) 143 | fmt.Println("\tKey: ", string(_kv.k)) 144 | fmt.Println("\tValue: ", string(_kv.v)) 145 | } 146 | continue 147 | } 148 | } 149 | 150 | wg.Done() 151 | } 152 | }() 153 | 154 | return c, wg 155 | } 156 | -------------------------------------------------------------------------------- /bin/csctl/cmd/upgrade.go: -------------------------------------------------------------------------------- 1 | package cmd 2 | 3 | import ( 4 | "encoding/json" 5 | "fmt" 6 | "strings" 7 | 8 | "github.com/coreos/etcd/clientv3" 9 | "github.com/spf13/cobra" 10 | mgo "gopkg.in/mgo.v2" 11 | "gopkg.in/mgo.v2/bson" 12 | 13 | "github.com/shunfei/cronsun" 14 | "github.com/shunfei/cronsun/conf" 15 | ) 16 | 17 | var prever string 18 | 19 | func init() { 20 | UpgradeCmd.Flags().StringVarP(&prever, "prever", "p", "", "previous version of cronsun you are used") 21 | } 22 | 23 | var UpgradeCmd = &cobra.Command{ 24 | Use: "upgrade", 25 | Short: "upgrade will upgrade data to the current version(" + cronsun.VersionNumber + ")", 26 | Run: func(cmd *cobra.Command, args []string) { 27 | var ea = NewExitAction() 28 | 29 | prever = strings.TrimLeft(strings.TrimSpace(prever), "v") 30 | if len(prever) < 5 { 31 | ea.Exit("invalid version number") 32 | } 33 | 34 | nodesById := getIPMapper(ea, prever) 35 | if prever < "0.3.0" { 36 | fmt.Println("upgrading data to version 0.3.0") 37 | if to_0_3_0(ea, nodesById) { 38 | return 39 | } 40 | } 41 | 42 | if prever < "0.3.1" { 43 | fmt.Println("upgrading data to version 0.3.1") 44 | if to_0_3_1(ea, nodesById) { 45 | return 46 | } 47 | } 48 | }, 49 | } 50 | 51 | func getIPMapper(ea *ExitAction, prever string) map[string]*cronsun.Node { 52 | nodes, err := cronsun.GetNodes() 53 | if err != nil { 54 | ea.Exit("failed to fetch nodes from MongoDB: %s", err.Error()) 55 | } 56 | 57 | var ipMapper = make(map[string]*cronsun.Node, len(nodes)) 58 | for _, n := range nodes { 59 | n.IP = strings.TrimSpace(n.IP) 60 | if n.IP == "" || n.ID == "" { 61 | continue 62 | } 63 | 64 | if prever < "0.3.0" { 65 | n.RmOldInfo() 66 | } 67 | ipMapper[n.IP] = n 68 | } 69 | 70 | return ipMapper 71 | } 72 | 73 | // to_0_3_0 can be run many times 74 | func to_0_3_0(ea *ExitAction, nodesById map[string]*cronsun.Node) (shouldStop bool) { 75 | var replaceIDs = func(list []string) { 76 | for i := range list { 77 | if node, ok := nodesById[list[i]]; ok { 78 | list[i] = node.ID 79 | } 80 | } 81 | } 82 | 83 | // update job data 84 | gresp, err := cronsun.DefalutClient.Get(conf.Config.Cmd, clientv3.WithPrefix()) 85 | ea.ExitOnErr(err) 86 | 87 | total := len(gresp.Kvs) 88 | upgraded := 0 89 | for i := range gresp.Kvs { 90 | job := cronsun.Job{} 91 | err = json.Unmarshal(gresp.Kvs[i].Value, &job) 92 | if err != nil { 93 | fmt.Printf("[Error] failed to decode job(%s) data: %s\n", string(gresp.Kvs[i].Key), err.Error()) 94 | continue 95 | } 96 | 97 | for _, rule := range job.Rules { 98 | replaceIDs(rule.ExcludeNodeIDs) 99 | replaceIDs(rule.NodeIDs) 100 | } 101 | 102 | d, err := json.Marshal(&job) 103 | if err != nil { 104 | fmt.Printf("[Error] failed to encode job(%s) data: %s\n", string(gresp.Kvs[i].Key), err.Error()) 105 | continue 106 | } 107 | 108 | _, err = cronsun.DefalutClient.Put(job.Key(), string(d)) 109 | if err != nil { 110 | fmt.Printf("[Warn] failed to restore job(%s) data: %s\n", string(gresp.Kvs[i].Key), err.Error()) 111 | continue 112 | } 113 | upgraded++ 114 | } 115 | if total != upgraded { 116 | shouldStop = true 117 | } 118 | fmt.Printf("%d of %d jobs has been upgraded.\n", upgraded, total) 119 | 120 | // migrate node group data 121 | nodeGroups, err := cronsun.GetNodeGroups() 122 | if err != nil { 123 | ea.Exit("[Error] failed to get node group datas: ", err.Error()) 124 | } 125 | 126 | total = len(nodeGroups) 127 | upgraded = 0 128 | for i := range nodeGroups { 129 | replaceIDs(nodeGroups[i].NodeIDs) 130 | if _, err = nodeGroups[i].Put(0); err != nil { 131 | fmt.Printf("[Warn] failed to restore node group(id: %s, name: %s) data: %s\n", nodeGroups[i].ID, nodeGroups[i].Name, err.Error()) 132 | continue 133 | } 134 | upgraded++ 135 | } 136 | if total != upgraded { 137 | shouldStop = true 138 | } 139 | fmt.Printf("%d of %d node group has been upgraded.\n", upgraded, total) 140 | 141 | // upgrade logs 142 | cronsun.GetDb().WithC(cronsun.Coll_JobLog, func(c *mgo.Collection) error { 143 | for ip, node := range nodesById { 144 | _, err = c.UpdateAll(bson.M{"node": ip}, bson.M{"$set": bson.M{"node": node.ID, "hostname": node.Hostname}}) 145 | if err != nil { 146 | fmt.Println("failed to upgrade job logs: ", err.Error()) 147 | break 148 | } 149 | } 150 | shouldStop = true 151 | return err 152 | }) 153 | 154 | // upgrade logs 155 | cronsun.GetDb().WithC(cronsun.Coll_JobLatestLog, func(c *mgo.Collection) error { 156 | for ip, node := range nodesById { 157 | _, err = c.UpdateAll(bson.M{"node": ip}, bson.M{"$set": bson.M{"node": node.ID, "hostname": node.Hostname}}) 158 | if err != nil { 159 | fmt.Println("failed to upgrade job latest logs: ", err.Error()) 160 | break 161 | } 162 | } 163 | shouldStop = true 164 | return err 165 | }) 166 | 167 | return 168 | } 169 | 170 | // to_0_3_0 can be run many times 171 | func to_0_3_1(ea *ExitAction, nodesById map[string]*cronsun.Node) (shouldStop bool) { 172 | // upgrade logs 173 | var err error 174 | cronsun.GetDb().WithC(cronsun.Coll_JobLog, func(c *mgo.Collection) error { 175 | for _, node := range nodesById { 176 | _, err = c.UpdateAll(bson.M{"node": node.ID}, bson.M{"$set": bson.M{"ip": node.IP}}) 177 | if err != nil { 178 | fmt.Println("failed to upgrade job logs: ", err.Error()) 179 | break 180 | } 181 | } 182 | shouldStop = true 183 | return err 184 | }) 185 | 186 | cronsun.GetDb().WithC(cronsun.Coll_JobLatestLog, func(c *mgo.Collection) error { 187 | for _, node := range nodesById { 188 | _, err = c.UpdateAll(bson.M{"node": node.ID}, bson.M{"$set": bson.M{"ip": node.IP}}) 189 | if err != nil { 190 | fmt.Println("failed to upgrade job latest logs: ", err.Error()) 191 | break 192 | } 193 | } 194 | shouldStop = true 195 | return err 196 | }) 197 | 198 | return 199 | } 200 | -------------------------------------------------------------------------------- /bin/node/server.go: -------------------------------------------------------------------------------- 1 | // node 服务 2 | // 用于在所需要执行 cron 任务的机器启动服务，替代 cron 执行所需的任务 3 | package main 4 | 5 | import ( 6 | "flag" 7 | slog "log" 8 | 9 | "go.uber.org/zap" 10 | "go.uber.org/zap/zapcore" 11 | 12 | "github.com/shunfei/cronsun" 13 | "github.com/shunfei/cronsun/conf" 14 | "github.com/shunfei/cronsun/event" 15 | "github.com/shunfei/cronsun/log" 16 | "github.com/shunfei/cronsun/node" 17 | ) 18 | 19 | var ( 20 | level = flag.Int("l", 0, "log level, -1:debug, 0:info, 1:warn, 2:error") 21 | confFile = flag.String("conf", "conf/files/base.json", "config file path") 22 | ) 23 | 24 | func main() { 25 | flag.Parse() 26 | 27 | lcf := zap.NewDevelopmentConfig() 28 | lcf.Level.SetLevel(zapcore.Level(*level)) 29 | lcf.Development = false 30 | lcf.DisableStacktrace = true 31 | logger, err := lcf.Build(zap.AddCallerSkip(1)) 32 | if err != nil { 33 | slog.Fatalln("new log err:", err.Error()) 34 | } 35 | log.SetLogger(logger.Sugar()) 36 | 37 | if err = cronsun.Init(*confFile, true); err != nil { 38 | log.Errorf(err.Error()) 39 | return 40 | } 41 | 42 | n, err := node.NewNode(conf.Config) 43 | if err != nil { 44 | log.Errorf(err.Error()) 45 | return 46 | } 47 | 48 | if err = n.Register(); err != nil { 49 | log.Errorf(err.Error()) 50 | return 51 | } 52 | 53 | if err = cronsun.StartProc(); err != nil { 54 | log.Warnf("[process key will not timeout]proc lease id set err: %s", err.Error()) 55 | } 56 | 57 | if err = n.Run(); err != nil { 58 | log.Errorf(err.Error()) 59 | return 60 | } 61 | 62 | log.Infof("cronsun %s service started, Ctrl+C or send kill sign to exit", n.String()) 63 | // 注册退出事件 64 | event.On(event.EXIT, n.Stop, conf.Exit, cronsun.Exit) 65 | // 注册监听配置更新事件 66 | event.On(event.WAIT, cronsun.Reload) 67 | // 监听退出信号 68 | event.Wait() 69 | // 处理退出事件 70 | event.Emit(event.EXIT, nil) 71 | log.Infof("exit success") 72 | } 73 | -------------------------------------------------------------------------------- /bin/web/server.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "flag" 5 | slog "log" 6 | "net" 7 | "time" 8 | 9 | "github.com/cockroachdb/cmux" 10 | "go.uber.org/zap" 11 | "go.uber.org/zap/zapcore" 12 | 13 | "github.com/shunfei/cronsun" 14 | "github.com/shunfei/cronsun/conf" 15 | "github.com/shunfei/cronsun/event" 16 | "github.com/shunfei/cronsun/log" 17 | "github.com/shunfei/cronsun/web" 18 | ) 19 | 20 | var ( 21 | level = flag.Int("l", 0, "log level, -1:debug, 0:info, 1:warn, 2:error") 22 | confFile = flag.String("conf", "conf/files/base.json", "config file path") 23 | network = flag.String("network", "", "network protocol of listen address: ipv4/ipv6, or empty use both") 24 | ) 25 | 26 | func main() { 27 | flag.Parse() 28 | 29 | lcf := zap.NewDevelopmentConfig() 30 | lcf.Level.SetLevel(zapcore.Level(*level)) 31 | lcf.Development = false 32 | logger, err := lcf.Build(zap.AddCallerSkip(1)) 33 | if err != nil { 34 | slog.Fatalln("new log err:", err.Error()) 35 | } 36 | log.SetLogger(logger.Sugar()) 37 | 38 | if err = cronsun.Init(*confFile, true); err != nil { 39 | log.Errorf(err.Error()) 40 | return 41 | } 42 | web.EnsureJobLogIndex() 43 | 44 | l, err := net.Listen(checkNetworkProtocol(*network), conf.Config.Web.BindAddr) 45 | if err != nil { 46 | log.Errorf(err.Error()) 47 | return 48 | } 49 | 50 | // Create a cmux. 51 | m := cmux.New(l) 52 | httpL := m.Match(cmux.HTTP1Fast()) 53 | httpServer, err := web.InitServer() 54 | if err != nil { 55 | log.Errorf(err.Error()) 56 | return 57 | } 58 | 59 | if conf.Config.Mail.Enable { 60 | var noticer cronsun.Noticer 61 | 62 | if len(conf.Config.Mail.HttpAPI) > 0 { 63 | noticer = &cronsun.HttpAPI{} 64 | } else { 65 | mailer, err := cronsun.NewMail(30 * time.Second) 66 | if err != nil { 67 | log.Errorf(err.Error()) 68 | return 69 | } 70 | noticer = mailer 71 | } 72 | go cronsun.StartNoticer(noticer) 73 | } 74 | 75 | period := int64(conf.Config.Web.LogCleaner.EveryMinute) 76 | var stopCleaner func(interface{}) 77 | if period > 0 { 78 | closeChan := web.RunLogCleaner(time.Duration(period)*time.Minute, time.Duration(conf.Config.Web.LogCleaner.ExpirationDays)*time.Hour*24) 79 | stopCleaner = func(i interface{}) { 80 | close(closeChan) 81 | } 82 | } 83 | 84 | go func() { 85 | err := httpServer.Serve(httpL) 86 | if err != nil { 87 | panic(err.Error()) 88 | } 89 | }() 90 | 91 | go m.Serve() 92 | 93 | log.Infof("cronsun web server started on %s, Ctrl+C or send kill sign to exit", conf.Config.Web.BindAddr) 94 | // 注册退出事件 95 | event.On(event.EXIT, conf.Exit, stopCleaner) 96 | // 监听退出信号 97 | event.Wait() 98 | event.Emit(event.EXIT, nil) 99 | log.Infof("exit success") 100 | } 101 | 102 | func checkNetworkProtocol(p string) string { 103 | switch p { 104 | case "ipv4": 105 | return "tcp4" 106 | case "ipv6": 107 | return "tcp6" 108 | } 109 | 110 | return "tcp" 111 | } 112 | -------------------------------------------------------------------------------- /build.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | 3 | function check_code() { 4 | EXCODE=$? 5 | if [ "$EXCODE" != "0" ]; then 6 | echo "build fail." 7 | exit $EXCODE 8 | fi 9 | } 10 | 11 | out="dist" 12 | echo "build file to ./$out" 13 | 14 | mkdir -p "$out/conf" 15 | 16 | go build -o ./$out/cronnode ./bin/node/server.go 17 | check_code 18 | go build -o ./$out/cronweb ./bin/web/server.go 19 | check_code 20 | go build -o ./$out/csctl ./bin/csctl/cmd.go 21 | check_code 22 | 23 | sources=`find ./conf/files -name "*.json.sample"` 24 | check_code 25 | for source in $sources;do 26 | yes | echo $source|sed "s/.*\/\(.*\.json\).*/cp -f & .\/$out\/conf\/\1/"|bash 27 | check_code 28 | done 29 | 30 | echo "build success." 31 | -------------------------------------------------------------------------------- /client.go: -------------------------------------------------------------------------------- 1 | package cronsun 2 | 3 | import ( 4 | "context" 5 | "fmt" 6 | "strings" 7 | "time" 8 | 9 | client "github.com/coreos/etcd/clientv3" 10 | 11 | "github.com/shunfei/cronsun/conf" 12 | ) 13 | 14 | var ( 15 | DefalutClient *Client 16 | ) 17 | 18 | type Client struct { 19 | *client.Client 20 | 21 | reqTimeout time.Duration 22 | } 23 | 24 | func NewClient(cfg *conf.Conf) (c *Client, err error) { 25 | cli, err := client.New(cfg.Etcd.Copy()) 26 | if err != nil { 27 | return 28 | } 29 | 30 | c = &Client{ 31 | Client: cli, 32 | 33 | reqTimeout: time.Duration(cfg.ReqTimeout) * time.Second, 34 | } 35 | return 36 | } 37 | 38 | func (c *Client) Put(key, val string, opts ...client.OpOption) (*client.PutResponse, error) { 39 | ctx, cancel := NewEtcdTimeoutContext(c) 40 | defer cancel() 41 | return c.Client.Put(ctx, key, val, opts...) 42 | } 43 | 44 | func (c *Client) PutWithModRev(key, val string, rev int64) (*client.PutResponse, error) { 45 | if rev == 0 { 46 | return c.Put(key, val) 47 | } 48 | 49 | ctx, cancel := NewEtcdTimeoutContext(c) 50 | tresp, err := DefalutClient.Txn(ctx). 51 | If(client.Compare(client.ModRevision(key), "=", rev)). 52 | Then(client.OpPut(key, val)). 53 | Commit() 54 | cancel() 55 | if err != nil { 56 | return nil, err 57 | } 58 | 59 | if !tresp.Succeeded { 60 | return nil, ErrValueMayChanged 61 | } 62 | 63 | resp := client.PutResponse(*tresp.Responses[0].GetResponsePut()) 64 | return &resp, nil 65 | } 66 | 67 | func (c *Client) Get(key string, opts ...client.OpOption) (*client.GetResponse, error) { 68 | ctx, cancel := NewEtcdTimeoutContext(c) 69 | defer cancel() 70 | return c.Client.Get(ctx, key, opts...) 71 | } 72 | 73 | func (c *Client) Delete(key string, opts ...client.OpOption) (*client.DeleteResponse, error) { 74 | ctx, cancel := NewEtcdTimeoutContext(c) 75 | defer cancel() 76 | return c.Client.Delete(ctx, key, opts...) 77 | } 78 | 79 | func (c *Client) Watch(key string, opts ...client.OpOption) client.WatchChan { 80 | return c.Client.Watch(context.Background(), key, opts...) 81 | } 82 | 83 | func (c *Client) Grant(ttl int64) (*client.LeaseGrantResponse, error) { 84 | ctx, cancel := NewEtcdTimeoutContext(c) 85 | defer cancel() 86 | return c.Client.Grant(ctx, ttl) 87 | } 88 | 89 | func (c *Client) Revoke(id client.LeaseID) (*client.LeaseRevokeResponse, error) { 90 | ctx, cancel := context.WithTimeout(context.Background(), c.reqTimeout) 91 | defer cancel() 92 | return c.Client.Revoke(ctx, id) 93 | } 94 | 95 | func (c *Client) KeepAliveOnce(id client.LeaseID) (*client.LeaseKeepAliveResponse, error) { 96 | ctx, cancel := NewEtcdTimeoutContext(c) 97 | defer cancel() 98 | return c.Client.KeepAliveOnce(ctx, id) 99 | } 100 | 101 | func (c *Client) GetLock(key string, id client.LeaseID) (bool, error) { 102 | key = conf.Config.Lock + key 103 | ctx, cancel := NewEtcdTimeoutContext(c) 104 | resp, err := DefalutClient.Txn(ctx). 105 | If(client.Compare(client.CreateRevision(key), "=", 0)). 106 | Then(client.OpPut(key, "", client.WithLease(id))). 107 | Commit() 108 | cancel() 109 | 110 | if err != nil { 111 | return false, err 112 | } 113 | 114 | return resp.Succeeded, nil 115 | } 116 | 117 | func (c *Client) DelLock(key string) error { 118 | _, err := c.Delete(conf.Config.Lock + key) 119 | return err 120 | } 121 | 122 | func IsValidAsKeyPath(s string) bool { 123 | return strings.IndexAny(s, "/\\") == -1 124 | } 125 | 126 | // etcdTimeoutContext return better error info 127 | type etcdTimeoutContext struct { 128 | context.Context 129 | 130 | etcdEndpoints []string 131 | } 132 | 133 | func (c *etcdTimeoutContext) Err() error { 134 | err := c.Context.Err() 135 | if err == context.DeadlineExceeded { 136 | err = fmt.Errorf("%s: etcd(%v) maybe lost", 137 | err, c.etcdEndpoints) 138 | } 139 | return err 140 | } 141 | 142 | // NewEtcdTimeoutContext return a new etcdTimeoutContext 143 | func NewEtcdTimeoutContext(c *Client) (context.Context, context.CancelFunc) { 144 | ctx, cancel := context.WithTimeout(context.Background(), c.reqTimeout) 145 | etcdCtx := &etcdTimeoutContext{} 146 | etcdCtx.Context = ctx 147 | etcdCtx.etcdEndpoints = c.Endpoints() 148 | return etcdCtx, cancel 149 | } 150 | -------------------------------------------------------------------------------- /common.go: -------------------------------------------------------------------------------- 1 | package cronsun 2 | 3 | import ( 4 | "fmt" 5 | "os" 6 | 7 | "github.com/shunfei/cronsun/conf" 8 | "github.com/shunfei/cronsun/db" 9 | ) 10 | 11 | var ( 12 | initialized bool 13 | 14 | _Uid int 15 | ) 16 | 17 | func Init(baseConfFile string, watchConfiFile bool) (err error) { 18 | if initialized { 19 | return 20 | } 21 | 22 | // init id creator 23 | if err = initID(); err != nil { 24 | return fmt.Errorf("Init UUID Generator failed: %s", err) 25 | } 26 | 27 | // init config 28 | if err = conf.Init(baseConfFile, watchConfiFile); err != nil { 29 | return fmt.Errorf("Init Config failed: %s", err) 30 | } 31 | 32 | // init etcd client 33 | if DefalutClient, err = NewClient(conf.Config); err != nil { 34 | return fmt.Errorf("Connect to ETCD %s failed: %s", 35 | conf.Config.Etcd.Endpoints, err) 36 | } 37 | 38 | // init mongoDB 39 | if mgoDB, err = db.NewMdb(conf.Config.Mgo); err != nil { 40 | return fmt.Errorf("Connect to MongoDB %s failed: %s", 41 | conf.Config.Mgo.Hosts, err) 42 | } 43 | 44 | _Uid = os.Getuid() 45 | 46 | initialized = true 47 | return 48 | } 49 | -------------------------------------------------------------------------------- /conf/files/base.json.sample: -------------------------------------------------------------------------------- 1 | { 2 | "Web": "@extend:web.json", 3 | "Node": "/cronsun/node/", 4 | "Proc": "/cronsun/proc/", 5 | "Cmd": "/cronsun/cmd/", 6 | "Once": "/cronsun/once/", 7 | "Csctl": "/cronsun/csctl/", 8 | "Lock": "/cronsun/lock/", 9 | "Group": "/cronsun/group/", 10 | "Noticer": "/cronsun/noticer/", 11 | "#Ttl": "节点超时时间，单位秒", 12 | "Ttl": 10, 13 | "#ReqTimeout": "etcd 请求超时时间，单位秒", 14 | "ReqTimeout": 2, 15 | "#ProcTtl": "执行中的任务信息过期时间，单位秒，0 为不过期", 16 | "ProcTtl": 600, 17 | "#ProcReq": "记录任务执行中的信息的执行时间阀值，单位秒，0 为不限制", 18 | "ProcReq": 5, 19 | "#LockTtl": "任务锁最大过期时间，单位秒,默认 600", 20 | "LockTtl": 600, 21 | "Etcd": "@extend:etcd.json", 22 | "Mgo": "@extend:db.json", 23 | "Mail": "@extend:mail.json", 24 | "Security": "@extend:security.json", 25 | "#comment": "PIDFile and UUIDFile just work for cronnode", 26 | "#PIDFile": "Given a none-empty string to write a pid file to the specialed path, or leave it empty to do nothing", 27 | "PIDFile": "/var/run/cronsun/cronnode.pid", 28 | "UUIDFile": "/etc/cronsun/CRONSUN_UUID" 29 | } 30 | -------------------------------------------------------------------------------- /conf/files/db.json.sample: -------------------------------------------------------------------------------- 1 | { 2 | "Hosts": [ 3 | "127.0.0.1:27017" 4 | ], 5 | "Database": "cronsun", 6 | "#AuthSource": "AuthSource Specify the database name associated with the user’s credentials.", 7 | "#AuthSource": "AuthSource defaults to the cronsun's Database.", 8 | "#AuthSource": "If connect mongodb like './bin/mongo mytest -u test -p 123 --authenticationDatabase admin' ", 9 | "#AuthSource": "the AuthSource is 'admin'. ", 10 | "AuthSource": "", 11 | "UserName": "", 12 | "Password": "", 13 | "#Timeout": "connect timeout duration/second", 14 | "Timeout": 15 15 | } 16 | -------------------------------------------------------------------------------- /conf/files/etcd.json.sample: -------------------------------------------------------------------------------- 1 | { 2 | "Endpoints":[ 3 | "http://127.0.0.1:2379" 4 | ], 5 | "Username":"", 6 | "Password":"", 7 | "#DialTimeout":"单位秒", 8 | "DialTimeout": 2 9 | } 10 | -------------------------------------------------------------------------------- /conf/files/mail.json.sample: -------------------------------------------------------------------------------- 1 | { 2 | "Enable": false, 3 | "To": ["na@nb.com"], 4 | "#HttpAPI": "如有此字段，则按 http api 方式发送", 5 | "#Keepalive": "如果此时间段内没有邮件发送，则关闭 SMTP 连接，单位/秒", 6 | "Keepalive": 60, 7 | "#doc": "https://godoc.org/github.com/go-gomail/gomail#Dialer", 8 | "Host": "smtp.exmail.qq.com", 9 | "Port": 25, 10 | "Username": "sendmail@nb.com", 11 | "Password": "nbhh", 12 | "SSL": false, 13 | "#LocalName": "LocalName is the hostname sent to the SMTP server with the HELO command. By default, 'localhost' is sent.", 14 | "LocalName": "localhost" 15 | } 16 | -------------------------------------------------------------------------------- /conf/files/security.json.sample: -------------------------------------------------------------------------------- 1 | { 2 | "open": false, 3 | "users": [ 4 | "www", "db" 5 | ], 6 | "ext": [ 7 | ".sh", ".py" 8 | ] 9 | } 10 | -------------------------------------------------------------------------------- /conf/files/web.json.sample: -------------------------------------------------------------------------------- 1 | { 2 | "BindAddr": ":7079", 3 | "Auth": { 4 | "#Enabled": "set to true to open auth. default username and password is admin@admin.com/admin", 5 | "Enabled": true 6 | }, 7 | "Session": { 8 | "StorePrefixPath": "/cronsun/sess/", 9 | "CookieName": "cronsun_uid", 10 | "Expiration": 8640000 11 | }, 12 | "#comment": "Delete the expired log (which store in mongodb) periodically", 13 | "LogCleaner": { 14 | "#comment": "if EveryMinute is 0, the LogCleaner will not run", 15 | "EveryMinute": 0, 16 | "ExpirationDays": 3 17 | } 18 | } 19 | -------------------------------------------------------------------------------- /csctl.go: -------------------------------------------------------------------------------- 1 | package cronsun 2 | 3 | import ( 4 | "encoding/json" 5 | "errors" 6 | 7 | client "github.com/coreos/etcd/clientv3" 8 | 9 | "github.com/shunfei/cronsun/conf" 10 | ) 11 | 12 | const ( 13 | NodeCmdUnknown NodeCmd = iota 14 | NodeCmdRmOld 15 | NodeCmdSync 16 | NodeCmdMax 17 | ) 18 | 19 | var ( 20 | InvalidNodeCmdErr = errors.New("invalid node command") 21 | 22 | NodeCmds = []string{ 23 | "unknown", 24 | "rmold", 25 | "sync", 26 | } 27 | ) 28 | 29 | type NodeCmd int 30 | 31 | func (cmd NodeCmd) String() string { 32 | if NodeCmdMax <= cmd || cmd <= NodeCmdUnknown { 33 | return "unknown" 34 | } 35 | return NodeCmds[cmd] 36 | } 37 | 38 | func ToNodeCmd(cmd string) (NodeCmd, error) { 39 | for nc := NodeCmdUnknown + 1; nc < NodeCmdMax; nc++ { 40 | if cmd == NodeCmds[nc] { 41 | return nc, nil 42 | } 43 | } 44 | return NodeCmdUnknown, InvalidNodeCmdErr 45 | } 46 | 47 | type CsctlCmd struct { 48 | // the command send to node 49 | Cmd NodeCmd 50 | // the node ids that needs to execute the command, empty means all node 51 | Include []string 52 | // the node ids that doesn't need to execute the command, empty means none 53 | Exclude []string 54 | } 55 | 56 | // 执行 csctl 发送的命令 57 | // 注册到 /cronsun/csctl/ 58 | func PutCsctl(cmd *CsctlCmd) error { 59 | if NodeCmdMax <= cmd.Cmd || cmd.Cmd <= NodeCmdUnknown { 60 | return InvalidNodeCmdErr 61 | } 62 | 63 | params, err := json.Marshal(cmd) 64 | if err != nil { 65 | return err 66 | } 67 | _, err = DefalutClient.Put(conf.Config.Csctl+NodeCmds[cmd.Cmd], string(params)) 68 | return err 69 | } 70 | 71 | func WatchCsctl() client.WatchChan { 72 | return DefalutClient.Watch(conf.Config.Csctl, client.WithPrefix()) 73 | } 74 | -------------------------------------------------------------------------------- /db/mgo.go: -------------------------------------------------------------------------------- 1 | package db 2 | 3 | import ( 4 | "net/url" 5 | "strings" 6 | "time" 7 | 8 | "gopkg.in/mgo.v2" 9 | "gopkg.in/mgo.v2/bson" 10 | ) 11 | 12 | type Config struct { 13 | Hosts []string 14 | // AuthSource Specify the database name associated with the user’s credentials. 15 | // authSource defaults to the database specified in the connection string. 16 | AuthSource string 17 | UserName string 18 | Password string 19 | Database string 20 | Timeout time.Duration // second 21 | } 22 | 23 | type Mdb struct { 24 | *Config 25 | *mgo.Session 26 | } 27 | 28 | func NewMdb(c *Config) (*Mdb, error) { 29 | m := &Mdb{ 30 | Config: c, 31 | } 32 | return m, m.connect() 33 | } 34 | 35 | func (m *Mdb) connect() error { 36 | // connectionString: [mongodb://][user:pass@]host1[:port1][,host2[:port2],...][/database][?options] 37 | // via: https://docs.mongodb.com/manual/reference/connection-string/ 38 | connectionString := strings.Join(m.Config.Hosts, ",") 39 | if len(m.Config.UserName) > 0 && len(m.Config.Password) > 0 { 40 | connectionString = m.Config.UserName + ":" + url.QueryEscape(m.Config.Password) + "@" + connectionString 41 | } 42 | 43 | if len(m.Config.Database) > 0 { 44 | connectionString += "/" + m.Config.Database 45 | } 46 | 47 | if len(m.Config.AuthSource) > 0 { 48 | connectionString += "?authSource=" + m.Config.AuthSource 49 | } 50 | 51 | session, err := mgo.DialWithTimeout(connectionString, m.Config.Timeout) 52 | if err != nil { 53 | return err 54 | } 55 | 56 | m.Session = session 57 | return nil 58 | } 59 | 60 | func (m *Mdb) WithC(collection string, job func(*mgo.Collection) error) error { 61 | s := m.Session.New() 62 | err := job(s.DB(m.Config.Database).C(collection)) 63 | s.Close() 64 | return err 65 | } 66 | 67 | func (self *Mdb) Upsert(collection string, selector interface{}, change interface{}) error { 68 | return self.WithC(collection, func(c *mgo.Collection) error { 69 | _, err := c.Upsert(selector, change) 70 | return err 71 | }) 72 | } 73 | 74 | func (self *Mdb) Insert(collection string, data ...interface{}) error { 75 | return self.WithC(collection, func(c *mgo.Collection) error { 76 | return c.Insert(data...) 77 | }) 78 | } 79 | 80 | func (self *Mdb) FindId(collection string, id interface{}, result interface{}) error { 81 | return self.WithC(collection, func(c *mgo.Collection) error { 82 | return c.Find(bson.M{"_id": id}).One(result) 83 | }) 84 | } 85 | 86 | func (self *Mdb) FindOne(collection string, query interface{}, result interface{}) error { 87 | return self.WithC(collection, func(c *mgo.Collection) error { 88 | return c.Find(query).One(result) 89 | }) 90 | } 91 | 92 | func (self *Mdb) RemoveId(collection string, id interface{}) error { 93 | return self.WithC(collection, func(c *mgo.Collection) error { 94 | return c.RemoveId(id) 95 | }) 96 | } 97 | -------------------------------------------------------------------------------- /db/mid/auto_inc.go: -------------------------------------------------------------------------------- 1 | package mid 2 | 3 | import ( 4 | "fmt" 5 | 6 | "gopkg.in/mgo.v2" 7 | "gopkg.in/mgo.v2/bson" 8 | ) 9 | 10 | var ( 11 | field = &Field{ 12 | Id: "seq", 13 | Collection: "_id", 14 | } 15 | ) 16 | 17 | type Field struct { 18 | Id string 19 | Collection string 20 | } 21 | 22 | // 如果不设置，则用默认设置 23 | func SetFieldName(id, collection string) { 24 | field.Id = id 25 | field.Collection = collection 26 | } 27 | 28 | //使collection 为 name 的 id 自增 1 并返回当前 id 的值 29 | func AutoInc(c *mgo.Collection, name string) (id int, err error) { 30 | return incr(c, name, 1) 31 | } 32 | 33 | //批量申请一段id 34 | func ApplyBatchIds(c *mgo.Collection, name string, amount int) (id int, err error) { 35 | return incr(c, name, amount) 36 | } 37 | 38 | func incr(c *mgo.Collection, name string, step int) (id int, err error) { 39 | result := make(map[string]interface{}) 40 | change := mgo.Change{ 41 | Update: bson.M{"$inc": bson.M{field.Id: step}}, 42 | Upsert: true, 43 | ReturnNew: true, 44 | } 45 | _, err = c.Find(bson.M{field.Collection: name}).Apply(change, result) 46 | if err != nil { 47 | return 48 | } 49 | id, ok := result[field.Id].(int) 50 | if ok { 51 | return 52 | } 53 | id64, ok := result[field.Id].(int64) 54 | if !ok { 55 | err = fmt.Errorf("%s is ont int or int64", field.Id) 56 | return 57 | } 58 | id = int(id64) 59 | return 60 | } 61 | -------------------------------------------------------------------------------- /doc/img/Cronsun_dashboard_en.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/shunfei/cronsun/0e5a7b9acb5534c64d003333dee9d714e24dd878/doc/img/Cronsun_dashboard_en.png -------------------------------------------------------------------------------- /doc/img/Cronsun_job_list_en.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/shunfei/cronsun/0e5a7b9acb5534c64d003333dee9d714e24dd878/doc/img/Cronsun_job_list_en.png -------------------------------------------------------------------------------- /doc/img/Cronsun_job_new_en.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/shunfei/cronsun/0e5a7b9acb5534c64d003333dee9d714e24dd878/doc/img/Cronsun_job_new_en.png -------------------------------------------------------------------------------- /doc/img/Cronsun_log_item_en.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/shunfei/cronsun/0e5a7b9acb5534c64d003333dee9d714e24dd878/doc/img/Cronsun_log_item_en.png -------------------------------------------------------------------------------- /doc/img/Cronsun_log_list_en.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/shunfei/cronsun/0e5a7b9acb5534c64d003333dee9d714e24dd878/doc/img/Cronsun_log_list_en.png -------------------------------------------------------------------------------- /doc/img/Cronsun_node_en.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/shunfei/cronsun/0e5a7b9acb5534c64d003333dee9d714e24dd878/doc/img/Cronsun_node_en.png -------------------------------------------------------------------------------- /doc/img/brief.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/shunfei/cronsun/0e5a7b9acb5534c64d003333dee9d714e24dd878/doc/img/brief.png -------------------------------------------------------------------------------- /doc/img/job.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/shunfei/cronsun/0e5a7b9acb5534c64d003333dee9d714e24dd878/doc/img/job.png -------------------------------------------------------------------------------- /doc/img/log.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/shunfei/cronsun/0e5a7b9acb5534c64d003333dee9d714e24dd878/doc/img/log.png -------------------------------------------------------------------------------- /doc/img/new_job.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/shunfei/cronsun/0e5a7b9acb5534c64d003333dee9d714e24dd878/doc/img/new_job.png -------------------------------------------------------------------------------- /doc/img/node.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/shunfei/cronsun/0e5a7b9acb5534c64d003333dee9d714e24dd878/doc/img/node.png -------------------------------------------------------------------------------- /errors.go: -------------------------------------------------------------------------------- 1 | package cronsun 2 | 3 | import "errors" 4 | 5 | var ( 6 | ErrNotFound = errors.New("Record not found.") 7 | ErrValueMayChanged = errors.New("The value has been changed by others on this time.") 8 | 9 | ErrEmptyJobName = errors.New("Name of job is empty.") 10 | ErrEmptyJobCommand = errors.New("Command of job is empty.") 11 | ErrIllegalJobId = errors.New("Invalid id that includes illegal characters such as '/' '\\'.") 12 | ErrIllegalJobGroupName = errors.New("Invalid job group name that includes illegal characters such as '/' '\\'.") 13 | 14 | ErrEmptyNodeGroupName = errors.New("Name of node group is empty.") 15 | ErrIllegalNodeGroupId = errors.New("Invalid node group id that includes illegal characters such as '/'.") 16 | 17 | ErrSecurityInvalidCmd = errors.New("Security error: the suffix of script file is not on the whitelist.") 18 | ErrSecurityInvalidUser = errors.New("Security error: the user is not on the whitelist.") 19 | ErrNilRule = errors.New("invalid job rule, empty timer.") 20 | ) 21 | -------------------------------------------------------------------------------- /event/event.go: -------------------------------------------------------------------------------- 1 | package event 2 | 3 | import ( 4 | "fmt" 5 | "os" 6 | "os/signal" 7 | "reflect" 8 | "syscall" 9 | ) 10 | 11 | const ( 12 | EXIT = "exit" 13 | WAIT = "wait" 14 | ) 15 | 16 | var ( 17 | Events = make(map[string][]func(interface{}), 2) 18 | ) 19 | 20 | func On(name string, fs ...func(interface{})) error { 21 | evs, ok := Events[name] 22 | if !ok { 23 | evs = make([]func(interface{}), 0, len(fs)) 24 | } 25 | 26 | for _, f := range fs { 27 | if f == nil { 28 | continue 29 | } 30 | 31 | fp := reflect.ValueOf(f).Pointer() 32 | for i := 0; i < len(evs); i++ { 33 | if reflect.ValueOf(evs[i]).Pointer() == fp { 34 | return fmt.Errorf("func[%v] already exists in event[%s]", fp, name) 35 | } 36 | } 37 | evs = append(evs, f) 38 | } 39 | Events[name] = evs 40 | return nil 41 | } 42 | 43 | func Emit(name string, arg interface{}) { 44 | evs, ok := Events[name] 45 | if !ok { 46 | return 47 | } 48 | 49 | for _, f := range evs { 50 | f(arg) 51 | } 52 | } 53 | 54 | func EmitAll(arg interface{}) { 55 | for _, fs := range Events { 56 | for _, f := range fs { 57 | f(arg) 58 | } 59 | } 60 | return 61 | } 62 | 63 | func Off(name string, f func(interface{})) error { 64 | evs, ok := Events[name] 65 | if !ok || len(evs) == 0 { 66 | return fmt.Errorf("envet[%s] doesn't have any funcs", name) 67 | } 68 | 69 | fp := reflect.ValueOf(f).Pointer() 70 | for i := 0; i < len(evs); i++ { 71 | if reflect.ValueOf(evs[i]).Pointer() == fp { 72 | evs = append(evs[:i], evs[i+1:]...) 73 | Events[name] = evs 74 | return nil 75 | } 76 | } 77 | 78 | return fmt.Errorf("%v func dones't exist in event[%s]", fp, name) 79 | } 80 | 81 | func OffAll(name string) error { 82 | Events[name] = nil 83 | return nil 84 | } 85 | 86 | // 等待信号 87 | // 如果信号参数为空，则会等待常见的终止信号 88 | // SIGINT 2 A 键盘中断（如break键被按下） 89 | // SIGTERM 15 A 终止信号 90 | func Wait(sig ...os.Signal) os.Signal { 91 | c := make(chan os.Signal, 1) 92 | if len(sig) == 0 { 93 | signal.Notify(c, syscall.SIGINT, syscall.SIGTERM) 94 | } else { 95 | signal.Notify(c, sig...) 96 | } 97 | return <-c 98 | } 99 | -------------------------------------------------------------------------------- /event/event_test.go: -------------------------------------------------------------------------------- 1 | package event 2 | 3 | import ( 4 | "testing" 5 | 6 | . "github.com/smartystreets/goconvey/convey" 7 | ) 8 | 9 | func TestEvent(t *testing.T) { 10 | i := []int{} 11 | f := func(s interface{}) { 12 | i = append(i, 1) 13 | } 14 | f2 := func(s interface{}) { 15 | i = append(i, 2) 16 | i = append(i, 3) 17 | } 18 | 19 | Convey("events package test", t, func() { 20 | Convey("init events package should be success", func() { 21 | So(len(i), ShouldEqual, 0) 22 | So(len(Events[EXIT]), ShouldEqual, 0) 23 | }) 24 | 25 | Convey("empty events execute Off function should not be success", func() { 26 | So(Off(EXIT, f), ShouldNotBeNil) 27 | }) 28 | 29 | Convey("multi execute On function for a function should not be success", func() { 30 | So(On(EXIT, f), ShouldBeNil) 31 | So(On(EXIT, f), ShouldNotBeNil) 32 | }) 33 | 34 | Convey("execute Emit function should be success", func() { 35 | Emit(EXIT, nil) 36 | So(len(i), ShouldEqual, 1) 37 | }) 38 | 39 | Convey("events package should be work", func() { 40 | So(On(EXIT, f2), ShouldBeNil) 41 | So(len(Events[EXIT]), ShouldEqual, 2) 42 | So(len(i), ShouldEqual, 1) 43 | 44 | So(Off(EXIT, f), ShouldBeNil) 45 | So(len(Events[EXIT]), ShouldEqual, 1) 46 | }) 47 | }) 48 | } 49 | -------------------------------------------------------------------------------- /go.mod: -------------------------------------------------------------------------------- 1 | module github.com/shunfei/cronsun 2 | 3 | require ( 4 | github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973 // indirect 5 | github.com/boltdb/bolt v1.3.1 // indirect 6 | github.com/cockroachdb/cmux v0.0.0-20170110192607-30d10be49292 7 | github.com/coreos/bbolt v1.3.0 // indirect 8 | github.com/coreos/etcd v3.3.9+incompatible 9 | github.com/coreos/go-semver v0.2.0 // indirect 10 | github.com/coreos/go-systemd v0.0.0-20180828140353-eee3db372b31 // indirect 11 | github.com/coreos/pkg v0.0.0-20180108230652-97fdf19511ea // indirect 12 | github.com/davecgh/go-spew v1.1.1 // indirect 13 | github.com/dgrijalva/jwt-go v3.2.0+incompatible // indirect 14 | github.com/fsnotify/fsnotify v1.4.7 15 | github.com/ghodss/yaml v1.0.0 // indirect 16 | github.com/go-gomail/gomail v0.0.0-20160411212932-81ebce5c23df 17 | github.com/gofrs/uuid v3.1.0+incompatible 18 | github.com/gogo/protobuf v1.1.1 // indirect 19 | github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b // indirect 20 | github.com/golang/groupcache v0.0.0-20180513044358-24b0969c4cb7 // indirect 21 | github.com/golang/protobuf v1.2.0 // indirect 22 | github.com/google/btree v0.0.0-20180813153112-4030bb1f1f0c // indirect 23 | github.com/gopherjs/gopherjs v0.0.0-20180825215210-0210a2f0f73c // indirect 24 | github.com/gorilla/context v1.1.1 // indirect 25 | github.com/gorilla/mux v1.6.2 26 | github.com/gorilla/websocket v1.4.0 // indirect 27 | github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0 // indirect 28 | github.com/grpc-ecosystem/grpc-gateway v1.4.1 // indirect 29 | github.com/inconshreveable/mousetrap v1.0.0 // indirect 30 | github.com/jonboulle/clockwork v0.1.0 // indirect 31 | github.com/jtolds/gls v4.2.1+incompatible // indirect 32 | github.com/matttproud/golang_protobuf_extensions v1.0.1 // indirect 33 | github.com/pkg/errors v0.8.0 // indirect 34 | github.com/pmezard/go-difflib v1.0.0 // indirect 35 | github.com/prometheus/client_golang v0.8.0 // indirect 36 | github.com/prometheus/client_model v0.0.0-20180712105110-5c3871d89910 // indirect 37 | github.com/prometheus/common v0.0.0-20180801064454-c7de2306084e // indirect 38 | github.com/prometheus/procfs v0.0.0-20180725123919-05ee40e3a273 // indirect 39 | github.com/rogpeppe/fastuuid v0.0.0-20150106093220-6724a57986af 40 | github.com/sirupsen/logrus v1.0.6 // indirect 41 | github.com/smartystreets/assertions v0.0.0-20180820201707-7c9eb446e3cf // indirect 42 | github.com/smartystreets/goconvey v0.0.0-20180222194500-ef6db91d284a 43 | github.com/soheilhy/cmux v0.1.4 // indirect 44 | github.com/spf13/cobra v0.0.3 45 | github.com/spf13/pflag v1.0.2 // indirect 46 | github.com/stretchr/testify v1.2.2 // indirect 47 | github.com/tmc/grpc-websocket-proxy v0.0.0-20171017195756-830351dc03c6 // indirect 48 | github.com/ugorji/go/codec v0.0.0-20180831062425-e253f1f20942 // indirect 49 | github.com/xiang90/probing v0.0.0-20160813154853-07dd2e8dfe18 // indirect 50 | go.uber.org/atomic v1.3.2 // indirect 51 | go.uber.org/multierr v1.1.0 // indirect 52 | go.uber.org/zap v1.9.1 53 | golang.org/x/net v0.0.0-20180826012351-8a410e7b638d // indirect 54 | golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33 // indirect 55 | golang.org/x/text v0.3.0 // indirect 56 | google.golang.org/genproto v0.0.0-20180831171423-11092d34479b // indirect 57 | google.golang.org/grpc v1.14.0 // indirect 58 | gopkg.in/alexcesaro/quotedprintable.v3 v3.0.0-20150716171945-2caba252f4dc // indirect 59 | gopkg.in/mgo.v2 v2.0.0-20180705113604-9856a29383ce 60 | ) 61 | -------------------------------------------------------------------------------- /group.go: -------------------------------------------------------------------------------- 1 | package cronsun 2 | 3 | import ( 4 | "encoding/json" 5 | "fmt" 6 | "strings" 7 | 8 | client "github.com/coreos/etcd/clientv3" 9 | 10 | "github.com/shunfei/cronsun/conf" 11 | "github.com/shunfei/cronsun/log" 12 | ) 13 | 14 | // 结点类型分组 15 | // 注册到 /cronsun/group/ 16 | type Group struct { 17 | ID string `json:"id"` 18 | Name string `json:"name"` 19 | 20 | NodeIDs []string `json:"nids"` 21 | } 22 | 23 | func GetGroupById(gid string) (g *Group, err error) { 24 | if len(gid) == 0 { 25 | return 26 | } 27 | resp, err := DefalutClient.Get(conf.Config.Group + gid) 28 | if err != nil || resp.Count == 0 { 29 | return 30 | } 31 | 32 | err = json.Unmarshal(resp.Kvs[0].Value, &g) 33 | return 34 | } 35 | 36 | // GetGroups 获取包含 nid 的 group 37 | // 如果 nid 为空，则获取所有的 group 38 | func GetGroups(nid string) (groups map[string]*Group, err error) { 39 | resp, err := DefalutClient.Get(conf.Config.Group, client.WithPrefix()) 40 | if err != nil { 41 | return 42 | } 43 | 44 | count := len(resp.Kvs) 45 | groups = make(map[string]*Group, count) 46 | if count == 0 { 47 | return 48 | } 49 | 50 | for _, g := range resp.Kvs { 51 | group := new(Group) 52 | if e := json.Unmarshal(g.Value, group); e != nil { 53 | log.Warnf("group[%s] umarshal err: %s", string(g.Key), e.Error()) 54 | continue 55 | } 56 | if len(nid) == 0 || group.Included(nid) { 57 | groups[group.ID] = group 58 | } 59 | } 60 | return 61 | } 62 | 63 | func WatchGroups() client.WatchChan { 64 | return DefalutClient.Watch(conf.Config.Group, client.WithPrefix(), client.WithPrevKV()) 65 | } 66 | 67 | func GetGroupFromKv(key, value []byte) (g *Group, err error) { 68 | g = new(Group) 69 | if err = json.Unmarshal(value, g); err != nil { 70 | err = fmt.Errorf("group[%s] umarshal err: %s", string(key), err.Error()) 71 | } 72 | return 73 | } 74 | 75 | func DeleteGroupById(id string) (*client.DeleteResponse, error) { 76 | return DefalutClient.Delete(GroupKey(id)) 77 | } 78 | 79 | func GroupKey(id string) string { 80 | return conf.Config.Group + id 81 | } 82 | 83 | func (g *Group) Key() string { 84 | return GroupKey(g.ID) 85 | } 86 | 87 | func (g *Group) Put(modRev int64) (*client.PutResponse, error) { 88 | b, err := json.Marshal(g) 89 | if err != nil { 90 | return nil, err 91 | } 92 | 93 | return DefalutClient.PutWithModRev(g.Key(), string(b), modRev) 94 | } 95 | 96 | func (g *Group) Check() error { 97 | g.ID = strings.TrimSpace(g.ID) 98 | if !IsValidAsKeyPath(g.ID) { 99 | return ErrIllegalNodeGroupId 100 | } 101 | 102 | g.Name = strings.TrimSpace(g.Name) 103 | if len(g.Name) == 0 { 104 | return ErrEmptyNodeGroupName 105 | } 106 | 107 | return nil 108 | } 109 | 110 | func (g *Group) Included(nid string) bool { 111 | for i, count := 0, len(g.NodeIDs); i < count; i++ { 112 | if nid == g.NodeIDs[i] { 113 | return true 114 | } 115 | } 116 | 117 | return false 118 | } 119 | -------------------------------------------------------------------------------- /id.go: -------------------------------------------------------------------------------- 1 | package cronsun 2 | 3 | import ( 4 | "encoding/hex" 5 | 6 | "github.com/rogpeppe/fastuuid" 7 | ) 8 | 9 | var generator *fastuuid.Generator 10 | 11 | func initID() (err error) { 12 | generator, err = fastuuid.NewGenerator() 13 | return 14 | } 15 | 16 | func NextID() string { 17 | id := generator.Next() 18 | return hex.EncodeToString(id[:]) 19 | } 20 | -------------------------------------------------------------------------------- /job_log.go: -------------------------------------------------------------------------------- 1 | package cronsun 2 | 3 | import ( 4 | "time" 5 | 6 | "gopkg.in/mgo.v2" 7 | "gopkg.in/mgo.v2/bson" 8 | 9 | "github.com/shunfei/cronsun/conf" 10 | "github.com/shunfei/cronsun/log" 11 | ) 12 | 13 | const ( 14 | Coll_JobLog = "job_log" 15 | Coll_JobLatestLog = "job_latest_log" 16 | Coll_Stat = "stat" 17 | ) 18 | 19 | // 任务执行记录 20 | type JobLog struct { 21 | Id bson.ObjectId `bson:"_id,omitempty" json:"id"` 22 | JobId string `bson:"jobId" json:"jobId"` // 任务 Id，索引 23 | JobGroup string `bson:"jobGroup" json:"jobGroup"` // 任务分组，配合 Id 跳转用 24 | User string `bson:"user" json:"user"` // 执行此次任务的用户 25 | Name string `bson:"name" json:"name"` // 任务名称 26 | Node string `bson:"node" json:"node"` // 运行此次任务的节点 id，索引 27 | Hostname string `bson:"hostname" json:"hostname"` // 运行此次任务的节点主机名称，索引 28 | IP string `bson:"ip" json:"ip"` // 运行此次任务的节点主机IP，索引 29 | Command string `bson:"command" json:"command,omitempty"` // 执行的命令，包括参数 30 | Output string `bson:"output" json:"output,omitempty"` // 任务输出的所有内容 31 | Success bool `bson:"success" json:"success"` // 是否执行成功 32 | BeginTime time.Time `bson:"beginTime" json:"beginTime"` // 任务开始执行时间，精确到毫秒，索引 33 | EndTime time.Time `bson:"endTime" json:"endTime"` // 任务执行完毕时间，精确到毫秒 34 | Cleanup time.Time `bson:"cleanup,omitempty" json:"-"` // 日志清除时间标志 35 | } 36 | 37 | type JobLatestLog struct { 38 | JobLog `bson:",inline"` 39 | RefLogId string `bson:"refLogId,omitempty" json:"refLogId"` 40 | } 41 | 42 | func GetJobLogById(id bson.ObjectId) (l *JobLog, err error) { 43 | err = mgoDB.FindId(Coll_JobLog, id, &l) 44 | return 45 | } 46 | 47 | var selectForJobLogList = bson.M{"command": 0, "output": 0} 48 | 49 | func GetJobLogList(query bson.M, page, size int, sort string) (list []*JobLog, total int, err error) { 50 | err = mgoDB.WithC(Coll_JobLog, func(c *mgo.Collection) error { 51 | total, err = c.Find(query).Count() 52 | if err != nil { 53 | return err 54 | } 55 | return c.Find(query).Select(selectForJobLogList).Sort(sort).Skip((page - 1) * size).Limit(size).All(&list) 56 | }) 57 | return 58 | } 59 | 60 | func GetJobLatestLogList(query bson.M, page, size int, sort string) (list []*JobLatestLog, total int, err error) { 61 | err = mgoDB.WithC(Coll_JobLatestLog, func(c *mgo.Collection) error { 62 | total, err = c.Find(query).Count() 63 | if err != nil { 64 | return err 65 | } 66 | return c.Find(query).Select(selectForJobLogList).Sort(sort).Skip((page - 1) * size).Limit(size).All(&list) 67 | }) 68 | return 69 | } 70 | 71 | func GetJobLatestLogListByJobIds(jobIds []string) (m map[string]*JobLatestLog, err error) { 72 | var list []*JobLatestLog 73 | 74 | err = mgoDB.WithC(Coll_JobLatestLog, func(c *mgo.Collection) error { 75 | return c.Find(bson.M{"jobId": bson.M{"$in": jobIds}}).Select(selectForJobLogList).Sort("beginTime").All(&list) 76 | }) 77 | if err != nil { 78 | return 79 | } 80 | 81 | m = make(map[string]*JobLatestLog, len(list)) 82 | for i := range list { 83 | m[list[i].JobId] = list[i] 84 | } 85 | return 86 | } 87 | 88 | func CreateJobLog(j *Job, t time.Time, rs string, success bool) { 89 | et := time.Now() 90 | j.Avg(t, et) 91 | 92 | jl := JobLog{ 93 | Id: bson.NewObjectId(), 94 | JobId: j.ID, 95 | 96 | JobGroup: j.Group, 97 | Name: j.Name, 98 | User: j.User, 99 | 100 | Node: j.runOn, 101 | Hostname: j.hostname, 102 | IP: j.ip, 103 | 104 | Command: j.Command, 105 | Output: rs, 106 | Success: success, 107 | 108 | BeginTime: t, 109 | EndTime: et, 110 | } 111 | 112 | if conf.Config.Web.LogCleaner.EveryMinute > 0 { 113 | var expiration int 114 | if j.LogExpiration > 0 { 115 | expiration = j.LogExpiration 116 | } else { 117 | expiration = conf.Config.Web.LogCleaner.ExpirationDays 118 | } 119 | jl.Cleanup = jl.EndTime.Add(time.Duration(expiration) * time.Hour * 24) 120 | } 121 | 122 | if err := mgoDB.Insert(Coll_JobLog, jl); err != nil { 123 | log.Errorf(err.Error()) 124 | } 125 | 126 | latestLog := &JobLatestLog{ 127 | RefLogId: jl.Id.Hex(), 128 | JobLog: jl, 129 | } 130 | latestLog.Id = "" 131 | if err := mgoDB.Upsert(Coll_JobLatestLog, bson.M{"node": jl.Node, "hostname": jl.Hostname, "ip": jl.IP, "jobId": jl.JobId, "jobGroup": jl.JobGroup}, latestLog); err != nil { 132 | log.Errorf(err.Error()) 133 | } 134 | 135 | var inc = bson.M{"total": 1} 136 | if jl.Success { 137 | inc["successed"] = 1 138 | } else { 139 | inc["failed"] = 1 140 | } 141 | 142 | err := mgoDB.Upsert(Coll_Stat, bson.M{"name": "job-day", "date": time.Now().Format("2006-01-02")}, bson.M{"$inc": inc}) 143 | if err != nil { 144 | log.Errorf("increase stat.job %s", err.Error()) 145 | } 146 | err = mgoDB.Upsert(Coll_Stat, bson.M{"name": "job"}, bson.M{"$inc": inc}) 147 | if err != nil { 148 | log.Errorf("increase stat.job %s", err.Error()) 149 | } 150 | } 151 | 152 | type StatExecuted struct { 153 | Total int64 `bson:"total" json:"total"` 154 | Successed int64 `bson:"successed" json:"successed"` 155 | Failed int64 `bson:"failed" json:"failed"` 156 | Date string `bson:"date" json:"date"` 157 | } 158 | 159 | func JobLogStat() (s *StatExecuted, err error) { 160 | err = mgoDB.FindOne(Coll_Stat, bson.M{"name": "job"}, &s) 161 | return 162 | } 163 | 164 | func JobLogDailyStat(begin, end time.Time) (ls []*StatExecuted, err error) { 165 | const oneDay = time.Hour * 24 166 | err = mgoDB.WithC(Coll_Stat, func(c *mgo.Collection) error { 167 | dateList := make([]string, 0, 8) 168 | 169 | cur := begin 170 | for { 171 | dateList = append(dateList, cur.Format("2006-01-02")) 172 | cur = cur.Add(oneDay) 173 | if cur.After(end) { 174 | break 175 | } 176 | } 177 | return c.Find(bson.M{"name": "job-day", "date": bson.M{"$in": dateList}}).Sort("date").All(&ls) 178 | }) 179 | 180 | return 181 | } 182 | -------------------------------------------------------------------------------- /log/log.go: -------------------------------------------------------------------------------- 1 | package log 2 | 3 | var ( 4 | DefaultLogger Logger 5 | ) 6 | 7 | type Logger interface { 8 | Debugf(format string, v ...interface{}) 9 | Infof(format string, v ...interface{}) 10 | Warnf(format string, v ...interface{}) 11 | Errorf(format string, v ...interface{}) 12 | Fatalf(format string, v ...interface{}) 13 | } 14 | 15 | func SetLogger(l Logger) { 16 | DefaultLogger = l 17 | } 18 | 19 | func Debugf(format string, v ...interface{}) { 20 | if DefaultLogger != nil { 21 | DefaultLogger.Debugf(format, v...) 22 | } 23 | } 24 | 25 | func Infof(format string, v ...interface{}) { 26 | if DefaultLogger != nil { 27 | DefaultLogger.Infof(format, v...) 28 | } 29 | } 30 | 31 | func Warnf(format string, v ...interface{}) { 32 | if DefaultLogger != nil { 33 | DefaultLogger.Warnf(format, v...) 34 | } 35 | } 36 | 37 | func Errorf(format string, v ...interface{}) { 38 | if DefaultLogger != nil { 39 | DefaultLogger.Errorf(format, v...) 40 | } 41 | } 42 | 43 | func Fatalf(format string, v ...interface{}) { 44 | if DefaultLogger != nil { 45 | DefaultLogger.Fatalf(format, v...) 46 | } 47 | } 48 | -------------------------------------------------------------------------------- /mdb.go: -------------------------------------------------------------------------------- 1 | package cronsun 2 | 3 | import ( 4 | "github.com/shunfei/cronsun/db" 5 | ) 6 | 7 | var ( 8 | mgoDB *db.Mdb 9 | ) 10 | 11 | func GetDb() *db.Mdb { 12 | return mgoDB 13 | } 14 | -------------------------------------------------------------------------------- /node.go: -------------------------------------------------------------------------------- 1 | package cronsun 2 | 3 | import ( 4 | "encoding/json" 5 | "fmt" 6 | "os" 7 | "strconv" 8 | "syscall" 9 | "time" 10 | 11 | client "github.com/coreos/etcd/clientv3" 12 | "gopkg.in/mgo.v2" 13 | "gopkg.in/mgo.v2/bson" 14 | 15 | "github.com/shunfei/cronsun/conf" 16 | "github.com/shunfei/cronsun/log" 17 | ) 18 | 19 | const ( 20 | Coll_Node = "node" 21 | ) 22 | 23 | // 执行 cron cmd 的进程 24 | // 注册到 /cronsun/node/ 25 | type Node struct { 26 | ID string `bson:"_id" json:"id"` // machine id 27 | PID string `bson:"pid" json:"pid"` // 进程 pid 28 | PIDFile string `bson:"-" json:"-"` 29 | IP string `bson:"ip" json:"ip"` // node ip 30 | Hostname string `bson:"hostname" json:"hostname"` 31 | 32 | Version string `bson:"version" json:"version"` 33 | UpTime time.Time `bson:"up" json:"up"` // 启动时间 34 | DownTime time.Time `bson:"down" json:"down"` // 上次关闭时间 35 | 36 | Alived bool `bson:"alived" json:"alived"` // 是否可用 37 | Connected bool `bson:"-" json:"connected"` // 当 Alived 为 true 时有效，表示心跳是否正常 38 | } 39 | 40 | func (n *Node) String() string { 41 | return "node[" + n.ID + "] pid[" + n.PID + "]" 42 | } 43 | 44 | func (n *Node) Put(opts ...client.OpOption) (*client.PutResponse, error) { 45 | return DefalutClient.Put(conf.Config.Node+n.ID, n.PID, opts...) 46 | } 47 | 48 | func (n *Node) Del() (*client.DeleteResponse, error) { 49 | return DefalutClient.Delete(conf.Config.Node + n.ID) 50 | } 51 | 52 | // 判断 node 是否已注册到 etcd 53 | // 存在则返回进行 pid，不存在返回 -1 54 | func (n *Node) Exist() (pid int, err error) { 55 | resp, err := DefalutClient.Get(conf.Config.Node + n.ID) 56 | if err != nil { 57 | return 58 | } 59 | 60 | if len(resp.Kvs) == 0 { 61 | return -1, nil 62 | } 63 | 64 | if pid, err = strconv.Atoi(string(resp.Kvs[0].Value)); err != nil { 65 | if _, err = DefalutClient.Delete(conf.Config.Node + n.ID); err != nil { 66 | return 67 | } 68 | return -1, nil 69 | } 70 | 71 | p, err := os.FindProcess(pid) 72 | if err != nil { 73 | return -1, nil 74 | } 75 | 76 | // TODO: 暂时不考虑 linux/unix 以外的系统 77 | if p != nil && p.Signal(syscall.Signal(0)) == nil { 78 | return 79 | } 80 | 81 | return -1, nil 82 | } 83 | 84 | func GetNodes() (nodes []*Node, err error) { 85 | return GetNodesBy(nil) 86 | } 87 | 88 | func GetNodesBy(query interface{}) (nodes []*Node, err error) { 89 | err = mgoDB.WithC(Coll_Node, func(c *mgo.Collection) error { 90 | return c.Find(query).All(&nodes) 91 | }) 92 | 93 | return 94 | } 95 | 96 | func GetNodesByID(id string) (node *Node, err error) { 97 | err = mgoDB.FindId(Coll_Node, id, &node) 98 | return 99 | } 100 | 101 | func RemoveNode(query interface{}) error { 102 | return mgoDB.WithC(Coll_Node, func(c *mgo.Collection) error { 103 | return c.Remove(query) 104 | }) 105 | } 106 | 107 | func ISNodeAlive(id string) (bool, error) { 108 | n := 0 109 | err := mgoDB.WithC(Coll_Node, func(c *mgo.Collection) error { 110 | var e error 111 | n, e = c.Find(bson.M{"_id": id, "alived": true}).Count() 112 | return e 113 | }) 114 | 115 | return n > 0, err 116 | } 117 | 118 | func GetNodeGroups() (list []*Group, err error) { 119 | resp, err := DefalutClient.Get(conf.Config.Group, client.WithPrefix(), client.WithSort(client.SortByKey, client.SortAscend)) 120 | if err != nil { 121 | return 122 | } 123 | 124 | list = make([]*Group, 0, resp.Count) 125 | for i := range resp.Kvs { 126 | g := Group{} 127 | err = json.Unmarshal(resp.Kvs[i].Value, &g) 128 | if err != nil { 129 | err = fmt.Errorf("node.GetGroups(key: %s) error: %s", string(resp.Kvs[i].Key), err.Error()) 130 | return 131 | } 132 | list = append(list, &g) 133 | } 134 | 135 | return 136 | } 137 | 138 | func WatchNode() client.WatchChan { 139 | return DefalutClient.Watch(conf.Config.Node, client.WithPrefix()) 140 | } 141 | 142 | // On 结点实例启动后，在 mongoDB 中记录存活信息 143 | func (n *Node) On() { 144 | n.Alived, n.Version, n.UpTime = true, Version, time.Now() 145 | n.SyncToMgo() 146 | } 147 | 148 | // On 结点实例停用后，在 mongoDB 中去掉存活信息 149 | func (n *Node) Down() { 150 | n.Alived, n.DownTime = false, time.Now() 151 | n.SyncToMgo() 152 | } 153 | 154 | func (n *Node) SyncToMgo() { 155 | if err := mgoDB.Upsert(Coll_Node, bson.M{"_id": n.ID}, n); err != nil { 156 | log.Errorf(err.Error()) 157 | } 158 | } 159 | 160 | // RmOldInfo remove old version(< 0.3.0) node info 161 | func (n *Node) RmOldInfo() { 162 | RemoveNode(bson.M{"_id": n.IP}) 163 | DefalutClient.Delete(conf.Config.Node + n.IP) 164 | } 165 | -------------------------------------------------------------------------------- /node/cron/LICENSE: -------------------------------------------------------------------------------- 1 | Copyright (C) 2012 Rob Figueiredo 2 | All Rights Reserved. 3 | 4 | MIT LICENSE 5 | 6 | Permission is hereby granted, free of charge, to any person obtaining a copy of 7 | this software and associated documentation files (the "Software"), to deal in 8 | the Software without restriction, including without limitation the rights to 9 | use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of 10 | the Software, and to permit persons to whom the Software is furnished to do so, 11 | subject to the following conditions: 12 | 13 | The above copyright notice and this permission notice shall be included in all 14 | copies or substantial portions of the Software. 15 | 16 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 17 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS 18 | FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR 19 | COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER 20 | IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN 21 | CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 22 | -------------------------------------------------------------------------------- /node/cron/README.md: -------------------------------------------------------------------------------- 1 | [![GoDoc](http://godoc.org/github.com/robfig/cron?status.png)](http://godoc.org/github.com/robfig/cron) 2 | [![Build Status](https://travis-ci.org/robfig/cron.svg?branch=master)](https://travis-ci.org/robfig/cron) 3 | -------------------------------------------------------------------------------- /node/cron/at.go: -------------------------------------------------------------------------------- 1 | package cron 2 | 3 | import ( 4 | "sort" 5 | "time" 6 | ) 7 | 8 | // TimeListSchedule will run at the specify giving time. 9 | type TimeListSchedule struct { 10 | timeList []time.Time 11 | } 12 | 13 | // At returns a crontab Schedule that activates every specify time. 14 | func At(tl []time.Time) *TimeListSchedule { 15 | sort.Slice(tl, func(i, j int) bool { return tl[i].Unix() < tl[j].Unix() }) 16 | return &TimeListSchedule{ 17 | timeList: tl, 18 | } 19 | } 20 | 21 | // Next returns the next time this should be run. 22 | // This rounds so that the next activation time will be on the second. 23 | func (schedule *TimeListSchedule) Next(t time.Time) time.Time { 24 | cur := 0 25 | for cur < len(schedule.timeList) { 26 | nextt := schedule.timeList[cur] 27 | cur++ 28 | if nextt.UnixNano() <= t.UnixNano() { 29 | continue 30 | } 31 | return nextt 32 | } 33 | return time.Time{} 34 | } 35 | -------------------------------------------------------------------------------- /node/cron/at_test.go: -------------------------------------------------------------------------------- 1 | package cron 2 | 3 | import ( 4 | "testing" 5 | "time" 6 | ) 7 | 8 | func TestTimeListNext(t *testing.T) { 9 | tests := []struct { 10 | startTime string 11 | times []string 12 | expected []string 13 | }{ 14 | // Simple cases 15 | { 16 | "2018-09-01 08:01:02", 17 | []string{"2018-09-01 10:01:02"}, 18 | []string{"2018-09-01 10:01:02"}, 19 | }, 20 | 21 | // sort list 22 | { 23 | "2018-09-01 08:01:02", 24 | []string{"2018-09-01 10:01:02", "2018-09-02 10:01:02"}, 25 | []string{"2018-09-01 10:01:02", "2018-09-02 10:01:02"}, 26 | }, 27 | 28 | // sort list with middle start time 29 | { 30 | "2018-09-01 10:11:02", 31 | []string{"2018-09-01 10:01:02", "2018-09-02 10:01:02"}, 32 | []string{"2018-09-02 10:01:02"}, 33 | }, 34 | 35 | // unsorted list 36 | { 37 | "2018-07-01 08:01:02", 38 | []string{"2018-09-01 10:01:00", "2018-08-01 10:00:00", "2018-09-01 10:00:00", "2018-08-02 10:01:02"}, 39 | []string{"2018-08-01 10:00:00", "2018-08-02 10:01:02", "2018-09-01 10:00:00", "2018-09-01 10:01:00"}, 40 | }, 41 | 42 | // unsorted list with middle start time 43 | { 44 | "2018-08-03 12:00:00", 45 | []string{"2018-09-01 10:01:00", "2018-08-01 10:00:00", "2018-09-01 10:00:00", "2018-08-02 10:01:02"}, 46 | []string{"2018-09-01 10:00:00", "2018-09-01 10:01:00"}, 47 | }, 48 | } 49 | 50 | for _, c := range tests { 51 | tls := At(getAtTimes(c.times)) 52 | nextTime := getAtTime(c.startTime) 53 | for _, trun := range c.expected { 54 | actual := tls.Next(nextTime) 55 | expected := getAtTime(trun) 56 | if actual != expected { 57 | t.Errorf("%s, \"%s\": (expected) %v != %v (actual)", 58 | c.startTime, c.times, expected, actual) 59 | } 60 | nextTime = actual 61 | } 62 | if actual := tls.Next(nextTime); !actual.IsZero() { 63 | t.Errorf("%s, \"%s\": next time should be zero, but got %v (actual)", 64 | c.startTime, c.times, actual) 65 | } 66 | 67 | } 68 | } 69 | 70 | func getAtTime(value string) time.Time { 71 | if value == "" { 72 | panic("time string is empty") 73 | } 74 | 75 | t, err := time.Parse("2006-01-02 15:04:05", value) 76 | if err != nil { 77 | panic(err) 78 | } 79 | 80 | return t 81 | } 82 | 83 | func getAtTimes(values []string) []time.Time { 84 | tl := []time.Time{} 85 | for _, v := range values { 86 | tl = append(tl, getAtTime(v)) 87 | } 88 | return tl 89 | } 90 | -------------------------------------------------------------------------------- /node/cron/constantdelay.go: -------------------------------------------------------------------------------- 1 | package cron 2 | 3 | import "time" 4 | 5 | // ConstantDelaySchedule represents a simple recurring duty cycle, e.g. "Every 5 minutes". 6 | // It does not support jobs more frequent than once a second. 7 | type ConstantDelaySchedule struct { 8 | Delay time.Duration 9 | } 10 | 11 | // Every returns a crontab Schedule that activates once every duration. 12 | // Delays of less than a second are not supported (will round up to 1 second). 13 | // Any fields less than a Second are truncated. 14 | func Every(duration time.Duration) ConstantDelaySchedule { 15 | if duration < time.Second { 16 | duration = time.Second 17 | } 18 | return ConstantDelaySchedule{ 19 | Delay: duration - time.Duration(duration.Nanoseconds())%time.Second, 20 | } 21 | } 22 | 23 | // Next returns the next time this should be run. 24 | // This rounds so that the next activation time will be on the second. 25 | func (schedule ConstantDelaySchedule) Next(t time.Time) time.Time { 26 | return t.Add(schedule.Delay - time.Duration(t.Nanosecond())*time.Nanosecond) 27 | } 28 | -------------------------------------------------------------------------------- /node/cron/constantdelay_test.go: -------------------------------------------------------------------------------- 1 | package cron 2 | 3 | import ( 4 | "testing" 5 | "time" 6 | ) 7 | 8 | func TestConstantDelayNext(t *testing.T) { 9 | tests := []struct { 10 | time string 11 | delay time.Duration 12 | expected string 13 | }{ 14 | // Simple cases 15 | {"Mon Jul 9 14:45 2012", 15*time.Minute + 50*time.Nanosecond, "Mon Jul 9 15:00 2012"}, 16 | {"Mon Jul 9 14:59 2012", 15 * time.Minute, "Mon Jul 9 15:14 2012"}, 17 | {"Mon Jul 9 14:59:59 2012", 15 * time.Minute, "Mon Jul 9 15:14:59 2012"}, 18 | 19 | // Wrap around hours 20 | {"Mon Jul 9 15:45 2012", 35 * time.Minute, "Mon Jul 9 16:20 2012"}, 21 | 22 | // Wrap around days 23 | {"Mon Jul 9 23:46 2012", 14 * time.Minute, "Tue Jul 10 00:00 2012"}, 24 | {"Mon Jul 9 23:45 2012", 35 * time.Minute, "Tue Jul 10 00:20 2012"}, 25 | {"Mon Jul 9 23:35:51 2012", 44*time.Minute + 24*time.Second, "Tue Jul 10 00:20:15 2012"}, 26 | {"Mon Jul 9 23:35:51 2012", 25*time.Hour + 44*time.Minute + 24*time.Second, "Thu Jul 11 01:20:15 2012"}, 27 | 28 | // Wrap around months 29 | {"Mon Jul 9 23:35 2012", 91*24*time.Hour + 25*time.Minute, "Thu Oct 9 00:00 2012"}, 30 | 31 | // Wrap around minute, hour, day, month, and year 32 | {"Mon Dec 31 23:59:45 2012", 15 * time.Second, "Tue Jan 1 00:00:00 2013"}, 33 | 34 | // Round to nearest second on the delay 35 | {"Mon Jul 9 14:45 2012", 15*time.Minute + 50*time.Nanosecond, "Mon Jul 9 15:00 2012"}, 36 | 37 | // Round up to 1 second if the duration is less. 38 | {"Mon Jul 9 14:45:00 2012", 15 * time.Millisecond, "Mon Jul 9 14:45:01 2012"}, 39 | 40 | // Round to nearest second when calculating the next time. 41 | {"Mon Jul 9 14:45:00.005 2012", 15 * time.Minute, "Mon Jul 9 15:00 2012"}, 42 | 43 | // Round to nearest second for both. 44 | {"Mon Jul 9 14:45:00.005 2012", 15*time.Minute + 50*time.Nanosecond, "Mon Jul 9 15:00 2012"}, 45 | } 46 | 47 | for _, c := range tests { 48 | actual := Every(c.delay).Next(getTime(c.time)) 49 | expected := getTime(c.expected) 50 | if actual != expected { 51 | t.Errorf("%s, \"%s\": (expected) %v != %v (actual)", c.time, c.delay, expected, actual) 52 | } 53 | } 54 | } 55 | -------------------------------------------------------------------------------- /node/cron/doc.go: -------------------------------------------------------------------------------- 1 | /* 2 | Package cron implements a cron spec parser and job runner. 3 | 4 | Usage 5 | 6 | Callers may register Funcs to be invoked on a given schedule. Cron will run 7 | them in their own goroutines. 8 | 9 | c := cron.New() 10 | c.AddFunc("0 30 * * * *", func() { fmt.Println("Every hour on the half hour") }) 11 | c.AddFunc("@hourly", func() { fmt.Println("Every hour") }) 12 | c.AddFunc("@every 1h30m", func() { fmt.Println("Every hour thirty") }) 13 | c.Start() 14 | .. 15 | // Funcs are invoked in their own goroutine, asynchronously. 16 | ... 17 | // Funcs may also be added to a running Cron 18 | c.AddFunc("@daily", func() { fmt.Println("Every day") }) 19 | .. 20 | // Inspect the cron job entries' next and previous run times. 21 | inspect(c.Entries()) 22 | .. 23 | c.Stop() // Stop the scheduler (does not stop any jobs already running). 24 | 25 | CRON Expression Format 26 | 27 | A cron expression represents a set of times, using 6 space-separated fields. 28 | 29 | Field name | Mandatory? | Allowed values | Allowed special characters 30 | ---------- | ---------- | -------------- | -------------------------- 31 | Seconds | Yes | 0-59 | * / , - 32 | Minutes | Yes | 0-59 | * / , - 33 | Hours | Yes | 0-23 | * / , - 34 | Day of month | Yes | 1-31 | * / , - ? 35 | Month | Yes | 1-12 or JAN-DEC | * / , - 36 | Day of week | Yes | 0-6 or SUN-SAT | * / , - ? 37 | 38 | Note: Month and Day-of-week field values are case insensitive. "SUN", "Sun", 39 | and "sun" are equally accepted. 40 | 41 | Special Characters 42 | 43 | Asterisk ( * ) 44 | 45 | The asterisk indicates that the cron expression will match for all values of the 46 | field; e.g., using an asterisk in the 5th field (month) would indicate every 47 | month. 48 | 49 | Slash ( / ) 50 | 51 | Slashes are used to describe increments of ranges. For example 3-59/15 in the 52 | 1st field (minutes) would indicate the 3rd minute of the hour and every 15 53 | minutes thereafter. The form "*\/..." is equivalent to the form "first-last/...", 54 | that is, an increment over the largest possible range of the field. The form 55 | "N/..." is accepted as meaning "N-MAX/...", that is, starting at N, use the 56 | increment until the end of that specific range. It does not wrap around. 57 | 58 | Comma ( , ) 59 | 60 | Commas are used to separate items of a list. For example, using "MON,WED,FRI" in 61 | the 5th field (day of week) would mean Mondays, Wednesdays and Fridays. 62 | 63 | Hyphen ( - ) 64 | 65 | Hyphens are used to define ranges. For example, 9-17 would indicate every 66 | hour between 9am and 5pm inclusive. 67 | 68 | Question mark ( ? ) 69 | 70 | Question mark may be used instead of '*' for leaving either day-of-month or 71 | day-of-week blank. 72 | 73 | Predefined schedules 74 | 75 | You may use one of several pre-defined schedules in place of a cron expression. 76 | 77 | Entry | Description | Equivalent To 78 | ----- | ----------- | ------------- 79 | @yearly (or @annually) | Run once a year, midnight, Jan. 1st | 0 0 0 1 1 * 80 | @monthly | Run once a month, midnight, first of month | 0 0 0 1 * * 81 | @weekly | Run once a week, midnight on Sunday | 0 0 0 * * 0 82 | @daily (or @midnight) | Run once a day, midnight | 0 0 0 * * * 83 | @hourly | Run once an hour, beginning of hour | 0 0 * * * * 84 | 85 | Intervals 86 | 87 | You may also schedule a job to execute at fixed intervals. This is supported by 88 | formatting the cron spec like this: 89 | 90 | @every 91 | 92 | where "duration" is a string accepted by time.ParseDuration 93 | (http://golang.org/pkg/time/#ParseDuration). 94 | 95 | For example, "@every 1h30m10s" would indicate a schedule that activates every 96 | 1 hour, 30 minutes, 10 seconds. 97 | 98 | Note: The interval does not take the job runtime into account. For example, 99 | if a job takes 3 minutes to run, and it is scheduled to run every 5 minutes, 100 | it will have only 2 minutes of idle time between each run. 101 | 102 | Time zones 103 | 104 | All interpretation and scheduling is done in the machine's local time zone (as 105 | provided by the Go time package (http://www.golang.org/pkg/time). 106 | 107 | Be aware that jobs scheduled during daylight-savings leap-ahead transitions will 108 | not be run! 109 | 110 | Thread safety 111 | 112 | Since the Cron service runs concurrently with the calling code, some amount of 113 | care must be taken to ensure proper synchronization. 114 | 115 | All cron methods are designed to be correctly synchronized as long as the caller 116 | ensures that invocations have a clear happens-before ordering between them. 117 | 118 | Implementation 119 | 120 | Cron entries are stored in an array, sorted by their next activation time. Cron 121 | sleeps until the next job is due to be run. 122 | 123 | Upon waking: 124 | - it runs each entry that is active on that second 125 | - it calculates the next run times for the jobs that were run 126 | - it re-sorts the array of entries by next activation time. 127 | - it goes to sleep until the soonest job. 128 | */ 129 | package cron 130 | -------------------------------------------------------------------------------- /node/cron/spec.go: -------------------------------------------------------------------------------- 1 | package cron 2 | 3 | import "time" 4 | 5 | // SpecSchedule specifies a duty cycle (to the second granularity), based on a 6 | // traditional crontab specification. It is computed initially and stored as bit sets. 7 | type SpecSchedule struct { 8 | Second, Minute, Hour, Dom, Month, Dow uint64 9 | } 10 | 11 | // bounds provides a range of acceptable values (plus a map of name to value). 12 | type bounds struct { 13 | min, max uint 14 | names map[string]uint 15 | } 16 | 17 | // The bounds for each field. 18 | var ( 19 | seconds = bounds{0, 59, nil} 20 | minutes = bounds{0, 59, nil} 21 | hours = bounds{0, 23, nil} 22 | dom = bounds{1, 31, nil} 23 | months = bounds{1, 12, map[string]uint{ 24 | "jan": 1, 25 | "feb": 2, 26 | "mar": 3, 27 | "apr": 4, 28 | "may": 5, 29 | "jun": 6, 30 | "jul": 7, 31 | "aug": 8, 32 | "sep": 9, 33 | "oct": 10, 34 | "nov": 11, 35 | "dec": 12, 36 | }} 37 | dow = bounds{0, 6, map[string]uint{ 38 | "sun": 0, 39 | "mon": 1, 40 | "tue": 2, 41 | "wed": 3, 42 | "thu": 4, 43 | "fri": 5, 44 | "sat": 6, 45 | }} 46 | ) 47 | 48 | const ( 49 | // Set the top bit if a star was included in the expression. 50 | starBit = 1 << 63 51 | ) 52 | 53 | // Next returns the next time this schedule is activated, greater than the given 54 | // time. If no time can be found to satisfy the schedule, return the zero time. 55 | func (s *SpecSchedule) Next(t time.Time) time.Time { 56 | // General approach: 57 | // For Month, Day, Hour, Minute, Second: 58 | // Check if the time value matches. If yes, continue to the next field. 59 | // If the field doesn't match the schedule, then increment the field until it matches. 60 | // While incrementing the field, a wrap-around brings it back to the beginning 61 | // of the field list (since it is necessary to re-verify previous field 62 | // values) 63 | 64 | // Start at the earliest possible time (the upcoming second). 65 | t = t.Add(1*time.Second - time.Duration(t.Nanosecond())*time.Nanosecond) 66 | 67 | // This flag indicates whether a field has been incremented. 68 | added := false 69 | 70 | // If no time is found within five years, return zero. 71 | yearLimit := t.Year() + 5 72 | 73 | WRAP: 74 | if t.Year() > yearLimit { 75 | return time.Time{} 76 | } 77 | 78 | // Find the first applicable month. 79 | // If it's this month, then do nothing. 80 | for 1< 0 152 | dowMatch bool = 1< 0 153 | ) 154 | if s.Dom&starBit > 0 || s.Dow&starBit > 0 { 155 | return domMatch && dowMatch 156 | } 157 | return domMatch || dowMatch 158 | } 159 | -------------------------------------------------------------------------------- /node/csctl.go: -------------------------------------------------------------------------------- 1 | package node 2 | 3 | import ( 4 | "encoding/json" 5 | 6 | "github.com/shunfei/cronsun" 7 | "github.com/shunfei/cronsun/log" 8 | ) 9 | 10 | func (n *Node) executCsctlCmd(key, value []byte) error { 11 | cmd := &cronsun.CsctlCmd{} 12 | err := json.Unmarshal(value, cmd) 13 | if err != nil { 14 | log.Warnf("invalid csctl command[%s] value[%s], err: %s", string(key), string(value), err.Error()) 15 | return err 16 | } 17 | 18 | if cronsun.NodeCmdMax <= cmd.Cmd || cmd.Cmd <= cronsun.NodeCmdUnknown { 19 | log.Warnf("invalid csctl command[%s] value[%s], err: %s", string(key), string(value)) 20 | return cronsun.InvalidNodeCmdErr 21 | } 22 | 23 | switch cmd.Cmd { 24 | case cronsun.NodeCmdRmOld: 25 | n.Node.RmOldInfo() 26 | case cronsun.NodeCmdSync: 27 | n.Node.SyncToMgo() 28 | } 29 | 30 | log.Infof("%s execute csctl command[%s] success", n.String(), cmd.Cmd.String()) 31 | return nil 32 | } 33 | -------------------------------------------------------------------------------- /node/group.go: -------------------------------------------------------------------------------- 1 | package node 2 | 3 | import ( 4 | "github.com/shunfei/cronsun" 5 | ) 6 | 7 | type Groups map[string]*cronsun.Group 8 | 9 | type jobLink struct { 10 | gname string 11 | // rule id 12 | rules map[string]bool 13 | } 14 | 15 | // map[group id]map[job id]*jobLink 16 | // 用于 group 发生变化的时候修改相应的 job 17 | type link map[string]map[string]*jobLink 18 | 19 | func newLink(size int) link { 20 | return make(link, size) 21 | } 22 | 23 | func (l link) add(gid, jid, rid, gname string) { 24 | js, ok := l[gid] 25 | if !ok { 26 | js = make(map[string]*jobLink, 4) 27 | l[gid] = js 28 | } 29 | 30 | j, ok := js[jid] 31 | if !ok { 32 | j = &jobLink{ 33 | gname: gname, 34 | rules: make(map[string]bool), 35 | } 36 | js[jid] = j 37 | } 38 | 39 | j.rules[rid] = true 40 | } 41 | 42 | func (l link) addJob(job *cronsun.Job) { 43 | for _, r := range job.Rules { 44 | for _, gid := range r.GroupIDs { 45 | l.add(gid, job.ID, r.ID, job.Group) 46 | } 47 | } 48 | } 49 | 50 | func (l link) del(gid, jid, rid string) { 51 | js, ok := l[gid] 52 | if !ok { 53 | return 54 | } 55 | 56 | j, ok := js[jid] 57 | if !ok { 58 | return 59 | } 60 | 61 | delete(j.rules, rid) 62 | if len(j.rules) == 0 { 63 | delete(js, jid) 64 | } 65 | } 66 | 67 | func (l link) delJob(job *cronsun.Job) { 68 | for _, r := range job.Rules { 69 | for _, gid := range r.GroupIDs { 70 | l.delGroupJob(gid, job.ID) 71 | } 72 | } 73 | } 74 | 75 | func (l link) delGroupJob(gid, jid string) { 76 | js, ok := l[gid] 77 | if !ok { 78 | return 79 | } 80 | 81 | delete(js, jid) 82 | } 83 | 84 | func (l link) delGroup(gid string) { 85 | delete(l, gid) 86 | } 87 | -------------------------------------------------------------------------------- /node/job.go: -------------------------------------------------------------------------------- 1 | package node 2 | 3 | import ( 4 | "github.com/shunfei/cronsun" 5 | ) 6 | 7 | type Jobs map[string]*cronsun.Job 8 | -------------------------------------------------------------------------------- /noticer.go: -------------------------------------------------------------------------------- 1 | package cronsun 2 | 3 | import ( 4 | "bytes" 5 | "encoding/json" 6 | "fmt" 7 | "io/ioutil" 8 | "net/http" 9 | "time" 10 | 11 | client "github.com/coreos/etcd/clientv3" 12 | "github.com/go-gomail/gomail" 13 | 14 | "github.com/shunfei/cronsun/conf" 15 | "github.com/shunfei/cronsun/log" 16 | ) 17 | 18 | type Noticer interface { 19 | Serve() 20 | Send(*Message) 21 | } 22 | 23 | type Message struct { 24 | Subject string 25 | Body string 26 | To []string 27 | } 28 | 29 | type Mail struct { 30 | cf *conf.MailConf 31 | open bool 32 | sc gomail.SendCloser 33 | timer *time.Timer 34 | msgChan chan *Message 35 | } 36 | 37 | func NewMail(timeout time.Duration) (m *Mail, err error) { 38 | var ( 39 | sc gomail.SendCloser 40 | done = make(chan struct{}) 41 | cf = conf.Config.Mail 42 | ) 43 | 44 | // qq 邮箱的 Auth 出错后， 501 命令超时 2min 才能退出 45 | go func() { 46 | sc, err = cf.Dialer.Dial() 47 | close(done) 48 | }() 49 | 50 | select { 51 | case <-done: 52 | case <-time.After(timeout): 53 | err = fmt.Errorf("connect to smtp timeout") 54 | } 55 | 56 | if err != nil { 57 | return 58 | } 59 | 60 | m = &Mail{ 61 | cf: cf, 62 | open: true, 63 | sc: sc, 64 | timer: time.NewTimer(time.Duration(cf.Keepalive) * time.Second), 65 | msgChan: make(chan *Message, 8), 66 | } 67 | return 68 | } 69 | 70 | func (m *Mail) Serve() { 71 | var err error 72 | sm := gomail.NewMessage() 73 | for { 74 | select { 75 | case msg := <-m.msgChan: 76 | m.timer.Reset(time.Duration(m.cf.Keepalive) * time.Second) 77 | if !m.open { 78 | if m.sc, err = m.cf.Dialer.Dial(); err != nil { 79 | log.Warnf("smtp send msg[%+v] err: %s", msg, err.Error()) 80 | continue 81 | } 82 | m.open = true 83 | } 84 | 85 | sm.Reset() 86 | sm.SetHeader("From", m.cf.Username) 87 | sm.SetHeader("To", msg.To...) 88 | sm.SetHeader("Subject", msg.Subject) 89 | sm.SetBody("text/plain", msg.Body) 90 | if err := gomail.Send(m.sc, sm); err != nil { 91 | log.Warnf("smtp send msg[%+v] err: %s", msg, err.Error()) 92 | } 93 | case <-m.timer.C: 94 | if m.open { 95 | if err = m.sc.Close(); err != nil { 96 | log.Warnf("close smtp server err: %s", err.Error()) 97 | } 98 | m.open = false 99 | } 100 | m.timer.Reset(time.Duration(m.cf.Keepalive) * time.Second) 101 | } 102 | } 103 | } 104 | 105 | func (m *Mail) Send(msg *Message) { 106 | m.msgChan <- msg 107 | } 108 | 109 | type HttpAPI struct{} 110 | 111 | func (h *HttpAPI) Serve() {} 112 | 113 | func (h *HttpAPI) Send(msg *Message) { 114 | body, err := json.Marshal(msg) 115 | if err != nil { 116 | log.Warnf("http api send msg[%+v] err: %s", msg, err.Error()) 117 | return 118 | } 119 | 120 | req, err := http.NewRequest("POST", conf.Config.Mail.HttpAPI, bytes.NewBuffer(body)) 121 | if err != nil { 122 | return 123 | } 124 | 125 | req.Header.Set("Content-Type", "application/json") 126 | resp, err := http.DefaultClient.Do(req) 127 | if err != nil { 128 | log.Warnf("http api send msg[%+v] err: %s", msg, err.Error()) 129 | return 130 | } 131 | defer resp.Body.Close() 132 | 133 | if resp.StatusCode == 200 { 134 | return 135 | } 136 | 137 | data, err := ioutil.ReadAll(resp.Body) 138 | if err != nil { 139 | log.Warnf("http api send msg[%+v] err: %s", msg, err.Error()) 140 | return 141 | } 142 | log.Warnf("http api send msg[%+v] err: %s", msg, string(data)) 143 | return 144 | } 145 | 146 | func StartNoticer(n Noticer) { 147 | go n.Serve() 148 | go monitorNodes(n) 149 | 150 | rch := DefalutClient.Watch(conf.Config.Noticer, client.WithPrefix()) 151 | var err error 152 | for wresp := range rch { 153 | for _, ev := range wresp.Events { 154 | switch { 155 | case ev.IsCreate(), ev.IsModify(): 156 | msg := new(Message) 157 | if err = json.Unmarshal(ev.Kv.Value, msg); err != nil { 158 | log.Warnf("msg[%s] umarshal err: %s", string(ev.Kv.Value), err.Error()) 159 | continue 160 | } 161 | 162 | if len(conf.Config.Mail.To) > 0 { 163 | msg.To = append(msg.To, conf.Config.Mail.To...) 164 | } 165 | n.Send(msg) 166 | } 167 | } 168 | } 169 | } 170 | 171 | func monitorNodes(n Noticer) { 172 | rch := WatchNode() 173 | 174 | for wresp := range rch { 175 | for _, ev := range wresp.Events { 176 | switch { 177 | case ev.Type == client.EventTypeDelete: 178 | id := GetIDFromKey(string(ev.Kv.Key)) 179 | log.Errorf("cronnode DELETE event detected, node UUID: %s", id) 180 | 181 | node, err := GetNodesByID(id) 182 | if err != nil { 183 | log.Warnf("failed to fetch node[%s] from mongodb: %s", id, err.Error()) 184 | continue 185 | } 186 | 187 | if node.Alived { 188 | n.Send(&Message{ 189 | Subject: fmt.Sprintf("[Cronsun Warning] Node[%s] break away cluster at %s", 190 | node.Hostname, time.Now().Format(time.RFC3339)), 191 | Body: fmt.Sprintf("Cronsun Node breaked away cluster, this might happened when node crash or network problems.\nUUID: %s\nHostname: %s\nIP: %s\n", id, node.Hostname, node.IP), 192 | To: conf.Config.Mail.To, 193 | }) 194 | } 195 | } 196 | } 197 | } 198 | } 199 | -------------------------------------------------------------------------------- /once.go: -------------------------------------------------------------------------------- 1 | package cronsun 2 | 3 | import ( 4 | client "github.com/coreos/etcd/clientv3" 5 | 6 | "github.com/shunfei/cronsun/conf" 7 | ) 8 | 9 | // 马上执行 job 任务 10 | // 注册到 /cronsun/once/group/ 11 | // value 12 | // 若执行单个结点，则值为 NodeID 13 | // 若 job 所在的结点都需执行，则值为空 "" 14 | func PutOnce(group, jobID, nodeID string) error { 15 | _, err := DefalutClient.Put(conf.Config.Once+group+"/"+jobID, nodeID) 16 | return err 17 | } 18 | 19 | func WatchOnce() client.WatchChan { 20 | return DefalutClient.Watch(conf.Config.Once, client.WithPrefix()) 21 | } 22 | -------------------------------------------------------------------------------- /proc.go: -------------------------------------------------------------------------------- 1 | package cronsun 2 | 3 | import ( 4 | "encoding/json" 5 | "fmt" 6 | "strings" 7 | "sync" 8 | "sync/atomic" 9 | "time" 10 | 11 | client "github.com/coreos/etcd/clientv3" 12 | 13 | "github.com/shunfei/cronsun/conf" 14 | "github.com/shunfei/cronsun/log" 15 | ) 16 | 17 | var ( 18 | lID *leaseID 19 | ) 20 | 21 | // 维持 lease id 服务 22 | func StartProc() error { 23 | lID = &leaseID{ 24 | ttl: conf.Config.ProcTtl, 25 | lk: new(sync.RWMutex), 26 | done: make(chan struct{}), 27 | } 28 | 29 | if lID.ttl == 0 { 30 | return nil 31 | } 32 | 33 | err := lID.set() 34 | go lID.keepAlive() 35 | return err 36 | } 37 | 38 | func Reload(i interface{}) { 39 | if lID.ttl == conf.Config.ProcTtl { 40 | return 41 | } 42 | 43 | close(lID.done) 44 | lID.done, lID.ttl = make(chan struct{}), conf.Config.ProcTtl 45 | if conf.Config.ProcTtl == 0 { 46 | return 47 | } 48 | 49 | if err := lID.set(); err != nil { 50 | log.Warnf("proc lease id set err: %s", err.Error()) 51 | } 52 | go lID.keepAlive() 53 | } 54 | 55 | func Exit(i interface{}) { 56 | if lID.done != nil { 57 | close(lID.done) 58 | } 59 | } 60 | 61 | type leaseID struct { 62 | ttl int64 63 | ID client.LeaseID 64 | lk *sync.RWMutex 65 | 66 | done chan struct{} 67 | } 68 | 69 | func (l *leaseID) get() client.LeaseID { 70 | if l.ttl == 0 { 71 | return -1 72 | } 73 | 74 | l.lk.RLock() 75 | id := l.ID 76 | l.lk.RUnlock() 77 | return id 78 | } 79 | 80 | func (l *leaseID) set() error { 81 | id := client.LeaseID(-1) 82 | resp, err := DefalutClient.Grant(l.ttl + 2) 83 | if err == nil { 84 | id = resp.ID 85 | } 86 | 87 | l.lk.Lock() 88 | l.ID = id 89 | l.lk.Unlock() 90 | return err 91 | } 92 | 93 | func (l *leaseID) keepAlive() { 94 | duration := time.Duration(l.ttl) * time.Second 95 | timer := time.NewTimer(duration) 96 | for { 97 | select { 98 | case <-l.done: 99 | return 100 | case <-timer.C: 101 | if l.ttl == 0 { 102 | return 103 | } 104 | 105 | id := l.get() 106 | if id > 0 { 107 | _, err := DefalutClient.KeepAliveOnce(l.ID) 108 | if err == nil { 109 | timer.Reset(duration) 110 | continue 111 | } 112 | 113 | log.Warnf("proc lease id[%x] keepAlive err: %s, try to reset...", id, err.Error()) 114 | } 115 | 116 | if err := l.set(); err != nil { 117 | log.Warnf("proc lease id set err: %s, try to reset after %d seconds...", err.Error(), l.ttl) 118 | } else { 119 | log.Infof("proc set lease id[%x] success", l.get()) 120 | } 121 | timer.Reset(duration) 122 | } 123 | } 124 | } 125 | 126 | // 当前执行中的任务信息 127 | // key: /cronsun/proc/node/group/jobId/pid 128 | // value: 开始执行时间 129 | // key 会自动过期，防止进程意外退出后没有清除相关 key，过期时间可配置 130 | type Process struct { 131 | // parse from key path 132 | ID string `json:"id"` // pid 133 | JobID string `json:"jobId"` 134 | Group string `json:"group"` 135 | NodeID string `json:"nodeId"` 136 | // parse from value 137 | ProcessVal 138 | 139 | running int32 140 | hasPut int32 141 | wg sync.WaitGroup 142 | done chan struct{} 143 | } 144 | 145 | type ProcessVal struct { 146 | Time time.Time `json:"time"` // 开始执行时间 147 | Killed bool `json:"killed"` // 是否强制杀死 148 | } 149 | 150 | func GetProcFromKey(key string) (proc *Process, err error) { 151 | ss := strings.Split(key, "/") 152 | var sslen = len(ss) 153 | if sslen < 5 { 154 | err = fmt.Errorf("invalid proc key [%s]", key) 155 | return 156 | } 157 | 158 | proc = &Process{ 159 | ID: ss[sslen-1], 160 | JobID: ss[sslen-2], 161 | Group: ss[sslen-3], 162 | NodeID: ss[sslen-4], 163 | } 164 | return 165 | } 166 | 167 | func (p *Process) Key() string { 168 | return conf.Config.Proc + p.NodeID + "/" + p.Group + "/" + p.JobID + "/" + p.ID 169 | } 170 | 171 | func (p *Process) Val() (string, error) { 172 | b, err := json.Marshal(&p.ProcessVal) 173 | if err != nil { 174 | return "", err 175 | } 176 | 177 | return string(b), nil 178 | } 179 | 180 | // 获取节点正在执行任务的数量 181 | func (j *Job) CountRunning() (int64, error) { 182 | resp, err := DefalutClient.Get(conf.Config.Proc+j.runOn+"/"+j.Group+"/"+j.ID, client.WithPrefix(), client.WithCountOnly()) 183 | if err != nil { 184 | return 0, err 185 | } 186 | 187 | return resp.Count, nil 188 | } 189 | 190 | // put 出错也进行 del 操作 191 | // 有可能某种原因，put 命令已经发送到 etcd server 192 | // 目前已知的 deadline 会出现此情况 193 | func (p *Process) put() (err error) { 194 | if atomic.LoadInt32(&p.running) != 1 { 195 | return 196 | } 197 | 198 | if !atomic.CompareAndSwapInt32(&p.hasPut, 0, 1) { 199 | return 200 | } 201 | 202 | id := lID.get() 203 | val, err := p.Val() 204 | if err != nil { 205 | return err 206 | } 207 | if id < 0 { 208 | if _, err = DefalutClient.Put(p.Key(), val); err != nil { 209 | return 210 | } 211 | } 212 | 213 | _, err = DefalutClient.Put(p.Key(), val, client.WithLease(id)) 214 | return 215 | } 216 | 217 | func (p *Process) del() error { 218 | if atomic.LoadInt32(&p.hasPut) != 1 { 219 | return nil 220 | } 221 | 222 | _, err := DefalutClient.Delete(p.Key()) 223 | return err 224 | } 225 | 226 | func (p *Process) Start() { 227 | if p == nil { 228 | return 229 | } 230 | 231 | if !atomic.CompareAndSwapInt32(&p.running, 0, 1) { 232 | return 233 | } 234 | 235 | if conf.Config.ProcReq == 0 { 236 | if err := p.put(); err != nil { 237 | log.Warnf("proc put[%s] err: %s", p.Key(), err.Error()) 238 | } 239 | return 240 | } 241 | 242 | p.done = make(chan struct{}) 243 | p.wg.Add(1) 244 | go func() { 245 | select { 246 | case <-p.done: 247 | case <-time.After(time.Duration(conf.Config.ProcReq) * time.Second): 248 | if err := p.put(); err != nil { 249 | log.Warnf("proc put[%s] err: %s", p.Key(), err.Error()) 250 | } 251 | } 252 | p.wg.Done() 253 | }() 254 | } 255 | 256 | func (p *Process) Stop() { 257 | if p == nil { 258 | return 259 | } 260 | 261 | if !atomic.CompareAndSwapInt32(&p.running, 1, 0) { 262 | return 263 | } 264 | 265 | if p.done != nil { 266 | close(p.done) 267 | } 268 | p.wg.Wait() 269 | 270 | if err := p.del(); err != nil { 271 | log.Warnf("proc del[%s] err: %s", p.Key(), err.Error()) 272 | } 273 | } 274 | 275 | func WatchProcs(nid string) client.WatchChan { 276 | return DefalutClient.Watch(conf.Config.Proc+nid, client.WithPrefix()) 277 | } 278 | -------------------------------------------------------------------------------- /release.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | 3 | version=v0.1 4 | if [[ $# -gt 0 ]]; then 5 | version="$1" 6 | fi 7 | 8 | 9 | declare -a goos=( 10 | linux 11 | darwin 12 | ) 13 | 14 | for os in "${goos[@]}"; do 15 | export GOOS=$os GOARCH=amd64 16 | echo building $GOOS-$GOARCH 17 | sh build.sh 18 | mv dist cronsun-$version 19 | 7z a cronsun-$version-$GOOS-$GOARCH.zip cronsun-$version 20 | rm -rf cronsun-$version 21 | echo 22 | done 23 | -------------------------------------------------------------------------------- /utils/argument_parser.go: -------------------------------------------------------------------------------- 1 | package utils 2 | 3 | import ( 4 | "errors" 5 | ) 6 | 7 | type fmsState int 8 | 9 | const ( 10 | stateArgumentOutside fmsState = iota 11 | stateArgumentStart 12 | stateArgumentEnd 13 | ) 14 | 15 | var errEndOfLine = errors.New("End of line") 16 | 17 | type cmdArgumentParser struct { 18 | s string 19 | i int 20 | length int 21 | state fmsState 22 | startToken byte 23 | shouldEscape bool 24 | currArgument []byte 25 | err error 26 | } 27 | 28 | func newCmdArgumentParser(s string) *cmdArgumentParser { 29 | return &cmdArgumentParser{ 30 | s: s, 31 | i: -1, 32 | length: len(s), 33 | currArgument: make([]byte, 0, 16), 34 | } 35 | } 36 | 37 | func (cap *cmdArgumentParser) parse() (arguments []string) { 38 | for { 39 | cap.next() 40 | 41 | if cap.err != nil { 42 | if cap.shouldEscape { 43 | cap.currArgument = append(cap.currArgument, '\\') 44 | } 45 | 46 | if len(cap.currArgument) > 0 { 47 | arguments = append(arguments, string(cap.currArgument)) 48 | } 49 | 50 | return 51 | } 52 | 53 | switch cap.state { 54 | case stateArgumentOutside: 55 | cap.detectStartToken() 56 | case stateArgumentStart: 57 | if !cap.detectEnd() { 58 | cap.detectContent() 59 | } 60 | case stateArgumentEnd: 61 | cap.state = stateArgumentOutside 62 | arguments = append(arguments, string(cap.currArgument)) 63 | cap.currArgument = cap.currArgument[:0] 64 | } 65 | } 66 | } 67 | 68 | func (cap *cmdArgumentParser) previous() { 69 | if cap.i >= 0 { 70 | cap.i-- 71 | } 72 | } 73 | 74 | func (cap *cmdArgumentParser) next() { 75 | if cap.length-cap.i == 1 { 76 | cap.err = errEndOfLine 77 | return 78 | } 79 | cap.i++ 80 | } 81 | 82 | func (cap *cmdArgumentParser) detectStartToken() { 83 | c := cap.s[cap.i] 84 | if c == ' ' { 85 | return 86 | } 87 | 88 | switch c { 89 | case '\\': 90 | cap.startToken = 0 91 | cap.shouldEscape = true 92 | case '"', '\'': 93 | cap.startToken = c 94 | default: 95 | cap.startToken = 0 96 | cap.previous() 97 | } 98 | cap.state = stateArgumentStart 99 | } 100 | 101 | func (cap *cmdArgumentParser) detectContent() { 102 | c := cap.s[cap.i] 103 | 104 | if cap.shouldEscape { 105 | switch c { 106 | case ' ', '\\', cap.startToken: 107 | cap.currArgument = append(cap.currArgument, c) 108 | default: 109 | cap.currArgument = append(cap.currArgument, '\\', c) 110 | } 111 | cap.shouldEscape = false 112 | return 113 | } 114 | 115 | if c == '\\' { 116 | cap.shouldEscape = true 117 | } else { 118 | cap.currArgument = append(cap.currArgument, c) 119 | } 120 | } 121 | 122 | func (cap *cmdArgumentParser) detectEnd() (detected bool) { 123 | c := cap.s[cap.i] 124 | 125 | if cap.startToken == 0 { 126 | if c == ' ' && !cap.shouldEscape { 127 | cap.state = stateArgumentEnd 128 | cap.previous() 129 | return true 130 | } 131 | return false 132 | } 133 | 134 | if c == cap.startToken && !cap.shouldEscape { 135 | cap.state = stateArgumentEnd 136 | return true 137 | } 138 | 139 | return false 140 | } 141 | 142 | func ParseCmdArguments(s string) (arguments []string) { 143 | return newCmdArgumentParser(s).parse() 144 | } 145 | -------------------------------------------------------------------------------- /utils/argument_parser_test.go: -------------------------------------------------------------------------------- 1 | package utils 2 | 3 | import ( 4 | "testing" 5 | 6 | . "github.com/smartystreets/goconvey/convey" 7 | ) 8 | 9 | func TestCmdArgumentParser(t *testing.T) { 10 | var args []string 11 | var str string 12 | 13 | Convey("Parse Cmd Arguments ["+str+"]", t, func() { 14 | args = ParseCmdArguments(str) 15 | So(len(args), ShouldEqual, 0) 16 | }) 17 | 18 | str = " " 19 | Convey("Parse Cmd Arguments ["+str+"]", t, func() { 20 | args = ParseCmdArguments(str) 21 | So(len(args), ShouldEqual, 0) 22 | }) 23 | 24 | str = "aa bbb ccc " 25 | Convey("Parse Cmd Arguments ["+str+"]", t, func() { 26 | args = ParseCmdArguments(str) 27 | So(len(args), ShouldEqual, 3) 28 | So(args[0], ShouldEqual, "aa") 29 | So(args[1], ShouldEqual, "bbb") 30 | So(args[2], ShouldEqual, "ccc") 31 | }) 32 | 33 | str = "' \\\"" 34 | Convey("Parse Cmd Arguments ["+str+"]", t, func() { 35 | args = ParseCmdArguments(str) 36 | So(len(args), ShouldEqual, 1) 37 | So(args[0], ShouldEqual, " \\\"") 38 | }) 39 | 40 | str = `a "b c"` // a "b c" 41 | Convey("Parse Cmd Arguments ["+str+"]", t, func() { 42 | args = ParseCmdArguments(str) 43 | So(len(args), ShouldEqual, 2) 44 | So(args[0], ShouldEqual, "a") 45 | So(args[1], ShouldEqual, "b c") 46 | }) 47 | 48 | str = `a '\''"` 49 | Convey("Parse Cmd Arguments ["+str+"]", t, func() { 50 | args = ParseCmdArguments(str) 51 | So(len(args), ShouldEqual, 2) 52 | So(args[0], ShouldEqual, "a") 53 | So(args[1], ShouldEqual, "'") 54 | }) 55 | 56 | str = ` \\a 'b c' c\ d\ ` 57 | Convey("Parse Cmd Arguments ["+str+"]", t, func() { 58 | args = ParseCmdArguments(str) 59 | So(len(args), ShouldEqual, 3) 60 | So(args[0], ShouldEqual, "\\a") 61 | So(args[1], ShouldEqual, "b c") 62 | So(args[2], ShouldEqual, "c d ") 63 | }) 64 | 65 | str = `\` 66 | Convey("Parse Cmd Arguments ["+str+"]", t, func() { 67 | args = ParseCmdArguments(str) 68 | So(len(args), ShouldEqual, 1) 69 | So(args[0], ShouldEqual, "\\") 70 | }) 71 | 72 | str = ` \ ` // \SPACE 73 | Convey("Parse Cmd Arguments ["+str+"]", t, func() { 74 | args = ParseCmdArguments(str) 75 | So(len(args), ShouldEqual, 1) 76 | So(args[0], ShouldEqual, " ") 77 | }) 78 | } 79 | -------------------------------------------------------------------------------- /utils/confutil.go: -------------------------------------------------------------------------------- 1 | // 加载json（可配置扩展字段）配置文件 2 | // 3 | // { 4 | // "Debug": true, 5 | // "Log": "@extend:./log.json" 6 | // } 7 | package utils 8 | 9 | import ( 10 | "bytes" 11 | "encoding/json" 12 | "fmt" 13 | "io/ioutil" 14 | "os" 15 | "path/filepath" 16 | "regexp" 17 | ) 18 | 19 | var ( 20 | extendTag = "@extend:" 21 | pwdTag = "@pwd@" 22 | rootTag = "@root@" 23 | root = "" 24 | ) 25 | 26 | // 设置扩展标识，如果不设置，默认为 '@extend:' 27 | func SetExtendTag(tag string) { 28 | extendTag = tag 29 | } 30 | 31 | func SetRoot(r string) { 32 | root = r 33 | } 34 | 35 | // 设置当前路径标识，如果不设置，默认为 '@pwd@' 36 | // @pwd@ 会被替换成当前文件的路径， 37 | // 至于是绝对路径还是相对路径，取决于读取文件时，传入的是绝对路径还是相对路径 38 | func SetPathTag(tag string) { 39 | pwdTag = tag 40 | } 41 | 42 | //加载json（可配置扩展字段）配置文件 43 | func LoadExtendConf(filePath string, v interface{}) error { 44 | data, err := extendFile(filePath) 45 | if err != nil { 46 | return err 47 | } 48 | return json.Unmarshal(data, v) 49 | } 50 | 51 | func extendFile(filePath string) (data []byte, err error) { 52 | fi, err := os.Stat(filePath) 53 | if err != nil { 54 | return 55 | } 56 | if fi.IsDir() { 57 | err = fmt.Errorf(filePath + " is not a file.") 58 | return 59 | } 60 | 61 | b, err := ioutil.ReadFile(filePath) 62 | if err != nil { 63 | return 64 | } 65 | 66 | if len(root) != 0 { 67 | b = bytes.Replace(b, []byte(rootTag), []byte(root), -1) 68 | } 69 | 70 | dir := filepath.Dir(filePath) 71 | return extendFileContent(dir, bytes.Replace(b, []byte(pwdTag), []byte(dir), -1)) 72 | } 73 | 74 | func extendFileContent(dir string, content []byte) (data []byte, err error) { 75 | //检查是不是规范的json 76 | test := new(interface{}) 77 | err = json.Unmarshal(content, &test) 78 | if err != nil { 79 | return 80 | } 81 | 82 | // 替换子json文件 83 | reg := regexp.MustCompile(`"` + extendTag + `.*?"`) 84 | data = reg.ReplaceAllFunc(content, func(match []byte) []byte { 85 | match = match[len(extendTag)+1 : len(match)-1] 86 | sb, e := extendFile(filepath.Join(dir, string(match))) 87 | if e != nil { 88 | err = fmt.Errorf("替换json配置[%s]失败：%s\n", match, e.Error()) 89 | } 90 | return sb 91 | }) 92 | return 93 | } 94 | -------------------------------------------------------------------------------- /utils/confutil_test.go: -------------------------------------------------------------------------------- 1 | package utils 2 | 3 | import ( 4 | "testing" 5 | 6 | . "github.com/smartystreets/goconvey/convey" 7 | ) 8 | 9 | func TestLoadExtendConf(t *testing.T) { 10 | testFile := "test.json" 11 | 12 | type conf struct { 13 | Debug bool 14 | Num int 15 | Log struct { 16 | Level int 17 | Path string 18 | } 19 | } 20 | Convey("confutil package test", t, func() { 21 | Convey("load test file should be success", func() { 22 | c := &conf{} 23 | err := LoadExtendConf(testFile, c) 24 | So(err, ShouldBeNil) 25 | So(c.Debug, ShouldBeTrue) 26 | So(c.Log.Path, ShouldEqual, "./tmp") 27 | }) 28 | }) 29 | } 30 | -------------------------------------------------------------------------------- /utils/local_ip.go: -------------------------------------------------------------------------------- 1 | package utils 2 | 3 | import ( 4 | "fmt" 5 | "net" 6 | ) 7 | 8 | // 获取本机 ip 9 | // 获取第一个非 loopback ip 10 | func LocalIP() (net.IP, error) { 11 | tables, err := net.Interfaces() 12 | if err != nil { 13 | return nil, err 14 | } 15 | for _, t := range tables { 16 | addrs, err := t.Addrs() 17 | if err != nil { 18 | return nil, err 19 | } 20 | for _, a := range addrs { 21 | ipnet, ok := a.(*net.IPNet) 22 | if !ok || ipnet.IP.IsLoopback() { 23 | continue 24 | } 25 | if v4 := ipnet.IP.To4(); v4 != nil { 26 | return v4, nil 27 | } 28 | } 29 | } 30 | return nil, fmt.Errorf("cannot find local IP address") 31 | } 32 | -------------------------------------------------------------------------------- /utils/string.go: -------------------------------------------------------------------------------- 1 | package utils 2 | 3 | import ( 4 | "math/rand" 5 | "time" 6 | ) 7 | 8 | // ASCII values 33 ~ 126 9 | const _dcl = 126 - 33 + 1 10 | 11 | var defaultCharacters [_dcl]byte 12 | 13 | func init() { 14 | for i := 0; i < _dcl; i++ { 15 | defaultCharacters[i] = byte(i + 33) 16 | } 17 | 18 | rand.Seed(time.Now().UnixNano()) 19 | } 20 | 21 | func RandString(length int, characters ...byte) string { 22 | if len(characters) == 0 { 23 | characters = defaultCharacters[:] 24 | } 25 | 26 | n := len(characters) 27 | var rs = make([]byte, length) 28 | 29 | for i := 0; i < length; i++ { 30 | rs[i] = characters[rand.Intn(n-1)] 31 | } 32 | 33 | return string(rs) 34 | } 35 | -------------------------------------------------------------------------------- /utils/test.json: -------------------------------------------------------------------------------- 1 | { 2 | "Debug": true, 3 | "Num": 1, 4 | "Log": "@extend:test1.json" 5 | } -------------------------------------------------------------------------------- /utils/test1.json: -------------------------------------------------------------------------------- 1 | { 2 | "Level": 2, 3 | "Path": "@pwd@/tmp" 4 | } -------------------------------------------------------------------------------- /version.go: -------------------------------------------------------------------------------- 1 | package cronsun 2 | 3 | import ( 4 | "fmt" 5 | "runtime" 6 | ) 7 | 8 | const VersionNumber = "0.3.5" 9 | 10 | var ( 11 | Version = fmt.Sprintf("v%s (build %s)", VersionNumber, runtime.Version()) 12 | ) 13 | -------------------------------------------------------------------------------- /web/authentication.go: -------------------------------------------------------------------------------- 1 | package web 2 | 3 | import ( 4 | "crypto/md5" 5 | "encoding/hex" 6 | "encoding/json" 7 | "fmt" 8 | "net/http" 9 | 10 | "strings" 11 | 12 | "github.com/shunfei/cronsun" 13 | "github.com/shunfei/cronsun/conf" 14 | "github.com/shunfei/cronsun/log" 15 | "github.com/shunfei/cronsun/utils" 16 | "gopkg.in/mgo.v2" 17 | "gopkg.in/mgo.v2/bson" 18 | ) 19 | 20 | func checkAuthBasicData() error { 21 | if conf.Config.Web.Auth.Enabled { 22 | log.Infof("Authentication enabled.") 23 | 24 | list, err := cronsun.GetAccounts(bson.M{"role": cronsun.Administrator, "status": cronsun.UserActived}) 25 | if err != nil { 26 | return fmt.Errorf("Failed to check available Administrators: %s.", err.Error()) 27 | } 28 | 29 | if len(list) == 0 { 30 | // create a default administrator with admin@admin.com/admin 31 | // the email and password can be change from the user profile page. 32 | salt := genSalt() 33 | err = cronsun.CreateAccount(&cronsun.Account{ 34 | Role: cronsun.Administrator, 35 | Email: "admin@admin.com", 36 | Salt: salt, 37 | Password: encryptPassword("admin", salt), 38 | Status: cronsun.UserActived, 39 | Unchangeable: true, 40 | }) 41 | if err != nil { 42 | return fmt.Errorf("Failed to create default Administrators: %s.", err.Error()) 43 | } 44 | } 45 | 46 | if err = cronsun.EnsureAccountIndex(); err != nil { 47 | log.Warnf("Failed to make db index on the `user` collection: %s.", err.Error()) 48 | } 49 | } 50 | 51 | return nil 52 | } 53 | 54 | func encryptPassword(pwd, salt string) string { 55 | m := md5.Sum([]byte(pwd + salt)) 56 | m = md5.Sum(m[:]) 57 | return hex.EncodeToString(m[:]) 58 | } 59 | 60 | func genSalt() string { 61 | return utils.RandString(8) 62 | } 63 | 64 | type Authentication struct{} 65 | 66 | func (this *Authentication) GetAuthSession(ctx *Context) { 67 | var authInfo = &struct { 68 | Role cronsun.Role `json:"role,omitempty"` 69 | Email string `json:"email,omitempty"` 70 | EnabledAuth bool `json:"enabledAuth"` 71 | }{} 72 | 73 | if !conf.Config.Web.Auth.Enabled { 74 | outJSONWithCode(ctx.W, http.StatusOK, authInfo) 75 | return 76 | } 77 | 78 | authInfo.EnabledAuth = true 79 | 80 | if ctx.Session.Email != "" { 81 | authInfo.Email = ctx.Session.Email 82 | authInfo.Role = ctx.Session.Data["role"].(cronsun.Role) 83 | outJSONWithCode(ctx.W, http.StatusOK, authInfo) 84 | return 85 | } 86 | 87 | if len(getStringVal("check", ctx.R)) > 0 { 88 | outJSONWithCode(ctx.W, http.StatusUnauthorized, nil) 89 | return 90 | } 91 | 92 | email := getStringVal("email", ctx.R) 93 | password := getStringVal("password", ctx.R) 94 | remember := getStringVal("remember", ctx.R) == "on" 95 | 96 | u, err := cronsun.GetAccountByEmail(email) 97 | if err != nil { 98 | if err == mgo.ErrNotFound { 99 | outJSONWithCode(ctx.W, http.StatusNotFound, "User ["+email+"] not found.") 100 | } else { 101 | outJSONWithCode(ctx.W, http.StatusInternalServerError, err.Error()) 102 | } 103 | return 104 | } 105 | 106 | if u.Password != encryptPassword(password, u.Salt) { 107 | outJSONWithCode(ctx.W, http.StatusBadRequest, "Incorrect password.") 108 | return 109 | } 110 | 111 | if u.Status != cronsun.UserActived { 112 | outJSONWithCode(ctx.W, http.StatusForbidden, "Access deny.") 113 | return 114 | } 115 | 116 | if !remember { 117 | if c, err := ctx.R.Cookie(conf.Config.Web.Session.CookieName); err == nil { 118 | c.MaxAge = 0 119 | c.Path = "/" 120 | http.SetCookie(ctx.W, c) 121 | } 122 | } 123 | 124 | ctx.Session.Email = u.Email 125 | ctx.Session.Data["role"] = u.Role 126 | ctx.Session.Store() 127 | 128 | authInfo.Role = u.Role 129 | authInfo.Email = u.Email 130 | 131 | err = cronsun.UpdateAccount(bson.M{"email": email}, bson.M{"session": ctx.Session.ID()}) 132 | outJSONWithCode(ctx.W, http.StatusOK, authInfo) 133 | } 134 | 135 | func (this *Authentication) DeleteAuthSession(ctx *Context) { 136 | ctx.Session.Email = "" 137 | delete(ctx.Session.Data, "role") 138 | ctx.Session.Store() 139 | 140 | outJSONWithCode(ctx.W, http.StatusOK, nil) 141 | } 142 | 143 | func (this *Authentication) SetPassword(ctx *Context) { 144 | var sp = &struct { 145 | Password string `json:"password"` 146 | NewPassword string `json:"newPassword"` 147 | }{} 148 | 149 | decoder := json.NewDecoder(ctx.R.Body) 150 | err := decoder.Decode(&sp) 151 | if err != nil { 152 | outJSONWithCode(ctx.W, http.StatusBadRequest, err.Error()) 153 | return 154 | } 155 | ctx.R.Body.Close() 156 | 157 | sp.Password = strings.TrimSpace(sp.Password) 158 | sp.NewPassword = strings.TrimSpace(sp.NewPassword) 159 | if sp.Password == "" { 160 | outJSONWithCode(ctx.W, http.StatusBadRequest, "Passowrd is required.") 161 | return 162 | } 163 | if sp.NewPassword == "" { 164 | outJSONWithCode(ctx.W, http.StatusBadRequest, "New passowrd is required.") 165 | return 166 | } 167 | 168 | var email = ctx.Session.Email 169 | u, err := cronsun.GetAccountByEmail(email) 170 | if err != nil { 171 | if err == mgo.ErrNotFound { 172 | outJSONWithCode(ctx.W, http.StatusNotFound, "User ["+email+"] not found.") 173 | } else { 174 | outJSONWithCode(ctx.W, http.StatusInternalServerError, err.Error()) 175 | } 176 | return 177 | } 178 | 179 | if u.Password != encryptPassword(sp.Password, u.Salt) { 180 | outJSONWithCode(ctx.W, http.StatusBadRequest, "Incorrect password.") 181 | return 182 | } 183 | 184 | salt := genSalt() 185 | update := bson.M{ 186 | "salt": salt, 187 | "password": encryptPassword(sp.NewPassword, salt), 188 | } 189 | 190 | if err = cronsun.UpdateAccount(bson.M{"email": email}, update); err != nil { 191 | if err == mgo.ErrNotFound { 192 | outJSONWithCode(ctx.W, http.StatusBadRequest, "User ["+email+"] not found.") 193 | } else { 194 | outJSONWithCode(ctx.W, http.StatusInternalServerError, err.Error()) 195 | } 196 | return 197 | } 198 | 199 | outJSONWithCode(ctx.W, http.StatusOK, nil) 200 | } 201 | -------------------------------------------------------------------------------- /web/configuration.go: -------------------------------------------------------------------------------- 1 | package web 2 | 3 | import "github.com/shunfei/cronsun/conf" 4 | 5 | type Configuration struct{} 6 | 7 | func (cnf *Configuration) Configuratios(ctx *Context) { 8 | r := struct { 9 | Security *conf.Security `json:"security"` 10 | Alarm bool `json:"alarm"` 11 | LogExpirationDays int `json:"log_expiration_days"` 12 | }{ 13 | Security: conf.Config.Security, 14 | Alarm: conf.Config.Mail.Enable, 15 | } 16 | 17 | if conf.Config.Web.LogCleaner.EveryMinute > 0 { 18 | r.LogExpirationDays = conf.Config.Web.LogCleaner.ExpirationDays 19 | } 20 | 21 | outJSON(ctx.W, r) 22 | } 23 | -------------------------------------------------------------------------------- /web/gen_bindata.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | BASEDIR=$(dirname $(realpath $0)) 3 | cd $BASEDIR/ui 4 | npm run build 5 | cd .. 6 | go-bindata -pkg "web" -prefix "ui/dist/" -o static_assets.go ./ui/dist/ 7 | 8 | VER=$(git rev-parse --short HEAD) 9 | sed -i '' -E "s/(build\.js\?v=).{7}/\1${VER}/g" $BASEDIR/ui/index.html 10 | -------------------------------------------------------------------------------- /web/info.go: -------------------------------------------------------------------------------- 1 | package web 2 | 3 | import ( 4 | "time" 5 | 6 | v3 "github.com/coreos/etcd/clientv3" 7 | 8 | "github.com/shunfei/cronsun" 9 | "github.com/shunfei/cronsun/conf" 10 | ) 11 | 12 | type Info struct{} 13 | 14 | func (inf *Info) Overview(ctx *Context) { 15 | var overview = struct { 16 | TotalJobs int64 `json:"totalJobs"` 17 | JobExecuted *cronsun.StatExecuted `json:"jobExecuted"` 18 | JobExecutedDaily []*cronsun.StatExecuted `json:"jobExecutedDaily"` 19 | }{} 20 | 21 | const day = 24 * time.Hour 22 | days := 7 23 | 24 | overview.JobExecuted, _ = cronsun.JobLogStat() 25 | end := time.Now() 26 | begin := end.Add(time.Duration(1-days) * day) 27 | statList, _ := cronsun.JobLogDailyStat(begin, end) 28 | list := make([]*cronsun.StatExecuted, days) 29 | cur := begin 30 | 31 | for i := 0; i < days; i++ { 32 | date := cur.Format("2006-01-02") 33 | var se *cronsun.StatExecuted 34 | 35 | for j := range statList { 36 | if statList[j].Date == date { 37 | se = statList[j] 38 | statList = statList[1:] 39 | break 40 | } 41 | } 42 | 43 | if se != nil { 44 | list[i] = se 45 | } else { 46 | list[i] = &cronsun.StatExecuted{Date: date} 47 | } 48 | 49 | cur = cur.Add(day) 50 | } 51 | 52 | overview.JobExecutedDaily = list 53 | gresp, err := cronsun.DefalutClient.Get(conf.Config.Cmd, v3.WithPrefix(), v3.WithCountOnly()) 54 | if err == nil { 55 | overview.TotalJobs = gresp.Count 56 | } 57 | 58 | outJSON(ctx.W, overview) 59 | } 60 | -------------------------------------------------------------------------------- /web/job_log.go: -------------------------------------------------------------------------------- 1 | package web 2 | 3 | import ( 4 | "math" 5 | "net/http" 6 | "strings" 7 | "time" 8 | 9 | "github.com/gorilla/mux" 10 | "gopkg.in/mgo.v2" 11 | "gopkg.in/mgo.v2/bson" 12 | 13 | "github.com/shunfei/cronsun" 14 | ) 15 | 16 | func EnsureJobLogIndex() { 17 | cronsun.GetDb().WithC(cronsun.Coll_JobLog, func(c *mgo.Collection) error { 18 | c.EnsureIndex(mgo.Index{ 19 | Key: []string{"beginTime"}, 20 | }) 21 | c.EnsureIndex(mgo.Index{ 22 | Key: []string{"hostname"}, 23 | }) 24 | c.EnsureIndex(mgo.Index{ 25 | Key: []string{"ip"}, 26 | }) 27 | 28 | return nil 29 | }) 30 | } 31 | 32 | type JobLog struct{} 33 | 34 | func (jl *JobLog) GetDetail(ctx *Context) { 35 | vars := mux.Vars(ctx.R) 36 | id := strings.TrimSpace(vars["id"]) 37 | if len(id) == 0 { 38 | outJSONWithCode(ctx.W, http.StatusBadRequest, "empty log id.") 39 | return 40 | } 41 | 42 | if !bson.IsObjectIdHex(id) { 43 | outJSONWithCode(ctx.W, http.StatusBadRequest, "invalid ObjectId.") 44 | return 45 | } 46 | 47 | logDetail, err := cronsun.GetJobLogById(bson.ObjectIdHex(id)) 48 | if err != nil { 49 | statusCode := http.StatusInternalServerError 50 | if err == mgo.ErrNotFound { 51 | statusCode = http.StatusNotFound 52 | err = nil 53 | } 54 | outJSONWithCode(ctx.W, statusCode, err) 55 | return 56 | } 57 | 58 | outJSON(ctx.W, logDetail) 59 | } 60 | 61 | func searchText(field string, keywords []string) (q []bson.M) { 62 | for _, k := range keywords { 63 | k = strings.TrimSpace(k) 64 | if len(k) == 0 { 65 | continue 66 | } 67 | q = append(q, bson.M{field: bson.M{"$regex": bson.RegEx{Pattern: k, Options: "i"}}}) 68 | } 69 | 70 | return q 71 | } 72 | 73 | func (jl *JobLog) GetList(ctx *Context) { 74 | hostnames := getStringArrayFromQuery("hostnames", ",", ctx.R) 75 | ips := getStringArrayFromQuery("ips", ",", ctx.R) 76 | names := getStringArrayFromQuery("names", ",", ctx.R) 77 | ids := getStringArrayFromQuery("ids", ",", ctx.R) 78 | begin := getTime(ctx.R.FormValue("begin")) 79 | end := getTime(ctx.R.FormValue("end")) 80 | page := getPage(ctx.R.FormValue("page")) 81 | failedOnly := ctx.R.FormValue("failedOnly") == "true" 82 | pageSize := getPageSize(ctx.R.FormValue("pageSize")) 83 | orderBy := "-beginTime" 84 | 85 | query := bson.M{} 86 | var textSearch = make([]bson.M, 0, 2) 87 | textSearch = append(textSearch, searchText("hostname", hostnames)...) 88 | textSearch = append(textSearch, searchText("name", names)...) 89 | 90 | if len(ips) > 0 { 91 | query["ip"] = bson.M{"$in": ips} 92 | } 93 | 94 | if len(ids) > 0 { 95 | query["jobId"] = bson.M{"$in": ids} 96 | } 97 | 98 | if !begin.IsZero() { 99 | query["beginTime"] = bson.M{"$gte": begin} 100 | } 101 | if !end.IsZero() { 102 | query["endTime"] = bson.M{"$lt": end.Add(time.Hour * 24)} 103 | } 104 | 105 | if failedOnly { 106 | query["success"] = false 107 | } 108 | 109 | if len(textSearch) > 0 { 110 | query["$or"] = textSearch 111 | } 112 | 113 | var pager struct { 114 | Total int `json:"total"` 115 | List []*cronsun.JobLog `json:"list"` 116 | } 117 | var err error 118 | if ctx.R.FormValue("latest") == "true" { 119 | var latestLogList []*cronsun.JobLatestLog 120 | latestLogList, pager.Total, err = cronsun.GetJobLatestLogList(query, page, pageSize, orderBy) 121 | for i := range latestLogList { 122 | latestLogList[i].JobLog.Id = bson.ObjectIdHex(latestLogList[i].RefLogId) 123 | pager.List = append(pager.List, &latestLogList[i].JobLog) 124 | } 125 | } else { 126 | pager.List, pager.Total, err = cronsun.GetJobLogList(query, page, pageSize, orderBy) 127 | } 128 | if err != nil { 129 | outJSONWithCode(ctx.W, http.StatusInternalServerError, err.Error()) 130 | return 131 | } 132 | 133 | pager.Total = int(math.Ceil(float64(pager.Total) / float64(pageSize))) 134 | outJSON(ctx.W, pager) 135 | } 136 | -------------------------------------------------------------------------------- /web/log_cleaner.go: -------------------------------------------------------------------------------- 1 | package web 2 | 3 | import ( 4 | "time" 5 | 6 | "github.com/shunfei/cronsun" 7 | "github.com/shunfei/cronsun/log" 8 | mgo "gopkg.in/mgo.v2" 9 | "gopkg.in/mgo.v2/bson" 10 | ) 11 | 12 | func RunLogCleaner(cleanPeriod, expiration time.Duration) (close chan struct{}) { 13 | t := time.NewTicker(cleanPeriod) 14 | close = make(chan struct{}) 15 | go func() { 16 | for { 17 | select { 18 | case <-t.C: 19 | cleanupLogs(expiration) 20 | case <-close: 21 | return 22 | } 23 | } 24 | }() 25 | 26 | return 27 | } 28 | 29 | func cleanupLogs(expiration time.Duration) { 30 | err := cronsun.GetDb().WithC(cronsun.Coll_JobLog, func(c *mgo.Collection) error { 31 | _, err := c.RemoveAll(bson.M{"$or": []bson.M{ 32 | bson.M{"$and": []bson.M{ 33 | bson.M{"cleanup": bson.M{"$exists": true}}, 34 | bson.M{"cleanup": bson.M{"$lte": time.Now()}}, 35 | }}, 36 | bson.M{"$and": []bson.M{ 37 | bson.M{"cleanup": bson.M{"$exists": false}}, 38 | bson.M{"endTime": bson.M{"$lte": time.Now().Add(-expiration)}}, 39 | }}, 40 | }}) 41 | 42 | return err 43 | }) 44 | 45 | if err != nil { 46 | log.Errorf("[Cleaner] Failed to remove expired logs: %s", err.Error()) 47 | return 48 | } 49 | 50 | } 51 | -------------------------------------------------------------------------------- /web/session/session.go: -------------------------------------------------------------------------------- 1 | package session 2 | 3 | import ( 4 | "bytes" 5 | "encoding/gob" 6 | "errors" 7 | "net/http" 8 | 9 | client "github.com/coreos/etcd/clientv3" 10 | "github.com/shunfei/cronsun" 11 | "github.com/shunfei/cronsun/conf" 12 | "github.com/shunfei/cronsun/log" 13 | "github.com/shunfei/cronsun/utils" 14 | ) 15 | 16 | func init() { 17 | gob.Register(cronsun.Administrator) 18 | gob.Register(cronsun.Developer) 19 | gob.Register(cronsun.Reporter) 20 | } 21 | 22 | var Manager SessionManager 23 | var cookieCharacters = []byte("0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ") 24 | 25 | type SessionManager interface { 26 | Get(w http.ResponseWriter, r *http.Request) (*Session, error) 27 | Store(*Session) error 28 | Destroy(w http.ResponseWriter, r *http.Request) 29 | CleanSeesionData(id string) 30 | } 31 | 32 | type storeData struct { 33 | leaseID client.LeaseID 34 | Email string 35 | Data map[interface{}]interface{} 36 | } 37 | 38 | type Session struct { 39 | m SessionManager 40 | key string 41 | storeData 42 | } 43 | 44 | func (s *Session) ID() string { 45 | return s.key 46 | } 47 | func (s *Session) Store() error { 48 | err := s.m.Store(s) 49 | if err != nil { 50 | log.Errorf("Failed to store session[%s]: %s", s.key, err.Error()) 51 | } 52 | return err 53 | } 54 | 55 | type EtcdStore struct { 56 | client *cronsun.Client 57 | conf conf.SessionConfig 58 | } 59 | 60 | func NewEtcdStore(cli *cronsun.Client, conf conf.SessionConfig) *EtcdStore { 61 | return &EtcdStore{ 62 | client: cli, 63 | conf: conf, 64 | } 65 | } 66 | 67 | func (this *EtcdStore) Get(w http.ResponseWriter, r *http.Request) (sess *Session, err error) { 68 | c, err := r.Cookie(this.conf.CookieName) 69 | if err != nil && err != http.ErrNoCookie { 70 | log.Infof("get cookie err: %s", err.Error()) 71 | } else { 72 | err = nil 73 | } 74 | 75 | sess = &Session{ 76 | m: this, 77 | storeData: storeData{ 78 | Data: make(map[interface{}]interface{}, 2), 79 | }, 80 | } 81 | 82 | if c == nil { 83 | sess.key = utils.RandString(32, cookieCharacters...) 84 | c = &http.Cookie{ 85 | Name: this.conf.CookieName, 86 | Value: sess.key, 87 | Path: "/", 88 | HttpOnly: true, 89 | Secure: false, 90 | MaxAge: this.conf.Expiration, 91 | } 92 | http.SetCookie(w, c) 93 | r.AddCookie(c) 94 | 95 | return 96 | } 97 | 98 | sess.key = c.Value 99 | resp, err := this.client.Get(this.storeKey(c.Value)) 100 | if err != nil { 101 | return 102 | } 103 | 104 | if len(resp.Kvs) == 0 { 105 | return sess, nil 106 | } 107 | 108 | var buffer = bytes.NewBuffer(resp.Kvs[0].Value) 109 | dec := gob.NewDecoder(buffer) 110 | err = dec.Decode(&sess.storeData) 111 | return 112 | } 113 | 114 | func (this *EtcdStore) Store(sess *Session) (err error) { 115 | var buffer = bytes.NewBuffer(nil) 116 | enc := gob.NewEncoder(buffer) 117 | err = enc.Encode(sess.storeData) 118 | if err != nil { 119 | return 120 | } 121 | 122 | if sess.leaseID == 0 { 123 | lresp, err := this.client.Grant(int64(this.conf.Expiration)) 124 | if err != nil { 125 | return errors.New("etcd create new lease faild: " + err.Error()) // err 126 | } 127 | sess.leaseID = lresp.ID 128 | } 129 | 130 | _, err = this.client.Put(this.storeKey(sess.key), buffer.String(), client.WithLease(sess.leaseID)) 131 | this.client.KeepAliveOnce(sess.leaseID) 132 | return 133 | } 134 | 135 | func (this *EtcdStore) Destroy(w http.ResponseWriter, r *http.Request) { 136 | c, err := r.Cookie(this.conf.CookieName) 137 | if err != nil || c == nil { 138 | return 139 | } 140 | this.CleanSeesionData(c.Value) 141 | } 142 | 143 | func (this *EtcdStore) CleanSeesionData(id string) { 144 | _, err := this.client.Delete(this.storeKey(id)) 145 | if err != nil { 146 | log.Errorf("Failed to remove session [%s] from etcd: %s", this.storeKey(id), err.Error()) 147 | } 148 | } 149 | 150 | func (this *EtcdStore) storeKey(key string) string { 151 | return this.conf.StorePrefixPath + key 152 | } 153 | -------------------------------------------------------------------------------- /web/ui/.babelrc: -------------------------------------------------------------------------------- 1 | { 2 | "presets": [ 3 | ["es2015", { "modules": false }] 4 | ] 5 | } -------------------------------------------------------------------------------- /web/ui/index.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | Cronsun Managerment 7 | 58 | 59 | 60 | 61 |
62 |
63 |
64 | 65 | 66 | 67 | 68 | -------------------------------------------------------------------------------- /web/ui/package.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "cronsun-web-ui", 3 | "version": "1.0.0", 4 | "description": "", 5 | "scripts": { 6 | "dev": "cross-env NODE_ENV=development webpack-dev-server --open --inline --hot --disableHostCheck=true", 7 | "build": "cross-env NODE_ENV=production webpack --progress --hide-modules && cp index.html ./dist/", 8 | "test": "echo \"Error: no test specified\" && exit 1" 9 | }, 10 | "private": true, 11 | "author": "heshitan@sunteng.com", 12 | "dependencies": { 13 | "chart.js": "^2.5.0", 14 | "jquery": "^3.1.1", 15 | "jquery.cookie": "^1.4.1", 16 | "semantic-ui": "^2.3.3", 17 | "vue": "^2.3.4", 18 | "vue-router": "^2.2.1", 19 | "vuex": "^2.3.1" 20 | }, 21 | "devDependencies": { 22 | "babel-core": "^6.0.0", 23 | "babel-loader": "^6.0.0", 24 | "babel-preset-es2015": "^6.0.0", 25 | "cross-env": "^3.0.0", 26 | "css-loader": "^0.26.1", 27 | "file-loader": "^0.9.0", 28 | "style-loader": "^0.13.1", 29 | "vue-loader": "^10.0.0", 30 | "vue-template-compiler": "^2.3.4", 31 | "webpack": "^2.1.0-beta.25", 32 | "webpack-dev-server": "^2.1.0-beta.9" 33 | } 34 | } 35 | -------------------------------------------------------------------------------- /web/ui/src/App.vue: -------------------------------------------------------------------------------- 1 | 8 | 9 | 50 | 51 | 122 | -------------------------------------------------------------------------------- /web/ui/src/components/Account.vue: -------------------------------------------------------------------------------- 1 | 30 | 31 | 74 | -------------------------------------------------------------------------------- /web/ui/src/components/AccountEdit.vue: -------------------------------------------------------------------------------- 1 | 55 | 56 | 134 | -------------------------------------------------------------------------------- /web/ui/src/components/Dash.vue: -------------------------------------------------------------------------------- 1 | 7 | 8 | 48 | 49 | 156 | -------------------------------------------------------------------------------- /web/ui/src/components/ExecuteJob.vue: -------------------------------------------------------------------------------- 1 | 14 | 15 | 91 | -------------------------------------------------------------------------------- /web/ui/src/components/JobEdit.vue: -------------------------------------------------------------------------------- 1 | 9 | 10 | 31 | -------------------------------------------------------------------------------- /web/ui/src/components/JobEditRule.vue: -------------------------------------------------------------------------------- 1 | 25 | 26 | 80 | -------------------------------------------------------------------------------- /web/ui/src/components/JobExecuting.vue: -------------------------------------------------------------------------------- 1 | 5 | 53 | 54 | 152 | -------------------------------------------------------------------------------- /web/ui/src/components/LogDetail.vue: -------------------------------------------------------------------------------- 1 | 7 | 49 | 50 | 106 | -------------------------------------------------------------------------------- /web/ui/src/components/Login.vue: -------------------------------------------------------------------------------- 1 | 9 | 10 | 31 | 32 | 72 | -------------------------------------------------------------------------------- /web/ui/src/components/Messager.vue: -------------------------------------------------------------------------------- 1 | 4 | 12 | 13 | -------------------------------------------------------------------------------- /web/ui/src/components/Node.vue: -------------------------------------------------------------------------------- 1 | 29 | 51 | 52 | 103 | -------------------------------------------------------------------------------- /web/ui/src/components/NodeGroup.vue: -------------------------------------------------------------------------------- 1 | 19 | 20 | 47 | 48 | 81 | -------------------------------------------------------------------------------- /web/ui/src/components/NodeGroupEdit.vue: -------------------------------------------------------------------------------- 1 | 22 | 23 | 95 | -------------------------------------------------------------------------------- /web/ui/src/components/Profile.vue: -------------------------------------------------------------------------------- 1 | 16 | 17 | 45 | -------------------------------------------------------------------------------- /web/ui/src/components/basic/Dropdown.vue: -------------------------------------------------------------------------------- 1 | 11 | 12 | 45 | -------------------------------------------------------------------------------- /web/ui/src/components/basic/Pager.vue: -------------------------------------------------------------------------------- 1 | 17 | 18 | 74 | -------------------------------------------------------------------------------- /web/ui/src/i18n/language.js: -------------------------------------------------------------------------------- 1 | import zhCN from './languages/zh-CN'; 2 | import en from './languages/en'; 3 | 4 | import jQuery from 'jquery'; 5 | require('jquery.cookie') 6 | 7 | var languages = { 8 | // key is lowercase 9 | 'en': en, 10 | 'zh-cn': zhCN 11 | } 12 | 13 | var supported = [ 14 | { name: 'English', code: 'en' }, 15 | { name: '简体中文', code: 'zh-CN' } 16 | ] 17 | 18 | var language; 19 | var locale = jQuery.cookie('locale') || navigator.language || 'en'; 20 | locale = locale.indexOf('en') === -1 ? 'zh-CN' : 'en'; 21 | setLocale(locale); 22 | 23 | function setLocale(loc) { 24 | var loc2 = loc.toLowerCase() 25 | if (languages[loc2]) { 26 | locale = loc 27 | language = languages[loc2] 28 | } 29 | } 30 | 31 | function getLocale() { 32 | return locale 33 | } 34 | 35 | function L(k) { 36 | var t = language[k]; 37 | if (typeof t === 'undefined') { 38 | t = k; 39 | } 40 | 41 | var tr = ''; 42 | var inTag = false; 43 | var num = ''; 44 | var stash = ''; 45 | for (var i = 0; i < t.length; i++) { 46 | var cc = t[i].charCodeAt(); 47 | if (cc === 123) { // { 48 | if (inTag) { 49 | tr += stash; 50 | stash = ''; 51 | } 52 | inTag = true; 53 | } else if (inTag && cc >= 48 && cc <= 57) { 54 | num += t[i]; 55 | } else if (inTag && cc === 125) { // } 56 | var index = parseInt(num); 57 | if (arguments.length - 1 > index) { 58 | tr += arguments[index + 1]; 59 | } 60 | num = ''; 61 | stash = ''; 62 | inTag = false; 63 | continue; 64 | } else { 65 | inTag = false; 66 | tr += stash; 67 | stash = ''; 68 | } 69 | 70 | if (inTag) { 71 | stash += t[i]; 72 | } else { 73 | tr += t[i]; 74 | } 75 | } 76 | 77 | return tr; 78 | } 79 | 80 | const lang = { getLocale, setLocale, supported, L }; 81 | export default lang; 82 | -------------------------------------------------------------------------------- /web/ui/src/i18n/languages/zh-CN.js: -------------------------------------------------------------------------------- 1 | var language = { 2 | 'email': '邮箱', 3 | 'password': '密码', 4 | 'new password': '新密码', 5 | 'remember me': '记住登录状态', 6 | 'login': '登入', 7 | 'actived': '正常', 8 | 'banned': '禁止登录', 9 | 'ban': '禁止登录', 10 | 'active': '正常', 11 | 'remove session': '移除登录状态', 12 | 'added time': '添加时间', 13 | 'add account': '添加账号', 14 | 'edit account': '编辑账号', 15 | 'role': '角色', 16 | 'save': '保存', 17 | 'your password has been change': '您的密码已经修改', 18 | 'dashboard': '仪表盘', 19 | 'log': '日志', 20 | 'job': '任务', 21 | 'node': '节点', 22 | 'account': '账号', 23 | 'total number of jobs': '任务总数', 24 | 'total number of executeds': '执行任务总次数', 25 | 'total number of nodes': '节点总数', 26 | 'job executed in past 7 days': '过去 7 天任务统计', 27 | 'node show as': '节点显示为', 28 | 'hostname': '主机名称', 29 | 30 | 'batch': '批量', 31 | 'job name': '任务名称', 32 | 'multiple names can separated by commas': '多个名称用英文逗号分隔', 33 | 'job ID': '任务 ID', 34 | 'multiple IDs can separated by commas': '多个 ID 用英文逗号分隔', 35 | 'multiple Hostnames can separated by commas': '多个主机名称用英文逗号分隔', 36 | 'multiple IPs can separated by commas': '多个 IP 用英文逗号分隔', 37 | 'starting date': '起始日期', 38 | 'end date': '截至日期', 39 | 'failure only': '只看失败的任务', 40 | 'latest result of each job on each node': '只看每个任务在每个节点上最后一次运行的结果', 41 | 'submit query': '查询', 42 | 'executing node': '执行节点', 43 | 'executing user': '执行用户', 44 | 'executing time': '执行时间', 45 | 'executing result': '执行结果', 46 | 'executing job': '执行任务', 47 | 'successed': '成功', 48 | 'failed': '失败', 49 | 'click to select a node and re-execute job': '点此选择节点重新执行任务', 50 | 'took {times}, {begin ~ end}': '耗时 {0}, {1}', 51 | 'executing job: {job}': '执行任务：“{0}”', 52 | 'cancel': '取消', 53 | 'execute now': '立刻执行任务', 54 | 'view executing jobs': '查看执行中的任务', 55 | 'view job list': '查看任务列表', 56 | 'starting time': '开始时间', 57 | 'process ID': '进程ID', 58 | 'kill process': '杀死进程', 59 | 'whether to kill the process': '是否杀死该进程', 60 | 'command has been sent to the node': '命令已经发送到节点', 61 | 62 | 'group filter': '分组过滤', 63 | 'node filter': '节点过滤', 64 | 'select a group': '选择分组', 65 | 'select a node': '选择节点', 66 | 'operation': '操作', 67 | 'status': '状态', 68 | 'group': '分组', 69 | 'user': '用户', 70 | 'name': '名称', 71 | 'latest executed': '最近执行时间', 72 | 'edit': '编辑', 73 | 'open': '开启', 74 | 'pause': '暂停', 75 | 'delete': '删除', 76 | 'all groups': '所有分组', 77 | 'all nodes': '所有节点', 78 | 'on {node} took {times}, {begin ~ end}': '于 {0} 耗时 {1}, {2}', 79 | 'next schedule: {nextTime}': '下个调度: {0}', 80 | 'create job': '新建任务', 81 | 'update job': '更新任务', 82 | 'output': '输出', 83 | 'command': '执行的命令', 84 | 'spend time': '耗时', 85 | 'result': '结果', 86 | 'loading configurations': '正在加载配置', 87 | 'log has been deleted': '日志已经被删除', 88 | 89 | 'job type': '任务类型', 90 | 'common job': '普通任务', 91 | 'single node single process': '单机单进程', 92 | 'group level common': '组级别普通任务', 93 | 'group level common help': '暂时没想到好名字，一个比较简单的说明是，把所有选中的节点视为一个大节点，那么该类型的任务就相当于在单个节点上的普通任务', 94 | 'warning on': '报警已开启', 95 | 'warning off': '报警已关闭', 96 | 'job group': '任务分组', 97 | 'script path': '任务脚本', 98 | '(only [{.suffixs}] files can be allowed)': '（只允许 [{0}] 文件）', 99 | 'user(optional)': '用户（可选）', 100 | 'user(required)': '用户（必选）', 101 | 'the user which to execute the command': '指定执行的用户', 102 | 'node group': '节点分组', 103 | 'warning receiver': '报警接收人（任务失败时下面的地址也会接收到告警）', 104 | 'e-mail address': '邮件地址', 105 | 'retries(number of retries when failed, 0 means no retry)': '重试（失败时重试次数，0 为不重试）', 106 | 'retry interval(in seconds)': '失败重试间隔时间（秒）', 107 | 'parallel number in one node(0 for no limits)': '一个节点上面该任务并行数（0 表示不限制）', 108 | 'timeout(in seconds, 0 for no limits)': '超时设置（单位“秒”，0 表示不限制）', 109 | 'log expiration(log expired after N days, 0 will use default setting: {n} days)': '日志过期（日志保存天数，0 表示使用默认设置：{0} 天）', 110 | '0 * * * * *, rules see the 「?」on the right': '0 * * * * *, 规则参考右边的「?」', 111 | '
, rules is same with Cron': '<秒> <分> <时> <日> <月> <周>，规则与 Cron 一样。' + 112 | '
如果要指定只在某个时间点执行一次（类似Linux系统的at命令），可以使用 "@at 2006-01-02 15:04:05" 这样来设定。' + 113 | '
也支持一些简写，例如 @daily 表示每天执行一次。' + 114 | '
更多请参考wiki。', 115 | 'and please running on those nodes': '同时在这些节点上面运行', 116 | 'do not running on those nodes': '不要在这些节点上面运行', 117 | 'the job dose not have a timer currently, please click the button below to add a timer': '当前任务没有定时器，点击下面按钮来添加定时器', 118 | 'timer': '定时器', 119 | 'add timer': '添加定时器', 120 | 'save job': '保存任务', 121 | 122 | 'node can not be deceted due to itself or network etc.': '因自身或网络等原因未检测到节点存活', 123 | 'node is in maintenance or is shutdown manually': '手动下线/维护中的', 124 | 'node is running': '正常运行的节点', 125 | '(total {n} nodes)': '（共 {0} 节点）', 126 | 'node damaged': '故障节点', 127 | 'node offline': '离线节点', 128 | 'node normaly': '正常节点', 129 | 'currently version': '当前版本号', 130 | 'version inconsistent, node: {version}': '版本不一致，节点版本 {0}', 131 | 'group manager': '分组管理', 132 | 'create node group': '添加节点分组', 133 | 'update node group': '更新节点分组', 134 | 'create group': '新建分组', 135 | 'group name': '分组名称', 136 | 'save group': '保存分组', 137 | 'delete group': '删除分组', 138 | 'include nodes': '包含的节点', 139 | 'select nodes': '选择节点', 140 | 'select groups': '选择分组', 141 | 'are you sure to delete the group {name}?': '确定删除分组 {0}?', 142 | 'are you sure to remove the node {nodeId}?': '确定删除节点 {0}?', 143 | 'node not found, was it removed?': '不存在的节点，被删除了吗？' 144 | } 145 | 146 | export default language; 147 | -------------------------------------------------------------------------------- /web/ui/src/libraries/functions.js: -------------------------------------------------------------------------------- 1 | var formatDuration = function(beginTime, endTime){ 2 | var d = new Date(endTime) - new Date(beginTime); 3 | var s = ''; 4 | var day = Math.floor(d/86400000); 5 | if (day >= 1) s += day.toString() + ' d '; 6 | 7 | d = d%86400000; 8 | var hour = Math.floor(d/3600000); 9 | if (hour >= 1) s += hour.toString() + ' hr '; 10 | 11 | d = d%3600000; 12 | var min = Math.floor(d/60000); 13 | if (min >= 1) s += min.toString() + ' min '; 14 | 15 | d = d%60000; 16 | var sec = Math.floor(d/1000); 17 | if (sec >= 1) s += sec.toString() + ' s '; 18 | 19 | d = Math.floor(d%1000); 20 | if (d >= 1) s += d.toString() + ' ms'; 21 | 22 | if (s.length == 0) s = '0 ms'; 23 | return s; 24 | } 25 | 26 | var formatTime = function(beginTime, endTime){ 27 | var now = new Date(); 28 | var bt = new Date(beginTime); 29 | var et = new Date(endTime); 30 | var s = _formatTime(now, bt) + ' ~ ' + _formatTime(now, et); 31 | return s; 32 | } 33 | 34 | var _formatTime = function(now, t){ 35 | var s = ''; 36 | if (now.getFullYear() != t.getFullYear()) { 37 | s += t.getFullYear().toString() + '-'; 38 | } 39 | s += formatNumber(t.getMonth()+1, 2).toString() + '-'; 40 | s += formatNumber(t.getDate(), 2) + ' ' + formatNumber(t.getHours(), 2) + ':' + formatNumber(t.getMinutes(), 2) + ':' + formatNumber(t.getSeconds(), 2); 41 | return s; 42 | } 43 | 44 | // i > 0 45 | var formatNumber = function(i, len){ 46 | var n = i == 0 ? 1 : Math.ceil(Math.log10(i+1)); 47 | if (n >= len) return i.toString(); 48 | return '0'.repeat(len-n) + i.toString(); 49 | } 50 | 51 | var split = function(str, sep){ 52 | if (typeof str != 'string' || str.length === 0) return []; 53 | return str.split(sep || ','); 54 | } 55 | 56 | export {formatDuration, formatTime, formatNumber, split}; 57 | -------------------------------------------------------------------------------- /web/ui/src/libraries/rest-client.js: -------------------------------------------------------------------------------- 1 | var sendXHR = function(opt) { 2 | var xhr = new XMLHttpRequest(); 3 | xhr.open(opt.method, opt.url, true); 4 | 5 | if (typeof opt.onexception == 'function') { 6 | var warpExceptionHandler = (msg)=>{ 7 | opt.onexception(msg); 8 | typeof opt.onend == 'function' && opt.onend(xhr); 9 | } 10 | xhr.onabort=()=>{warpExceptionHandler('request aborted.')}; 11 | xhr.onerror=()=>{warpExceptionHandler('request error.')}; 12 | xhr.ontimeout=()=>{warpExceptionHandler('request timeout.')}; 13 | } 14 | 15 | xhr.onreadystatechange = function(){ 16 | if (xhr.readyState !== XMLHttpRequest.DONE) { 17 | return; 18 | } 19 | 20 | var data; 21 | if (typeof xhr.response != 'object') { 22 | try { 23 | data = JSON.parse(xhr.response) 24 | } catch(e) { 25 | data = xhr.response; 26 | } 27 | } else { 28 | data = xhr.response; 29 | } 30 | 31 | if (xhr.status != opt.successCode) { 32 | typeof opt.onfailed == 'function' && opt.onfailed(data, xhr); 33 | } else if (xhr.status === opt.successCode && typeof opt.onsucceed == 'function') { 34 | opt.onsucceed(data, xhr); 35 | } else if (opt.specialHandlers && typeof opt.specialHandlers[xhr.status] === 'function') { 36 | opt.specialHandlers[xhr.status](data, xhr); 37 | } 38 | 39 | typeof opt.onend == 'function' && opt.onend(xhr); 40 | } 41 | 42 | if (typeof opt.data == 'object') { 43 | opt.data = JSON.stringify(opt.data); 44 | } 45 | xhr.send(opt.data); 46 | } 47 | 48 | class request { 49 | constructor(url, method, data, specialHandlers){ 50 | this._url = url; 51 | this._method = method; 52 | this._data = data; 53 | this._specialHandlers = specialHandlers; 54 | } 55 | 56 | do(){ 57 | sendXHR({ 58 | method: this._method, 59 | url: this._url, 60 | data: this._data, 61 | successCode: this._successCode, 62 | onsucceed: this._onsucceed, 63 | onfailed: this._onfailed, 64 | onexception: this._onexception, 65 | onend: this._onend, 66 | specialHandlers: this._specialHandlers 67 | }); 68 | } 69 | 70 | onsucceed(successCode, f){ 71 | this._successCode = successCode; 72 | this._onsucceed = f; 73 | return this; 74 | } 75 | 76 | onfailed(f){ 77 | this._onfailed = f; 78 | return this; 79 | } 80 | 81 | onexception(f){ 82 | this._onexception = f; 83 | return this; 84 | } 85 | 86 | onend(f){ 87 | this._onend = f; 88 | return this; 89 | } 90 | } 91 | 92 | export default class Rest { 93 | // specialStatusHandle = map[int]function(data, xhr) 94 | constructor(prefix, defaultFailedHandler, defaultExceptionHandler, specialStatusHandles){ 95 | this.prefix = prefix; 96 | this.defaultFailedHandler = defaultFailedHandler; // function(url, resp){} 97 | this.defaultExceptionHandler = defaultExceptionHandler; 98 | this.specialStatusHandles = specialStatusHandles; 99 | }; 100 | 101 | handleSpecialStatus(code, h) { 102 | this.mh[code] = h 103 | }; 104 | 105 | GET(url){ 106 | return new request(this.prefix+url, 'GET', null, this.specialStatusHandles) 107 | .onfailed(this.defaultFailedHandler) 108 | .onexception(this.defaultExceptionHandler); 109 | }; 110 | 111 | POST(url, formdata){ 112 | return new request(this.prefix+url, 'POST', formdata, this.specialStatusHandles) 113 | .onfailed(this.defaultFailedHandler) 114 | .onexception(this.defaultExceptionHandler); 115 | }; 116 | 117 | PUT(url, formdata){ 118 | return new request(this.prefix+url, 'PUT', formdata, this.specialStatusHandles) 119 | .onfailed(this.defaultFailedHandler) 120 | .onexception(this.defaultExceptionHandler); 121 | }; 122 | 123 | DELETE(url){ 124 | return new request(this.prefix+url, 'DELETE', null, this.specialStatusHandles) 125 | .onfailed(this.defaultFailedHandler) 126 | .onexception(this.defaultExceptionHandler); 127 | } 128 | } 129 | -------------------------------------------------------------------------------- /web/ui/src/main.js: -------------------------------------------------------------------------------- 1 | window.$ = window.jQuery = require('jquery'); 2 | require('semantic'); 3 | require('semantic-ui/dist/semantic.min.css'); 4 | import store from './vuex/store'; 5 | 6 | import Vue from 'vue'; 7 | import Lang from './i18n/language'; 8 | // global restful client 9 | import Rest from './libraries/rest-client.js'; 10 | import VueRouter from 'vue-router'; 11 | import App from './App.vue'; 12 | import Dash from './components/Dash.vue'; 13 | import Log from './components/Log.vue'; 14 | import LogDetail from './components/LogDetail.vue'; 15 | import Job from './components/Job.vue'; 16 | import JobEdit from './components/JobEdit.vue'; 17 | import JobExecuting from './components/JobExecuting.vue'; 18 | import Node from './components/Node.vue'; 19 | import NodeGroup from './components/NodeGroup.vue'; 20 | import NodeGroupEdit from './components/NodeGroupEdit.vue'; 21 | import Account from './components/Account.vue'; 22 | import AccountEdit from './components/AccountEdit.vue'; 23 | import Profile from './components/Profile.vue'; 24 | import Login from './components/Login.vue'; 25 | 26 | Vue.config.debug = true; 27 | 28 | Vue.use((Vue) => { 29 | Vue.prototype.$L = Lang.L 30 | Vue.prototype.$Lang = Lang 31 | }); 32 | 33 | // global event bus 34 | var bus = new Vue(); 35 | Vue.use((Vue) => { 36 | Vue.prototype.$bus = bus; 37 | }); 38 | 39 | 40 | var restApi = new Rest('/v1/', (msg) => { 41 | bus.$emit('error', msg); 42 | }, (msg) => { 43 | bus.$emit('error', msg); 44 | }, { 45 | 401: (data, xhr) => { 46 | bus.$emit('goLogin') 47 | } 48 | }); 49 | var loadNodes = function(resolve) { 50 | restApi.GET('nodes').onsucceed(200, (resp) => { 51 | var nodes = {}; 52 | for (var i in resp) { 53 | nodes[resp[i].id] = resp[i]; 54 | } 55 | store.commit('setNodes', nodes); 56 | if(typeof resolve == "function"){ 57 | resolve(); 58 | } 59 | }).do(); 60 | } 61 | 62 | Vue.use((Vue, options) => { 63 | Vue.prototype.$rest = restApi; 64 | }, null); 65 | 66 | Vue.use(VueRouter); 67 | 68 | Vue.use((Vue) => { 69 | Vue.prototype.$loadConfiguration = () => { 70 | restApi.GET('configurations').onsucceed(200, (resp) => { 71 | const Config = (Vue, options) => { 72 | Vue.prototype.$appConfig = resp; 73 | } 74 | Vue.use(Config); 75 | bus.$emit('conf_loaded', resp); 76 | loadNodes(); 77 | 78 | }).onfailed((data, xhr) => { 79 | var msg = data ? data : xhr.status + ' ' + xhr.statusText; 80 | bus.$emit('error', msg); 81 | }).do(); 82 | } 83 | }); 84 | 85 | const onConfigLoaded = (Vue, options) => { 86 | let loaded = false; 87 | let queue = []; 88 | let appConfig; 89 | 90 | Vue.prototype.$onConfigLoaded = (f) => { 91 | if (loaded) { 92 | f(appConfig); 93 | return; 94 | } 95 | queue.push(f); 96 | } 97 | 98 | bus.$on('conf_loaded', (c) => { 99 | loaded = true; 100 | appConfig = c; 101 | queue.forEach((f) => { 102 | f(appConfig) 103 | }) 104 | }); 105 | } 106 | Vue.use(onConfigLoaded); 107 | 108 | var routes = [ 109 | {path: '/', component: Dash}, 110 | {path: '/log', component: Log}, 111 | {path: '/log/:id', component: LogDetail}, 112 | {path: '/job', component: Job}, 113 | {path: '/job/create', component: JobEdit}, 114 | {path: '/job/edit/:group/:id', component: JobEdit}, 115 | {path: '/job/executing', component: JobExecuting}, 116 | {path: '/node', component: Node}, 117 | {path: '/node/group', component: NodeGroup}, 118 | {path: '/node/group/create', component: NodeGroupEdit}, 119 | {path: '/node/group/:id', component: NodeGroupEdit}, 120 | {path: '/admin/account/list', component: Account}, 121 | {path: '/admin/account/add', component: AccountEdit}, 122 | {path: '/admin/account/edit', component: AccountEdit}, 123 | {path: '/user/setpwd', component: Profile}, 124 | {path: '/login', component: Login} 125 | ]; 126 | 127 | var router = new VueRouter({ 128 | routes: routes 129 | }); 130 | 131 | bus.$on('goLogin', () => { 132 | store.commit('setEmail', ''); 133 | store.commit('setRole', 0); 134 | router.push('/login'); 135 | }); 136 | 137 | var initConf = new Promise((resolve) => { 138 | restApi.GET('session?check=1').onsucceed(200, (resp) => { 139 | store.commit('enabledAuth', resp.enabledAuth); 140 | store.commit('setEmail', resp.email); 141 | store.commit('setRole', resp.role); 142 | 143 | restApi.GET('version').onsucceed(200, (resp) => { 144 | store.commit('setVersion', resp); 145 | }).do(); 146 | 147 | restApi.GET('configurations').onsucceed(200, (resp) => { 148 | Vue.use((Vue) => Vue.prototype.$appConfig = resp); 149 | bus.$emit('conf_loaded', resp); 150 | loadNodes(resolve); 151 | setInterval(loadNodes, 60*1000); 152 | 153 | }).onfailed((data, xhr) => { 154 | bus.$emit('error', data ? data : xhr.status + ' ' + xhr.statusText); 155 | resolve(); 156 | }).do(); 157 | }).onfailed((data, xhr) => { 158 | if (xhr.status !== 401) { 159 | bus.$emit('error', data); 160 | } else { 161 | store.commit('enabledAuth', true); 162 | } 163 | router.push('/login'); 164 | resolve() 165 | }).do(); 166 | }) 167 | 168 | initConf.then(() => { 169 | new Vue({ 170 | el: '#app', 171 | render: h => h(App), 172 | router: router 173 | }); 174 | }) 175 | 176 | 177 | 178 | -------------------------------------------------------------------------------- /web/ui/src/vuex/store.js: -------------------------------------------------------------------------------- 1 | import Vue from 'vue'; 2 | import Vuex from 'vuex'; 3 | 4 | Vue.use(Vuex); 5 | 6 | const store = new Vuex.Store({ 7 | state: { 8 | version: '', 9 | enabledAuth: false, 10 | user: { 11 | email: '', 12 | role: 0 13 | }, 14 | nodes: {}, 15 | showWithHostname: false 16 | }, 17 | 18 | getters: { 19 | version: function (state) { 20 | return state.version; 21 | }, 22 | 23 | email: function (state) { 24 | return state.user.email; 25 | }, 26 | 27 | role: function (state) { 28 | return state.user.role; 29 | }, 30 | 31 | enabledAuth: function (state) { 32 | return state.enabledAuth; 33 | }, 34 | 35 | nodes: function (state) { 36 | return state.nodes; 37 | }, 38 | 39 | showWithHostname: function (state) { 40 | return state.showWithHostname; 41 | }, 42 | 43 | hostshows: function (state) { 44 | return (id) => _hostshows(id, state, true); 45 | }, 46 | 47 | hostshowsWithoutTip: function (state) { 48 | return (id) => _hostshows(id, state, false); 49 | }, 50 | 51 | getNodeByID: function (state) { 52 | return (id) => { 53 | return state.nodes[id] 54 | } 55 | }, 56 | 57 | dropdownNodes: function (state) { 58 | var dn = []; 59 | var nodes = state.nodes; 60 | for (var i in nodes) { 61 | dn.push({ 62 | value: nodes[i].id, 63 | name: _hostshows(nodes[i].id, state, true) 64 | }); 65 | } 66 | return dn; 67 | } 68 | }, 69 | 70 | mutations: { 71 | setVersion: function (state, v) { 72 | state.version = v; 73 | }, 74 | 75 | setEmail: function (state, email) { 76 | state.user.email = email; 77 | }, 78 | 79 | setRole: function (state, role) { 80 | state.user.role = role; 81 | }, 82 | 83 | enabledAuth: function (state, enabledAuth) { 84 | state.enabledAuth = enabledAuth; 85 | }, 86 | 87 | setNodes: function (state, nodes) { 88 | state.nodes = nodes; 89 | }, 90 | 91 | setShowWithHostname: function (state, b) { 92 | state.showWithHostname = b; 93 | } 94 | } 95 | }) 96 | 97 | function _hostshows(id, state, tip) { 98 | if (!state.nodes[id]) { 99 | if (tip) id += '(node not found)'; 100 | return id; 101 | } 102 | 103 | var show = state.showWithHostname ? state.nodes[id].hostname : state.nodes[id].ip; 104 | if (!show) { 105 | show = id 106 | if (tip) show += '(need to upgrade)'; 107 | } 108 | return show; 109 | } 110 | 111 | export default store 112 | -------------------------------------------------------------------------------- /web/ui/webpack.config.js: -------------------------------------------------------------------------------- 1 | var path = require('path') 2 | var webpack = require('webpack') 3 | 4 | var fontPublicPath = process.env.NODE_ENV === 'production' ? '/ui/' : ''; 5 | module.exports = { 6 | entry: './src/main.js', 7 | output: { 8 | path: path.resolve(__dirname, './dist'), 9 | publicPath: '/', 10 | filename: 'build.js' 11 | }, 12 | module: { 13 | rules: [ 14 | { 15 | test: /\.vue$/, 16 | loader: 'vue-loader', 17 | options: { 18 | loaders: { 19 | // Since sass-loader (weirdly) has SCSS as its default parse mode, we map 20 | // the "scss" and "sass" values for the lang attribute to the right configs here. 21 | // other preprocessors should work out of the box, no loader config like this nessessary. 22 | 'scss': 'vue-style-loader!css-loader!sass-loader', 23 | 'sass': 'vue-style-loader!css-loader!sass-loader?indentedSyntax' 24 | } 25 | // other vue-loader options go here 26 | } 27 | }, 28 | { 29 | test: /\.js$/, 30 | loader: 'babel-loader', 31 | exclude: /node_modules/ 32 | }, 33 | { 34 | test: /\.(png|jpg|gif|svg|ttf|woff|woff2|eot)\w*/, 35 | loader: 'file-loader', 36 | options: { 37 | publicPath: fontPublicPath, 38 | name: '[name].[ext]?[hash]' 39 | } 40 | }, 41 | { 42 | test: /\.css$/, 43 | loader: 'style-loader!css-loader' 44 | } 45 | ] 46 | }, 47 | resolve: { 48 | alias: { 49 | 'vue$': 'vue/dist/vue.common.js', 50 | 'semantic$': 'semantic-ui/dist/semantic.min.js', 51 | 'semanticcss$': 'semantic-ui/dist/semantic.min.css', 52 | 'charts$': 'chart.js/dist/Chart.min.js' 53 | } 54 | }, 55 | devServer: { 56 | proxy: { 57 | '/v1': { 58 | target: 'http://127.0.0.1:7079', 59 | secure: false 60 | } 61 | }, 62 | historyApiFallback: true, 63 | noInfo: true 64 | }, 65 | performance: { 66 | hints: false 67 | }, 68 | devtool: '#eval-source-map' 69 | } 70 | 71 | if (process.env.NODE_ENV === 'production') { 72 | module.exports.devtool = '#source-map' 73 | // http://vue-loader.vuejs.org/en/workflow/production.html 74 | module.exports.plugins = (module.exports.plugins || []).concat([ 75 | new webpack.DefinePlugin({ 76 | 'process.env': { 77 | NODE_ENV: '"production"' 78 | } 79 | }), 80 | new webpack.optimize.UglifyJsPlugin({ 81 | sourceMap: true, 82 | compress: { 83 | warnings: false 84 | } 85 | }), 86 | new webpack.LoaderOptionsPlugin({ 87 | minimize: true 88 | }) 89 | ]) 90 | } 91 | --------------------------------------------------------------------------------