├── .github └── workflows │ └── test.yml ├── Dockerfile ├── LICENSE ├── Makefile ├── README.md ├── cmd └── go-mysql-mongodb │ └── main.go ├── etc └── river.toml ├── go.mod ├── go.sum ├── mongodb ├── client.go └── client_test.go ├── river ├── config.go ├── master.go ├── river.go ├── river_test.go ├── rule.go ├── status.go └── sync.go └── tests ├── check_contains ├── river.toml └── run.sh /.github/workflows/test.yml: -------------------------------------------------------------------------------- 1 | name: test 2 | 3 | on: 4 | push: 5 | branches: [ master ] 6 | pull_request: 7 | branches: [ master ] 8 | 9 | jobs: 10 | 11 | build: 12 | runs-on: ubuntu-latest 13 | steps: 14 | - uses: actions/checkout@v2 15 | 16 | - name: Set up Go 17 | uses: actions/setup-go@v2 18 | with: 19 | go-version: 1.15 20 | 21 | - name: Setup MySQL 22 | uses: shogo82148/actions-setup-mysql@v1 23 | with: 24 | mysql-version: 5.7 25 | user: test 26 | password: secret 27 | my-cnf: | 28 | binlog_format=row 29 | log-bin=mysql-bin 30 | server-id=1 31 | 32 | - name: Setup MongoDB 33 | uses: supercharge/mongodb-github-action@1.3.0 34 | 35 | - name: Unit Test 36 | run: | 37 | mysql --user 'root' --host '127.0.0.1' -e 'create database test;' 38 | GOMODULE=1 go test --race ./... 39 | 40 | - name: Integration Test 41 | run: | 42 | make build 43 | make integration-test 44 | 45 | 46 | -------------------------------------------------------------------------------- /Dockerfile: -------------------------------------------------------------------------------- 1 | FROM golang:alpine 2 | 3 | MAINTAINER WangXiangUSTC 4 | 5 | COPY . /go/src/github.com/WangXiangUSTC/go-mysql-mongodb 6 | 7 | RUN cd /go/src/github.com/siddontang/go-mysql-mongodb/ && \ 8 | go build -o bin/go-mysql-mongodb ./cmd/go-mysql-mongodb && \ 9 | cp -f ./bin/go-mysql-mongodb /go/bin/go-mysql-mongodb 10 | 11 | ENTRYPOINT ["go-mysql-mongodb"] 12 | 13 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | The MIT License (MIT) 2 | 3 | Copyright (c) 2015 siddontang 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | all: build 2 | 3 | build: build-go-mysql-mongodb 4 | 5 | build-go-mysql-mongodb: 6 | go build -o bin/go-mysql-mongodb ./cmd/go-mysql-mongodb 7 | 8 | unit-test: 9 | go test -timeout 1m --race ./... 10 | 11 | integration-test: 12 | ./tests/run.sh 13 | 14 | clean: 15 | go clean -i ./... 16 | @rm -rf bin 17 | 18 | 19 | update_vendor: 20 | which glide >/dev/null || curl https://glide.sh/get | sh 21 | which glide-vc || go get -v -u github.com/sgotti/glide-vc 22 | glide --verbose update --strip-vendor --skip-test 23 | @echo "removing test files" 24 | glide vc --only-code --no-tests 25 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # go-mysql-mongodb ![test](https://github.com/WangXiangUSTC/go-mysql-mongodb/workflows/test/badge.svg) 2 | 3 | go-mysql-mongodb is a service syncing your MySQL data into MongoDB automatically. 4 | 5 | It uses `mysqldump` to fetch the origin data at first, then syncs data incrementally with binlog. 6 | 7 | ## Install 8 | 9 | + Install Go (1.6+) and set your [GOPATH](https://golang.org/doc/code.html#GOPATH) 10 | + `go get github.com/WangXiangUSTC/go-mysql-mongodb`, it will print some messages in console, skip it. :-) 11 | + cd `$GOPATH/src/github.com/WangXiangUSTC/go-mysql-mongodb` 12 | + `make` 13 | 14 | ## How to use? 15 | 16 | + Create tables in MySQL. 17 | + Config base, see the example config [river.toml](./etc/river.toml). 18 | + Set MySQL source in the config file, see [Source](#source) below. 19 | + Customize MySQL and MongoDB mapping rule in the config file, see [Rule](#rule) below. 20 | + Start `./bin/go-mysql-mongodb -config=./etc/river.toml` and enjoy it. 21 | 22 | ## Notice 23 | 24 | + binlog format must be **row**. 25 | + binlog row image must be **full** for MySQL, you may lose some field data if you update PK data in MySQL with minimal or noblob binlog row image. MariaDB only supports full row image. 26 | + Can not alter table format at runtime. 27 | + MySQL table which will be synced should have a PK(primary key), multi-columns PK is allowed now, e,g, if the PKs is (a, b), we will use "a:b" as the key. The PK data will be used as "\_id" in MongoDB. And you can also config the id's constituent part with other columns. 28 | + `mysqldump` must exist in the same node with go-mysql-mongodb, if not, go-mysql-mongodb will try to sync binlog only. 29 | + Don't change too many rows at the same time in one SQL. 30 | 31 | ## Source 32 | 33 | In go-mysql-mongodb, you must decide which tables you want to sync into MongoDB in the source config. 34 | 35 | The format in config file is below: 36 | 37 | ``` 38 | [[source]] 39 | schema = "test" 40 | tables = ["t1", t2] 41 | 42 | [[source]] 43 | schema = "test_1" 44 | tables = ["t3", t4] 45 | ``` 46 | 47 | `schema` is the database name, and `tables` include the table that need to be synced. 48 | 49 | ## Rule 50 | 51 | By default, go-mysql-mongodb will use MySQL table name as the MongoDB's database and collection name, use MySQL table field name as the MongoDB's field name. 52 | e.g, if a table is named blog, the default database and collection in MongoDB are both named blog, if the table field is named title, 53 | the default field name is also named title. 54 | 55 | Rule can let you change this name mapping. Rule format in config file is below: 56 | 57 | ``` 58 | [[rule]] 59 | schema = "test" 60 | table = "t1" 61 | database = "t" 62 | collection = "t" 63 | 64 | [rule.field] 65 | mysql = "title" 66 | mongodb = "my_title" 67 | ``` 68 | 69 | In the example above, we will use a new database and collection both named "t" instead of the default "t1", and use "my_title" instead of the field name "title". 70 | 71 | ## Rule field types 72 | 73 | In order to map a mysql column on different mongodb types you can define the field type as follows: 74 | 75 | ``` 76 | [[rule]] 77 | schema = "test" 78 | table = "t1" 79 | database = "t" 80 | collection = "t" 81 | 82 | [rule.field] 83 | // This will map column title to mongodb search my_title 84 | title="my_title" 85 | 86 | // This will map column title to mongodb search my_title and use the array type 87 | title="my_title,list" 88 | 89 | // This will map column title to mongodb search title and use the array type 90 | title=",list" 91 | ``` 92 | 93 | Modifier "list" will translates a MySQL string field like "a,b,c" on a MongoDB array type '{"a", "b", "c"}' this is especially useful if you need to use those fields on filtering on MongoDB. 94 | 95 | ## Wildcard table 96 | 97 | go-mysql-mongodb only allows you determine which table to be synced, but sometimes, if you split a big table into multi sub tables, like 1024, table_0000, table_0001, ... table_1023, it is very hard to write rules for every table. 98 | 99 | go-mysql-mongodb supports using wildcard table, e.g: 100 | 101 | ``` 102 | [[source]] 103 | schema = "test" 104 | tables = ["test_river_[0-9]{4}"] 105 | 106 | [[rule]] 107 | schema = "test" 108 | table = "test_river_[0-9]{4}" 109 | database = "river" 110 | collection = "river" 111 | ``` 112 | 113 | "test_river_[0-9]{4}" is a wildcard table definition, which represents "test_river_0000" to "test_river_9999", at the same time, the table in the rule must be the same as it. 114 | 115 | In the above example, if you have 1024 sub tables, all tables will be synced into MongoDB with database "river" and collection "river". 116 | 117 | 118 | ## Filter fields 119 | 120 | You can use `filter` to sync specified fields, like: 121 | 122 | ``` 123 | [[rule]] 124 | schema = "test" 125 | table = "tfilter" 126 | database = "test" 127 | collection = "tfilter" 128 | 129 | # Only sync following columns 130 | filter = ["id", "name"] 131 | ``` 132 | 133 | In the above example, we will only sync MySQL table tfiler's columns `id` and `name` to MongoDB. 134 | 135 | ## Why write this tool? 136 | At first, I use [tungsten-replicator](https://github.com/vmware/tungsten-replicator) to synchronize MySQL data to MongoDB, but I found this tool more cumbersome, especially when initializing data at the beginning, and needed to deploy at least two services(one master and one slave). Later, I use [go-mysql-elasticsearch](https://github.com/siddontang/go-mysql-elasticsearch) to sync MySQL data to Elasticsearch, I found this tool is very simple to use. So I rewrite this tool to synchronize MySQL data to MongoDB, and named it `go-mysql-mongodb`. 137 | 138 | 139 | ## Feedback 140 | 141 | go-mysql-mongodb is still in development, and we will try to use it in production later. Any feedback is very welcome. 142 | -------------------------------------------------------------------------------- /cmd/go-mysql-mongodb/main.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "flag" 5 | "os" 6 | "os/signal" 7 | "runtime" 8 | "syscall" 9 | 10 | "github.com/WangXiangUSTC/go-mysql-mongodb/river" 11 | "github.com/juju/errors" 12 | "github.com/ngaut/log" 13 | ) 14 | 15 | var configFile = flag.String("config", "./etc/river.toml", "go-mysql-mongodb config file") 16 | var my_addr = flag.String("my_addr", "", "MySQL addr") 17 | var my_user = flag.String("my_user", "", "MySQL user") 18 | var my_pass = flag.String("my_pass", "", "MySQL password") 19 | var mongo_addr = flag.String("mongo_addr", "", "MongoDB addr") 20 | var data_dir = flag.String("data_dir", "", "path for go-mysql-mongodb to save data") 21 | var server_id = flag.Int("server_id", 0, "MySQL server id, as a pseudo slave") 22 | var flavor = flag.String("flavor", "", "flavor: mysql or mariadb") 23 | var execution = flag.String("exec", "", "mysqldump execution path") 24 | var logLevel = flag.String("log_level", "info", "log level") 25 | 26 | func main() { 27 | runtime.GOMAXPROCS(runtime.NumCPU()) 28 | flag.Parse() 29 | 30 | log.SetLevelByString(*logLevel) 31 | 32 | sc := make(chan os.Signal, 1) 33 | signal.Notify(sc, 34 | os.Kill, 35 | os.Interrupt, 36 | syscall.SIGHUP, 37 | syscall.SIGINT, 38 | syscall.SIGTERM, 39 | syscall.SIGQUIT) 40 | 41 | cfg, err := river.NewConfigWithFile(*configFile) 42 | if err != nil { 43 | println(errors.ErrorStack(err)) 44 | return 45 | } 46 | 47 | if len(*my_addr) > 0 { 48 | cfg.MyAddr = *my_addr 49 | } 50 | 51 | if len(*my_user) > 0 { 52 | cfg.MyUser = *my_user 53 | } 54 | 55 | if len(*my_pass) > 0 { 56 | cfg.MyPassword = *my_pass 57 | } 58 | 59 | if *server_id > 0 { 60 | cfg.ServerID = uint32(*server_id) 61 | } 62 | 63 | if len(*mongo_addr) > 0 { 64 | cfg.MongoAddr = *mongo_addr 65 | } 66 | 67 | if len(*data_dir) > 0 { 68 | cfg.DataDir = *data_dir 69 | } 70 | 71 | if len(*flavor) > 0 { 72 | cfg.Flavor = *flavor 73 | } 74 | 75 | if len(*execution) > 0 { 76 | cfg.DumpExec = *execution 77 | } 78 | 79 | r, err := river.NewRiver(cfg) 80 | if err != nil { 81 | println(errors.ErrorStack(err)) 82 | return 83 | } 84 | 85 | r.Start() 86 | 87 | select { 88 | case n := <-sc: 89 | log.Infof("receive signal %v, closing", n) 90 | case <-r.Ctx().Done(): 91 | log.Infof("context is done with %v, closing", r.Ctx().Err()) 92 | } 93 | 94 | r.Close() 95 | } 96 | -------------------------------------------------------------------------------- /etc/river.toml: -------------------------------------------------------------------------------- 1 | # MySQL address, user and password 2 | # user must have replication privilege in MySQL. 3 | my_addr = "127.0.0.1:3306" 4 | my_user = "root" 5 | my_pass = "" 6 | my_charset = "utf8" 7 | 8 | # If you want sync all database data, you need set my_alldb="yes" 9 | my_alldb = "no" 10 | 11 | # MongoDB address 12 | mongo_addr = "127.0.0.1:27017" 13 | mongo_user = "" 14 | mongo_pass = "" 15 | 16 | # Path to store data, like master.info, if not set or empty, 17 | # we must use this to support breakpoint resume syncing. 18 | # TODO: support other storage, like etcd. 19 | data_dir = "./var" 20 | 21 | # Inner Http status address 22 | stat_addr = "127.0.0.1:12800" 23 | 24 | # pseudo server id like a slave 25 | server_id = 1001 26 | 27 | # mysql or mariadb 28 | flavor = "mysql" 29 | 30 | # mysqldump execution path 31 | # if not set or empty, ignore mysqldump. 32 | mysqldump = "mysqldump" 33 | 34 | # minimal items to be inserted in one bulk 35 | bulk_size = 128 36 | 37 | # force flush the pending requests if we don't have enough items >= bulk_size 38 | flush_bulk_time = "200ms" 39 | 40 | # MySQL data source 41 | [[source]] 42 | schema = "test" 43 | 44 | # Only below tables will be synced into MongoDB. 45 | # "t_[0-9]{4}" is a wildcard table format, you can use it if you have many sub tables, like table_0000 - table_1023 46 | # I don't think it is necessary to sync all tables in a database. 47 | tables = ["t", "t_[0-9]{4}", "tfield", "tfilter"] 48 | 49 | # Below is for special rule mapping 50 | 51 | # Very simple example 52 | # 53 | # desc t; 54 | # +-------+--------------+------+-----+---------+-------+ 55 | # | Field | Type | Null | Key | Default | Extra | 56 | # +-------+--------------+------+-----+---------+-------+ 57 | # | id | int(11) | NO | PRI | NULL | | 58 | # | name | varchar(256) | YES | | NULL | | 59 | # +-------+--------------+------+-----+---------+-------+ 60 | # 61 | # The table `t` will be synced to MongoDB database `test` and collection `t`. 62 | [[rule]] 63 | schema = "test" 64 | table = "t" 65 | database = "test" 66 | collection = "t" 67 | 68 | # Wildcard table rule, the wildcard table must be in source tables 69 | # All tables which match the wildcard format will be synced to MongoDB database `test` and collection `t`. 70 | # In this example, all tables must have same schema with above table `t`; 71 | [[rule]] 72 | schema = "test" 73 | table = "t_[0-9]{4}" 74 | database = "test" 75 | collection = "t" 76 | 77 | # Simple field rule 78 | # 79 | # desc tfield; 80 | # +----------+--------------+------+-----+---------+-------+ 81 | # | Field | Type | Null | Key | Default | Extra | 82 | # +----------+--------------+------+-----+---------+-------+ 83 | # | id | int(11) | NO | PRI | NULL | | 84 | # | tags | varchar(256) | YES | | NULL | | 85 | # | keywords | varchar(256) | YES | | NULL | | 86 | # +----------+--------------+------+-----+---------+-------+ 87 | # 88 | [[rule]] 89 | schema = "test" 90 | table = "tfield" 91 | database = "test" 92 | collection = "tfield" 93 | 94 | [rule.field] 95 | # Map column `id` to MongoDB field `mongo_id` 96 | id="mongo_id" 97 | # Map column `tags` to MongoDB field `mongo_tags` with array type 98 | tags="mongo_tags,list" 99 | # Map column `keywords` to MongoDB with array type 100 | keywords=",list" 101 | 102 | # Filter rule 103 | # 104 | # desc tfilter; 105 | # +-------+--------------+------+-----+---------+-------+ 106 | # | Field | Type | Null | Key | Default | Extra | 107 | # +-------+--------------+------+-----+---------+-------+ 108 | # | id | int(11) | NO | PRI | NULL | | 109 | # | c1 | int(11) | YES | | 0 | | 110 | # | c2 | int(11) | YES | | 0 | | 111 | # | name | varchar(256) | YES | | NULL | | 112 | # +-------+--------------+------+-----+---------+-------+ 113 | # 114 | [[rule]] 115 | schema = "test" 116 | table = "tfilter" 117 | database = "test" 118 | collection = "tfilter" 119 | 120 | # Only sync following columns 121 | filter = ["id", "name"] 122 | 123 | # id rule 124 | # 125 | # desc tid_[0-9]{4}; 126 | # +----------+--------------+------+-----+---------+-------+ 127 | # | Field | Type | Null | Key | Default | Extra | 128 | # +----------+--------------+------+-----+---------+-------+ 129 | # | id | int(11) | NO | PRI | NULL | | 130 | # | tag | varchar(256) | YES | | NULL | | 131 | # | desc | varchar(256) | YES | | NULL | | 132 | # +----------+--------------+------+-----+---------+-------+ 133 | # 134 | [[rule]] 135 | schema = "test" 136 | table = "tid_[0-9]{4}" 137 | database = "test" 138 | collection = "t" 139 | # The mongodb doc's _id will be `id`:`tag` 140 | # It is useful for merge muliple table into one type while theses tables have same PK 141 | id = ["id", "tag"] 142 | -------------------------------------------------------------------------------- /go.mod: -------------------------------------------------------------------------------- 1 | module github.com/WangXiangUSTC/go-mysql-mongodb 2 | 3 | go 1.15 4 | 5 | require ( 6 | github.com/BurntSushi/toml v0.3.1 7 | github.com/juju/errors v0.0.0-20200330140219-3fe23663418f 8 | github.com/juju/testing v0.0.0-20201216035041-2be42bba85f3 // indirect 9 | github.com/ngaut/log v0.0.0-20160810023011-cec23d3e10b0 10 | github.com/pingcap/check v0.0.0-20200212061837-5e12011dc712 11 | github.com/satori/go.uuid v1.1.0 // indirect 12 | github.com/siddontang/go v0.0.0-20151227044929-354e14e6c093 13 | github.com/siddontang/go-mysql v0.0.0-20170505012730-ead11cac47bd 14 | golang.org/x/net v0.0.0-20200904194848-62affa334b73 15 | gopkg.in/check.v1 v1.0.0-20200902074654-038fdea0a05b 16 | gopkg.in/mgo.v2 v2.0.0-20190816093944-a6b53ec6cb22 17 | ) 18 | -------------------------------------------------------------------------------- /go.sum: -------------------------------------------------------------------------------- 1 | github.com/BurntSushi/toml v0.3.1 h1:WXkYYl6Yr3qBf1K79EBnL4mak0OimBfB0XUf9Vl28OQ= 2 | github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU= 3 | github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E= 4 | github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= 5 | github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= 6 | github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= 7 | github.com/google/renameio v0.1.0/go.mod h1:KWCgfxg9yswjAJkECMjeO8J8rahYeXnNhOm40UhjYkI= 8 | github.com/juju/ansiterm v0.0.0-20160907234532-b99631de12cf/go.mod h1:UJSiEoRfvx3hP73CvoARgeLjaIOjybY9vj8PUPPFGeU= 9 | github.com/juju/clock v0.0.0-20190205081909-9c5c9712527c/go.mod h1:nD0vlnrUjcjJhqN5WuCWZyzfd5AHZAC9/ajvbSx69xA= 10 | github.com/juju/cmd v0.0.0-20171107070456-e74f39857ca0/go.mod h1:yWJQHl73rdSX4DHVKGqkAip+huBslxRwS8m9CrOLq18= 11 | github.com/juju/collections v0.0.0-20200605021417-0d0ec82b7271/go.mod h1:5XgO71dV1JClcOJE+4dzdn4HrI5LiyKd7PlVG6eZYhY= 12 | github.com/juju/errors v0.0.0-20150916125642-1b5e39b83d18/go.mod h1:W54LbzXuIE0boCoNJfwqpmkKJ1O4TCTZMetAt6jGk7Q= 13 | github.com/juju/errors v0.0.0-20200330140219-3fe23663418f h1:MCOvExGLpaSIzLYB4iQXEHP4jYVU6vmzLNQPdMVrxnM= 14 | github.com/juju/errors v0.0.0-20200330140219-3fe23663418f/go.mod h1:W54LbzXuIE0boCoNJfwqpmkKJ1O4TCTZMetAt6jGk7Q= 15 | github.com/juju/gnuflag v0.0.0-20171113085948-2ce1bb71843d/go.mod h1:2PavIy+JPciBPrBUjwbNvtwB6RQlve+hkpll6QSNmOE= 16 | github.com/juju/httpprof v0.0.0-20141217160036-14bf14c30767/go.mod h1:+MaLYz4PumRkkyHYeXJ2G5g5cIW0sli2bOfpmbaMV/g= 17 | github.com/juju/loggo v0.0.0-20170605014607-8232ab8918d9/go.mod h1:vgyd7OREkbtVEN/8IXZe5Ooef3LQePvuBm9UWj6ZL8U= 18 | github.com/juju/loggo v0.0.0-20200526014432-9ce3a2e09b5e h1:FdDd7bdI6cjq5vaoYlK1mfQYfF9sF2VZw8VEZMsl5t8= 19 | github.com/juju/loggo v0.0.0-20200526014432-9ce3a2e09b5e/go.mod h1:vgyd7OREkbtVEN/8IXZe5Ooef3LQePvuBm9UWj6ZL8U= 20 | github.com/juju/mutex v0.0.0-20171110020013-1fe2a4bf0a3a/go.mod h1:Y3oOzHH8CQ0Ppt0oCKJ2JFO81/EsWenH5AEqigLH+yY= 21 | github.com/juju/retry v0.0.0-20151029024821-62c620325291/go.mod h1:OohPQGsr4pnxwD5YljhQ+TZnuVRYpa5irjugL1Yuif4= 22 | github.com/juju/retry v0.0.0-20180821225755-9058e192b216/go.mod h1:OohPQGsr4pnxwD5YljhQ+TZnuVRYpa5irjugL1Yuif4= 23 | github.com/juju/testing v0.0.0-20180402130637-44801989f0f7/go.mod h1:63prj8cnj0tU0S9OHjGJn+b1h0ZghCndfnbQolrYTwA= 24 | github.com/juju/testing v0.0.0-20190723135506-ce30eb24acd2/go.mod h1:63prj8cnj0tU0S9OHjGJn+b1h0ZghCndfnbQolrYTwA= 25 | github.com/juju/testing v0.0.0-20201216035041-2be42bba85f3 h1:ram7cW6jDPTu6cv9xDMwa+tO7RsO4BdsubxrJ4EEw+E= 26 | github.com/juju/testing v0.0.0-20201216035041-2be42bba85f3/go.mod h1:IbSKFoKW0bzmbDZ7rBwF/L3lO3b1bpmOIhTXQl/WJxw= 27 | github.com/juju/utils v0.0.0-20180424094159-2000ea4ff043/go.mod h1:6/KLg8Wz/y2KVGWEpkK9vMNGkOnu4k/cqs8Z1fKjTOk= 28 | github.com/juju/utils v0.0.0-20200116185830-d40c2fe10647/go.mod h1:6/KLg8Wz/y2KVGWEpkK9vMNGkOnu4k/cqs8Z1fKjTOk= 29 | github.com/juju/utils/v2 v2.0.0-20200923005554-4646bfea2ef1/go.mod h1:fdlDtQlzundleLLz/ggoYinEt/LmnrpNKcNTABQATNI= 30 | github.com/juju/version v0.0.0-20161031051906-1f41e27e54f2/go.mod h1:kE8gK5X0CImdr7qpSKl3xB2PmpySSmfj7zVbkZFs81U= 31 | github.com/juju/version v0.0.0-20180108022336-b64dbd566305/go.mod h1:kE8gK5X0CImdr7qpSKl3xB2PmpySSmfj7zVbkZFs81U= 32 | github.com/juju/version v0.0.0-20191219164919-81c1be00b9a6/go.mod h1:kE8gK5X0CImdr7qpSKl3xB2PmpySSmfj7zVbkZFs81U= 33 | github.com/julienschmidt/httprouter v1.1.1-0.20151013225520-77a895ad01eb/go.mod h1:SYymIcj16QtmaHHD7aYtjjsJG7VTCxuUUipMqKk8s4w= 34 | github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck= 35 | github.com/kr/pretty v0.1.0 h1:L/CwN0zerZDmRFUapSPitk6f+Q3+0za1rQkzVuMiMFI= 36 | github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo= 37 | github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ= 38 | github.com/kr/text v0.1.0 h1:45sCR5RtlFHMR4UwH9sdQ5TC8v0qDQCHnXt+kaKSTVE= 39 | github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI= 40 | github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY= 41 | github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE= 42 | github.com/lunixbochs/vtclean v0.0.0-20160125035106-4fbf7632a2c6/go.mod h1:pHhQNgMf3btfWnGBVipUOjRYhoOsdGqdm/+2c2E2WMI= 43 | github.com/masterzen/azure-sdk-for-go v3.2.0-beta.0.20161014135628-ee4f0065d00c+incompatible/go.mod h1:mf8fjOu33zCqxUjuiU3I8S1lJMyEAlH+0F2+M5xl3hE= 44 | github.com/masterzen/simplexml v0.0.0-20160608183007-4572e39b1ab9/go.mod h1:kCEbxUJlNDEBNbdQMkPSp6yaKcRXVI6f4ddk8Riv4bc= 45 | github.com/masterzen/winrm v0.0.0-20161014151040-7a535cd943fc/go.mod h1:CfZSN7zwz5gJiFhZJz49Uzk7mEBHIceWmbFmYx7Hf7E= 46 | github.com/masterzen/xmlpath v0.0.0-20140218185901-13f4951698ad/go.mod h1:A0zPC53iKKKcXYxr4ROjpQRQ5FgJXtelNdSmHHuq/tY= 47 | github.com/mattn/go-colorable v0.0.6/go.mod h1:9vuHe8Xs5qXnSaW/c/ABM9alt+Vo+STaOChaDxuIBZU= 48 | github.com/mattn/go-isatty v0.0.0-20160806122752-66b8e73f3f5c/go.mod h1:M+lRXTBqGeGNdLjl/ufCoiOlB5xdOkqRJdNxMWT7Zi4= 49 | github.com/ngaut/log v0.0.0-20160810023011-cec23d3e10b0 h1:yAdflNJJ0W/AGi5dapdvp9jZHnkGV6ZOlW1A3z/oTY8= 50 | github.com/ngaut/log v0.0.0-20160810023011-cec23d3e10b0/go.mod h1:ueVCjKQllPmX7uEvCYnZD5b8qjidGf1TCH61arVe4SU= 51 | github.com/niemeyer/pretty v0.0.0-20200227124842-a10e7caefd8e h1:fD57ERR4JtEqsWbfPhv4DMiApHyliiK5xCTNVSPiaAs= 52 | github.com/niemeyer/pretty v0.0.0-20200227124842-a10e7caefd8e/go.mod h1:zD1mROLANZcx1PVRCS0qkT7pwLkGfwJo4zjcN/Tysno= 53 | github.com/nu7hatch/gouuid v0.0.0-20131221200532-179d4d0c4d8d/go.mod h1:YUTz3bUH2ZwIWBy3CJBeOBEugqcmXREj14T+iG/4k4U= 54 | github.com/pingcap/check v0.0.0-20190102082844-67f458068fc8/go.mod h1:B1+S9LNcuMyLH/4HMTViQOJevkGiik3wW2AN9zb2fNQ= 55 | github.com/pingcap/check v0.0.0-20200212061837-5e12011dc712 h1:R8gStypOBmpnHEx1qi//SaqxJVI4inOqljg/Aj5/390= 56 | github.com/pingcap/check v0.0.0-20200212061837-5e12011dc712/go.mod h1:PYMCGwN0JHjoqGr3HrZoD+b8Tgx8bKnArhSq8YVzUMc= 57 | github.com/pingcap/errors v0.11.0 h1:DCJQB8jrHbQ1VVlMFIrbj2ApScNNotVmkSNplu2yUt4= 58 | github.com/pingcap/errors v0.11.0/go.mod h1:Oi8TUi2kEtXXLMJk9l1cGmz20kV3TaQ0usTwv5KuLY8= 59 | github.com/pingcap/log v0.0.0-20191012051959-b742a5d432e9 h1:AJD9pZYm72vMgPcQDww9rkZ1DnWfl0pXV3BOWlkYIjA= 60 | github.com/pingcap/log v0.0.0-20191012051959-b742a5d432e9/go.mod h1:4rbK1p9ILyIfb6hU7OG2CiWSqMXnp3JMbiaVJ6mvoY8= 61 | github.com/pkg/errors v0.8.1 h1:iURUrRGxPUNPdy5/HRSm+Yj6okJ6UtLINN0Q9M4+h3I= 62 | github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= 63 | github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= 64 | github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= 65 | github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4= 66 | github.com/satori/go.uuid v1.1.0 h1:B9KXyj+GzIpJbV7gmr873NsY6zpbxNy24CBtGrk7jHo= 67 | github.com/satori/go.uuid v1.1.0/go.mod h1:dA0hQrYB0VpLJoorglMZABFdXlWrHn1NEOzdhQKdks0= 68 | github.com/siddontang/go v0.0.0-20151227044929-354e14e6c093 h1:v+V9ov2DhGtWxlJ5btyC+wC+Kyvd6BoxbR/60uFZRYw= 69 | github.com/siddontang/go v0.0.0-20151227044929-354e14e6c093/go.mod h1:3yhqj7WBBfRhbBlzyOC3gUxftwsU0u8gqevxwIHQpMw= 70 | github.com/siddontang/go-mysql v0.0.0-20170505012730-ead11cac47bd h1:xjMfGXbn1q64NRCiKAhAOpoTCY8V6G7MdpUoiVYflQs= 71 | github.com/siddontang/go-mysql v0.0.0-20170505012730-ead11cac47bd/go.mod h1:wzjXB9ICbSvAycqKa006qypXiHt48N2SJMblRU9sCrg= 72 | github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= 73 | github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= 74 | github.com/stretchr/testify v1.4.0 h1:2E4SXV/wtOkTonXsotYi4li6zVWxYlZuYNCXe9XRJyk= 75 | github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4= 76 | go.uber.org/atomic v1.3.2/go.mod h1:gD2HeocX3+yG+ygLZcrzQJaqmWj9AIm7n08wl/qW/PE= 77 | go.uber.org/atomic v1.5.0 h1:OI5t8sDa1Or+q8AeE+yKeB/SDYioSHAgcVljj9JIETY= 78 | go.uber.org/atomic v1.5.0/go.mod h1:sABNBOSYdrvTF6hTgEIbc7YasKWGhgEQZyfxyTvoXHQ= 79 | go.uber.org/multierr v1.1.0/go.mod h1:wR5kodmAFQ0UK8QlbwjlSNy0Z68gJhDJUG5sjR94q/0= 80 | go.uber.org/multierr v1.3.0/go.mod h1:VgVr7evmIr6uPjLBxg28wmKNXyqE9akIJ5XnfpiKl+4= 81 | go.uber.org/multierr v1.4.0 h1:f3WCSC2KzAcBXGATIxAB1E2XuCpNU255wNKZ505qi3E= 82 | go.uber.org/multierr v1.4.0/go.mod h1:VgVr7evmIr6uPjLBxg28wmKNXyqE9akIJ5XnfpiKl+4= 83 | go.uber.org/tools v0.0.0-20190618225709-2cfd321de3ee h1:0mgffUl7nfd+FpvXMVz4IDEaUSmT1ysygQC7qYo7sG4= 84 | go.uber.org/tools v0.0.0-20190618225709-2cfd321de3ee/go.mod h1:vJERXedbb3MVM5f9Ejo0C68/HhF8uaILCdgjnY+goOA= 85 | go.uber.org/zap v1.9.1/go.mod h1:vwi/ZaCAaUcBkycHslxD9B2zi4UTXhF60s6SWpuDF0Q= 86 | go.uber.org/zap v1.12.0 h1:dySoUQPFBGj6xwjmBzageVL8jGi8uxc6bEmJQjA06bw= 87 | go.uber.org/zap v1.12.0/go.mod h1:zwrFLgMcdUuIBviXEYEH1YKNaOBnKXsx2IPda5bBwHM= 88 | golang.org/x/crypto v0.0.0-20180214000028-650f4a345ab4/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= 89 | golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= 90 | golang.org/x/crypto v0.0.0-20190510104115-cbcb75029529/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= 91 | golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= 92 | golang.org/x/crypto v0.0.0-20200820211705-5c72a883971a/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= 93 | golang.org/x/lint v0.0.0-20190930215403-16217165b5de h1:5hukYrvBGR8/eNkX5mdUezrA6JiaEZDtJb9Ei+1LlBs= 94 | golang.org/x/lint v0.0.0-20190930215403-16217165b5de/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc= 95 | golang.org/x/mod v0.0.0-20190513183733-4bf6d317e70e/go.mod h1:mXi4GBBbnImb6dmsKGUJ2LatrhH/nqhxcFungHvyanc= 96 | golang.org/x/net v0.0.0-20180406214816-61147c48b25b/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= 97 | golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= 98 | golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= 99 | golang.org/x/net v0.0.0-20190620200207-3b0461eec859 h1:R/3boaszxrf1GEUWTVDzSKVwLmSJpwZ1yqXm8j0v2QI= 100 | golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= 101 | golang.org/x/net v0.0.0-20200904194848-62affa334b73 h1:MXfv8rhZWmFeqX3GNZRsd6vOLoaCHjYEX3qkRo3YBUA= 102 | golang.org/x/net v0.0.0-20200904194848-62affa334b73/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA= 103 | golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= 104 | golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= 105 | golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 106 | golang.org/x/sys v0.0.0-20200323222414-85ca7c5b95cd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 107 | golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= 108 | golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs= 109 | golang.org/x/tools v0.0.0-20190621195816-6e04913cbbac/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc= 110 | golang.org/x/tools v0.0.0-20191029041327-9cc4af7d6b2c/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= 111 | golang.org/x/tools v0.0.0-20191029190741-b9c20aec41a5/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= 112 | golang.org/x/tools v0.0.0-20191107010934-f79515f33823 h1:akkRBeitX2EZP59KdtKw310CI4WGPCNPyrLbE7WZA8Y= 113 | golang.org/x/tools v0.0.0-20191107010934-f79515f33823/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= 114 | golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= 115 | gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= 116 | gopkg.in/check.v1 v1.0.0-20160105164936-4f90aeace3a2/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= 117 | gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127 h1:qIbj1fsPNlZgppZ+VLlY7N33q108Sa+fhmuc+sWQYwY= 118 | gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= 119 | gopkg.in/check.v1 v1.0.0-20200902074654-038fdea0a05b h1:QRR6H1YWRnHb4Y/HeNFCTJLFVxaq6wH4YuVdsUOr75U= 120 | gopkg.in/check.v1 v1.0.0-20200902074654-038fdea0a05b/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= 121 | gopkg.in/errgo.v1 v1.0.0-20161222125816-442357a80af5/go.mod h1:u0ALmqvLRxLI95fkdCEWrE6mhWYZW1aMOJHp5YXLHTg= 122 | gopkg.in/errgo.v2 v2.1.0/go.mod h1:hNsd1EY+bozCKY1Ytp96fpM3vjJbqLJn88ws8XvfDNI= 123 | gopkg.in/httprequest.v1 v1.1.1/go.mod h1:/CkavNL+g3qLOrpFHVrEx4NKepeqR4XTZWNj4sGGjz0= 124 | gopkg.in/mgo.v2 v2.0.0-20160818015218-f2b6f6c918c4/go.mod h1:yeKp02qBN3iKW1OzL3MGk2IdtZzaj7SFntXj72NppTA= 125 | gopkg.in/mgo.v2 v2.0.0-20190816093944-a6b53ec6cb22 h1:VpOs+IwYnYBaFnrNAeB8UUWtL3vEUnzSCL1nVjPhqrw= 126 | gopkg.in/mgo.v2 v2.0.0-20190816093944-a6b53ec6cb22/go.mod h1:yeKp02qBN3iKW1OzL3MGk2IdtZzaj7SFntXj72NppTA= 127 | gopkg.in/natefinch/lumberjack.v2 v2.0.0 h1:1Lc07Kr7qY4U2YPouBjpCLxpiyxIVoxqXgkXLknAOE8= 128 | gopkg.in/natefinch/lumberjack.v2 v2.0.0/go.mod h1:l0ndWWf7gzL7RNwBG7wST/UCcT4T24xpD6X8LsfU/+k= 129 | gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7/go.mod h1:dt/ZhP58zS4L8KSrWDmTeBkI65Dw0HsyUHuEVlX15mw= 130 | gopkg.in/yaml.v2 v2.0.0-20170712054546-1be3d31502d6/go.mod h1:JAlM8MvJe8wmxCU4Bli9HhUf9+ttbYbLASfIpnQbh74= 131 | gopkg.in/yaml.v2 v2.2.2 h1:ZCJp+EgiOT7lHqUV2J862kp8Qj64Jo6az82+3Td9dZw= 132 | gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= 133 | gopkg.in/yaml.v2 v2.3.0 h1:clyUAQHOM3G0M3f5vQj7LuJrETvjVot3Z5el9nffUtU= 134 | gopkg.in/yaml.v2 v2.3.0/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= 135 | honnef.co/go/tools v0.0.1-2019.2.3 h1:3JgtbtFHMiCmsznwGVTUWbgGov+pVqnlf1dEJTNAXeM= 136 | honnef.co/go/tools v0.0.1-2019.2.3/go.mod h1:a3bituU0lyd329TUQxRnasdCoJDkEUEAqEt0JzvZhAg= 137 | launchpad.net/gocheck v0.0.0-20140225173054-000000000087/go.mod h1:hj7XX3B/0A+80Vse0e+BUHsHMTEhd0O4cpUHr/e/BUM= 138 | launchpad.net/xmlpath v0.0.0-20130614043138-000000000004/go.mod h1:vqyExLOM3qBx7mvYRkoxjSCF945s0mbe7YynlKYXtsA= 139 | -------------------------------------------------------------------------------- /mongodb/client.go: -------------------------------------------------------------------------------- 1 | package mongodb 2 | 3 | import ( 4 | "fmt" 5 | //"github.com/ngaut/log" 6 | "gopkg.in/mgo.v2" 7 | "gopkg.in/mgo.v2/bson" 8 | ) 9 | 10 | type Client struct { 11 | Addr string 12 | Username string 13 | Password string 14 | c *mgo.Session 15 | } 16 | 17 | type ClientConfig struct { 18 | Addr string 19 | Username string 20 | Password string 21 | } 22 | 23 | func NewClient(conf *ClientConfig) (*Client, error) { 24 | var err error 25 | 26 | c := new(Client) 27 | c.Addr = conf.Addr 28 | c.Username = conf.Username 29 | c.Password = conf.Password 30 | if len(c.Username) > 0 && len(c.Password) > 0 { 31 | c.c, err = mgo.Dial(fmt.Sprintf("mongodb://%s:%s@%s", c.Username, c.Password, c.Addr)) 32 | } else { 33 | c.c, err = mgo.Dial(fmt.Sprintf("mongodb://%s", c.Addr)) 34 | } 35 | 36 | if err != nil { 37 | return nil, err 38 | } 39 | 40 | return c, nil 41 | } 42 | 43 | type ResponseItem struct { 44 | ID string `json:"_id"` 45 | Database string `json:"_database"` 46 | Collection string `json:"_collection"` 47 | Found bool `json:"found"` 48 | Source map[string]interface{} `json:"_source"` 49 | } 50 | 51 | type Response struct { 52 | Code int 53 | ResponseItem 54 | } 55 | 56 | const ( 57 | ActionCreate = "create" 58 | ActionUpdate = "update" 59 | ActionDelete = "delete" 60 | ActionInsert = "insert" 61 | ) 62 | 63 | type BulkRequest struct { 64 | Action string 65 | Database string 66 | Collection string 67 | ID string 68 | Filter map[string]interface{} 69 | Data map[string]interface{} 70 | } 71 | 72 | type BulkResponse struct { 73 | Code int 74 | Took int `json:"took"` 75 | Errors bool `json:"errors"` 76 | 77 | Items []map[string]*BulkResponseItem `json:"items"` 78 | } 79 | 80 | type BulkResponseItem struct { 81 | Database string `json:"_database"` 82 | Collection string `json:"_collection"` 83 | ID string `json:"_id"` 84 | Status int `json:"status"` 85 | Found bool `json:"found"` 86 | } 87 | 88 | func (c *Client) Bulk(items []*BulkRequest) error { 89 | colDict := map[string]*mgo.Bulk{} 90 | var database string 91 | var collection string 92 | for _, item := range items { 93 | database = item.Database 94 | collection = item.Collection 95 | key := fmt.Sprintf("%s_%s", database, collection) 96 | if _, ok := colDict[key]; ok { 97 | // do nothing 98 | } else { 99 | coll := c.c.DB(database).C(collection) 100 | colDict[key] = coll.Bulk() 101 | } 102 | switch item.Action { 103 | case ActionDelete: 104 | colDict[key].Remove(bson.M{"_id": item.ID}) 105 | case ActionUpdate: 106 | colDict[key].Upsert(bson.M{"_id": item.ID}, bson.M{"$set": item.Data}) 107 | case ActionInsert: 108 | item.Data["_id"] = item.ID 109 | colDict[key].Upsert(bson.M{"_id": item.ID}, bson.M{"$set": item.Data}) 110 | 111 | } 112 | } 113 | for _, v := range colDict { 114 | _, err := v.Run() 115 | if err != nil { 116 | return err 117 | } 118 | } 119 | 120 | return nil 121 | } 122 | 123 | func (c *Client) DeleteDB(database string) error { 124 | db := c.c.DB(database) 125 | if db == nil { 126 | return nil 127 | } 128 | return db.DropDatabase() 129 | } 130 | 131 | // Update creates or updates the data 132 | func (c *Client) Update(database string, collection string, id string, data map[string]interface{}) error { 133 | _, err := c.c.DB(database).C(collection).Upsert(bson.M{"_id": id}, bson.M{"$set": data}) 134 | return err 135 | } 136 | 137 | // Exists checks whether id exists or not. 138 | func (c *Client) Exists(database string, collection string, id string) (bool, error) { 139 | resp, err := c.Get(database, collection, id) 140 | if err != nil { 141 | return false, err 142 | } 143 | 144 | return resp.Found, nil 145 | } 146 | 147 | // Delete deletes the item by id. 148 | func (c *Client) Delete(database string, collection string, id string) error { 149 | return c.c.DB(database).C(collection).Remove(bson.M{"_id": id}) 150 | } 151 | 152 | func (c *Client) Get(database string, collection string, id string) (*Response, error) { 153 | resp := new(Response) 154 | resp.ID = id 155 | resp.Database = database 156 | resp.Collection = collection 157 | 158 | var result [](map[string]interface{}) 159 | err := c.c.DB(database).C(collection).Find(bson.M{"_id": id}).All(&result) 160 | if err != nil { 161 | return nil, err 162 | } 163 | if len(result) == 0 { 164 | resp.Found = false 165 | return resp, nil 166 | } 167 | 168 | resp.Code = 200 169 | resp.Found = true 170 | resp.Source = result[0] 171 | return resp, err 172 | } 173 | -------------------------------------------------------------------------------- /mongodb/client_test.go: -------------------------------------------------------------------------------- 1 | package mongodb 2 | 3 | import ( 4 | "flag" 5 | "fmt" 6 | "testing" 7 | 8 | . "github.com/pingcap/check" 9 | ) 10 | 11 | var host = flag.String("host", "127.0.0.1", "MongoDB host") 12 | var port = flag.Int("port", 27017, "MongoDB port") 13 | 14 | func Test(t *testing.T) { 15 | TestingT(t) 16 | } 17 | 18 | type mongoTestSuite struct { 19 | c *Client 20 | } 21 | 22 | var _ = Suite(&mongoTestSuite{}) 23 | 24 | func (s *mongoTestSuite) SetUpSuite(c *C) { 25 | cfg := new(ClientConfig) 26 | cfg.Addr = fmt.Sprintf("%s:%d", *host, *port) 27 | cfg.Username = "" 28 | cfg.Password = "" 29 | var err error 30 | s.c, err = NewClient(cfg) 31 | c.Assert(err, IsNil) 32 | } 33 | 34 | func (s *mongoTestSuite) TearDownSuite(c *C) { 35 | 36 | } 37 | 38 | func makeTestData(arg1 string, arg2 string) map[string]interface{} { 39 | m := make(map[string]interface{}) 40 | m["name"] = arg1 41 | m["content"] = arg2 42 | 43 | return m 44 | } 45 | 46 | func (s *mongoTestSuite) TestSimple(c *C) { 47 | database := "dummy" 48 | collection := "blog" 49 | 50 | //key1 := "name" 51 | //key2 := "content" 52 | 53 | err := s.c.Update(database, collection, "1", makeTestData("abc", "hello world")) 54 | c.Assert(err, IsNil) 55 | 56 | exists, err := s.c.Exists(database, collection, "1") 57 | c.Assert(err, IsNil) 58 | c.Assert(exists, Equals, true) 59 | 60 | r, err := s.c.Get(database, collection, "1") 61 | c.Assert(err, IsNil) 62 | c.Assert(r.Code, Equals, 200) 63 | c.Assert(r.ID, Equals, "1") 64 | 65 | err = s.c.Delete(database, collection, "1") 66 | c.Assert(err, IsNil) 67 | 68 | exists, err = s.c.Exists(database, collection, "1") 69 | c.Assert(err, IsNil) 70 | c.Assert(exists, Equals, false) 71 | 72 | items := make([]*BulkRequest, 10) 73 | 74 | for i := 0; i < 10; i++ { 75 | id := fmt.Sprintf("%d", i) 76 | req := new(BulkRequest) 77 | req.Action = ActionDelete 78 | req.Database = database 79 | req.Collection = collection 80 | req.ID = id 81 | items[i] = req 82 | } 83 | 84 | err = s.c.Bulk(items) 85 | c.Assert(err, IsNil) 86 | } 87 | -------------------------------------------------------------------------------- /river/config.go: -------------------------------------------------------------------------------- 1 | package river 2 | 3 | import ( 4 | "io/ioutil" 5 | "time" 6 | 7 | "github.com/BurntSushi/toml" 8 | "github.com/juju/errors" 9 | ) 10 | 11 | type SourceConfig struct { 12 | Schema string `toml:"schema"` 13 | Tables []string `toml:"tables"` 14 | } 15 | 16 | type Config struct { 17 | MyAddr string `toml:"my_addr"` 18 | MyUser string `toml:"my_user"` 19 | MyPassword string `toml:"my_pass"` 20 | MyCharset string `toml:"my_charset"` 21 | AllDB string `toml:"my_alldb"` 22 | 23 | MongoAddr string `toml:"mongo_addr"` 24 | MongoUser string `toml:"mongo_user"` 25 | MongoPassword string `toml:"mongo_pass"` 26 | 27 | StatAddr string `toml:"stat_addr"` 28 | 29 | ServerID uint32 `toml:"server_id"` 30 | Flavor string `toml:"flavor"` 31 | DataDir string `toml:"data_dir"` 32 | 33 | DumpExec string `toml:"mysqldump"` 34 | 35 | Sources []SourceConfig `toml:"source"` 36 | 37 | Rules []*Rule `toml:"rule"` 38 | 39 | BulkSize int `toml:"bulk_size"` 40 | 41 | FlushBulkTime TomlDuration `toml:"flush_bulk_time"` 42 | } 43 | 44 | func NewConfigWithFile(name string) (*Config, error) { 45 | data, err := ioutil.ReadFile(name) 46 | if err != nil { 47 | return nil, errors.Trace(err) 48 | } 49 | 50 | return NewConfig(string(data)) 51 | } 52 | 53 | func NewConfig(data string) (*Config, error) { 54 | var c Config 55 | 56 | _, err := toml.Decode(data, &c) 57 | if err != nil { 58 | return nil, errors.Trace(err) 59 | } 60 | 61 | return &c, nil 62 | } 63 | 64 | type TomlDuration struct { 65 | time.Duration 66 | } 67 | 68 | func (d *TomlDuration) UnmarshalText(text []byte) error { 69 | var err error 70 | d.Duration, err = time.ParseDuration(string(text)) 71 | return err 72 | } 73 | -------------------------------------------------------------------------------- /river/master.go: -------------------------------------------------------------------------------- 1 | package river 2 | 3 | import ( 4 | "bytes" 5 | "os" 6 | "path" 7 | "sync" 8 | "time" 9 | 10 | "github.com/BurntSushi/toml" 11 | "github.com/juju/errors" 12 | "github.com/ngaut/log" 13 | "github.com/siddontang/go-mysql/mysql" 14 | "github.com/siddontang/go/ioutil2" 15 | ) 16 | 17 | type masterInfo struct { 18 | sync.RWMutex 19 | 20 | Name string `toml:"bin_name"` 21 | Pos uint32 `toml:"bin_pos"` 22 | 23 | filePath string 24 | lastSaveTime time.Time 25 | } 26 | 27 | func loadMasterInfo(dataDir string) (*masterInfo, error) { 28 | var m masterInfo 29 | 30 | if len(dataDir) == 0 { 31 | return &m, nil 32 | } 33 | 34 | m.filePath = path.Join(dataDir, "master.info") 35 | m.lastSaveTime = time.Now() 36 | 37 | if err := os.MkdirAll(dataDir, 0755); err != nil { 38 | return nil, errors.Trace(err) 39 | } 40 | 41 | f, err := os.Open(m.filePath) 42 | if err != nil && !os.IsNotExist(errors.Cause(err)) { 43 | return nil, errors.Trace(err) 44 | } else if os.IsNotExist(errors.Cause(err)) { 45 | return &m, nil 46 | } 47 | defer f.Close() 48 | 49 | _, err = toml.DecodeReader(f, &m) 50 | return &m, errors.Trace(err) 51 | } 52 | 53 | func (m *masterInfo) Save(pos mysql.Position) error { 54 | log.Infof("save position %s", pos) 55 | 56 | m.Lock() 57 | defer m.Unlock() 58 | 59 | m.Name = pos.Name 60 | m.Pos = pos.Pos 61 | 62 | if len(m.filePath) == 0 { 63 | return nil 64 | } 65 | 66 | n := time.Now() 67 | if n.Sub(m.lastSaveTime) < time.Second { 68 | return nil 69 | } 70 | 71 | m.lastSaveTime = n 72 | var buf bytes.Buffer 73 | e := toml.NewEncoder(&buf) 74 | 75 | e.Encode(m) 76 | 77 | var err error 78 | if err = ioutil2.WriteFileAtomic(m.filePath, buf.Bytes(), 0644); err != nil { 79 | log.Errorf("canal save master info to file %s err %v", m.filePath, err) 80 | } 81 | 82 | return errors.Trace(err) 83 | } 84 | 85 | func (m *masterInfo) Position() mysql.Position { 86 | m.RLock() 87 | defer m.RUnlock() 88 | 89 | return mysql.Position{ 90 | m.Name, 91 | m.Pos, 92 | } 93 | } 94 | 95 | func (m *masterInfo) Close() error { 96 | pos := m.Position() 97 | 98 | return m.Save(pos) 99 | } 100 | -------------------------------------------------------------------------------- /river/river.go: -------------------------------------------------------------------------------- 1 | package river 2 | 3 | import ( 4 | "crypto/md5" 5 | "fmt" 6 | "hash" 7 | "regexp" 8 | "sync" 9 | 10 | "github.com/WangXiangUSTC/go-mysql-mongodb/mongodb" 11 | "github.com/juju/errors" 12 | "github.com/ngaut/log" 13 | "github.com/siddontang/go-mysql/canal" 14 | "golang.org/x/net/context" 15 | ) 16 | 17 | type River struct { 18 | c *Config 19 | 20 | canal *canal.Canal 21 | 22 | rules map[string]*Rule 23 | 24 | md5Ctx hash.Hash 25 | 26 | ctx context.Context 27 | cancel context.CancelFunc 28 | 29 | wg sync.WaitGroup 30 | 31 | mongo *mongodb.Client 32 | 33 | st *stat 34 | 35 | master *masterInfo 36 | 37 | syncCh chan interface{} 38 | } 39 | 40 | func NewRiver(c *Config) (*River, error) { 41 | r := new(River) 42 | 43 | r.c = c 44 | r.rules = make(map[string]*Rule) 45 | r.syncCh = make(chan interface{}, 4096) 46 | r.ctx, r.cancel = context.WithCancel(context.Background()) 47 | 48 | r.md5Ctx = md5.New() 49 | 50 | var err error 51 | if r.master, err = loadMasterInfo(c.DataDir); err != nil { 52 | return nil, errors.Trace(err) 53 | } 54 | 55 | if err = r.newCanal(); err != nil { 56 | return nil, errors.Trace(err) 57 | } 58 | 59 | if err = r.prepareRule(); err != nil { 60 | return nil, errors.Trace(err) 61 | } 62 | 63 | if err = r.prepareCanal(); err != nil { 64 | return nil, errors.Trace(err) 65 | } 66 | 67 | // We must use binlog full row image 68 | if err = r.canal.CheckBinlogRowImage("FULL"); err != nil { 69 | return nil, errors.Trace(err) 70 | } 71 | cfg := new(mongodb.ClientConfig) 72 | cfg.Addr = r.c.MongoAddr 73 | cfg.Username = r.c.MongoUser 74 | cfg.Password = r.c.MongoPassword 75 | r.mongo, err = mongodb.NewClient(cfg) 76 | if err != nil { 77 | return nil, errors.Trace(err) 78 | } 79 | r.st = &stat{r: r} 80 | go r.st.Run(r.c.StatAddr) 81 | 82 | return r, nil 83 | } 84 | 85 | func (r *River) newCanal() error { 86 | cfg := canal.NewDefaultConfig() 87 | cfg.Addr = r.c.MyAddr 88 | cfg.User = r.c.MyUser 89 | cfg.Password = r.c.MyPassword 90 | cfg.Charset = r.c.MyCharset 91 | cfg.Flavor = r.c.Flavor 92 | 93 | cfg.ServerID = r.c.ServerID 94 | cfg.Dump.ExecutionPath = r.c.DumpExec 95 | cfg.Dump.DiscardErr = false 96 | 97 | var err error 98 | r.canal, err = canal.NewCanal(cfg) 99 | return errors.Trace(err) 100 | } 101 | 102 | func (r *River) prepareCanal() error { 103 | if r.c.AllDB == "yes" { 104 | log.Infof("dump all database") 105 | sql := "select SCHEMA_NAME from information_schema.SCHEMATA" 106 | res, err := r.canal.Execute(sql) 107 | if err != nil { 108 | return errors.Trace(err) 109 | } 110 | 111 | for i := 0; i < res.Resultset.RowNumber(); i++ { 112 | db, _ := res.GetString(i, 0) 113 | if db == "information_schema" || db == "performance_schema" || db == "sys" || db == "mysql" { 114 | continue 115 | } else { 116 | r.canal.AddDumpDatabases(db) 117 | } 118 | } 119 | } else { 120 | var db string 121 | dbs := map[string]struct{}{} 122 | tables := make([]string, 0, len(r.rules)) 123 | for _, rule := range r.rules { 124 | db = rule.Schema 125 | dbs[rule.Schema] = struct{}{} 126 | tables = append(tables, rule.Table) 127 | } 128 | 129 | if len(dbs) == 1 { 130 | log.Infof("dump database: %s, tables: %s", db, tables) 131 | // one db, we can shrink using table 132 | r.canal.AddDumpTables(db, tables...) 133 | } else { 134 | // many dbs, can only assign databases to dump 135 | keys := make([]string, 0, len(dbs)) 136 | for key, _ := range dbs { 137 | keys = append(keys, key) 138 | } 139 | log.Infof("dump database: %s", keys) 140 | r.canal.AddDumpDatabases(keys...) 141 | } 142 | } 143 | 144 | r.canal.SetEventHandler(&eventHandler{r}) 145 | 146 | return nil 147 | } 148 | 149 | func (r *River) newRule(schema, table string) error { 150 | key := ruleKey(schema, table) 151 | 152 | if _, ok := r.rules[key]; ok { 153 | return errors.Errorf("duplicate source %s, %s defined in config", schema, table) 154 | } 155 | 156 | r.rules[key] = newDefaultRule(schema, table) 157 | return nil 158 | } 159 | 160 | func (r *River) parseSource() (map[string][]string, error) { 161 | wildTables := make(map[string][]string, len(r.c.Sources)) 162 | 163 | // first, check sources 164 | for _, s := range r.c.Sources { 165 | for _, table := range s.Tables { 166 | if len(s.Schema) == 0 { 167 | return nil, errors.Errorf("empty schema not allowed for source") 168 | } 169 | 170 | if regexp.QuoteMeta(table) != table { 171 | if _, ok := wildTables[ruleKey(s.Schema, table)]; ok { 172 | return nil, errors.Errorf("duplicate wildcard table defined for %s.%s", s.Schema, table) 173 | } 174 | 175 | tables := []string{} 176 | 177 | sql := fmt.Sprintf(`SELECT table_name FROM information_schema.tables WHERE 178 | table_name RLIKE "%s" AND table_schema = "%s";`, table, s.Schema) 179 | 180 | res, err := r.canal.Execute(sql) 181 | if err != nil { 182 | return nil, errors.Trace(err) 183 | } 184 | 185 | for i := 0; i < res.Resultset.RowNumber(); i++ { 186 | f, _ := res.GetString(i, 0) 187 | err := r.newRule(s.Schema, f) 188 | if err != nil { 189 | return nil, errors.Trace(err) 190 | } 191 | 192 | tables = append(tables, f) 193 | } 194 | 195 | wildTables[ruleKey(s.Schema, table)] = tables 196 | } else { 197 | err := r.newRule(s.Schema, table) 198 | if err != nil { 199 | return nil, errors.Trace(err) 200 | } 201 | } 202 | } 203 | } 204 | 205 | if len(r.rules) == 0 && r.c.AllDB != "yes" { 206 | return nil, errors.Errorf("no source data defined") 207 | } 208 | 209 | return wildTables, nil 210 | } 211 | 212 | func (r *River) prepareRule() error { 213 | wildtables, err := r.parseSource() 214 | if err != nil { 215 | return errors.Trace(err) 216 | } 217 | 218 | if r.c.Rules != nil { 219 | // then, set custom mapping rule 220 | for _, rule := range r.c.Rules { 221 | if len(rule.Schema) == 0 { 222 | return errors.Errorf("empty schema not allowed for rule") 223 | } 224 | 225 | if regexp.QuoteMeta(rule.Table) != rule.Table { 226 | //wildcard table 227 | tables, ok := wildtables[ruleKey(rule.Schema, rule.Table)] 228 | if !ok { 229 | return errors.Errorf("wildcard table for %s.%s is not defined in source", rule.Schema, rule.Table) 230 | } 231 | 232 | if len(rule.Database) == 0 { 233 | return errors.Errorf("wildcard table rule %s.%s must have a database, can not empty", rule.Schema, rule.Table) 234 | } 235 | 236 | rule.prepare() 237 | 238 | for _, table := range tables { 239 | rr := r.rules[ruleKey(rule.Schema, table)] 240 | rr.Database = rule.Database 241 | rr.Collection = rule.Collection 242 | rr.ID = rule.ID 243 | rr.FieldMapping = rule.FieldMapping 244 | } 245 | } else { 246 | key := ruleKey(rule.Schema, rule.Table) 247 | if _, ok := r.rules[key]; !ok { 248 | return errors.Errorf("rule %s, %s not defined in source", rule.Schema, rule.Table) 249 | } 250 | rule.prepare() 251 | r.rules[key] = rule 252 | } 253 | } 254 | } 255 | 256 | for _, rule := range r.rules { 257 | if rule.TableInfo, err = r.canal.GetTable(rule.Schema, rule.Table); err != nil { 258 | return errors.Trace(err) 259 | } 260 | } 261 | 262 | return nil 263 | } 264 | 265 | func ruleKey(schema string, table string) string { 266 | return fmt.Sprintf("%s:%s", schema, table) 267 | } 268 | 269 | func (r *River) Start() error { 270 | r.wg.Add(1) 271 | go r.syncLoop() 272 | 273 | pos := r.master.Position() 274 | if err := r.canal.StartFrom(pos); err != nil { 275 | log.Errorf("start canal err %v", err) 276 | return errors.Trace(err) 277 | } 278 | 279 | return nil 280 | } 281 | 282 | func (r *River) Ctx() context.Context { 283 | return r.ctx 284 | } 285 | 286 | func (r *River) Close() { 287 | log.Infof("closing river") 288 | 289 | r.cancel() 290 | 291 | r.canal.Close() 292 | 293 | r.master.Close() 294 | 295 | r.wg.Wait() 296 | } 297 | -------------------------------------------------------------------------------- /river/river_test.go: -------------------------------------------------------------------------------- 1 | package river 2 | 3 | import ( 4 | "flag" 5 | "fmt" 6 | "os" 7 | "testing" 8 | "time" 9 | 10 | "github.com/WangXiangUSTC/go-mysql-mongodb/mongodb" 11 | "github.com/siddontang/go-mysql/client" 12 | . "gopkg.in/check.v1" 13 | ) 14 | 15 | var my_addr = flag.String("my_addr", "127.0.0.1:3306", "MySQL addr") 16 | var mongo_addr = flag.String("mongo_addr", "127.0.0.1:27017", "Elasticsearch addr") 17 | 18 | func Test(t *testing.T) { 19 | TestingT(t) 20 | } 21 | 22 | type riverTestSuite struct { 23 | c *client.Conn 24 | r *River 25 | } 26 | 27 | var _ = Suite(&riverTestSuite{}) 28 | 29 | func (s *riverTestSuite) SetUpSuite(c *C) { 30 | var err error 31 | s.c, err = client.Connect(*my_addr, "root", "", "test") 32 | c.Assert(err, IsNil) 33 | 34 | s.testExecute(c, "SET SESSION binlog_format = 'ROW'") 35 | 36 | schema := ` 37 | CREATE TABLE IF NOT EXISTS %s ( 38 | id INT, 39 | title VARCHAR(256), 40 | content VARCHAR(256), 41 | mylist VARCHAR(256), 42 | tenum ENUM("e1", "e2", "e3"), 43 | tset SET("a", "b", "c"), 44 | PRIMARY KEY(id)) ENGINE=INNODB; 45 | ` 46 | 47 | s.testExecute(c, "DROP TABLE IF EXISTS test_river") 48 | s.testExecute(c, fmt.Sprintf(schema, "test_river")) 49 | s.testExecute(c, fmt.Sprintf(schema, "test_for_id")) 50 | 51 | for i := 0; i < 10; i++ { 52 | table := fmt.Sprintf("test_river_%04d", i) 53 | s.testExecute(c, fmt.Sprintf("DROP TABLE IF EXISTS %s", table)) 54 | s.testExecute(c, fmt.Sprintf(schema, table)) 55 | } 56 | 57 | cfg := new(Config) 58 | cfg.MyAddr = *my_addr 59 | cfg.MyUser = "root" 60 | cfg.MyPassword = "" 61 | cfg.MyCharset = "utf8" 62 | cfg.MongoAddr = *mongo_addr 63 | 64 | cfg.ServerID = 1001 65 | cfg.Flavor = "mysql" 66 | 67 | cfg.DataDir = "/tmp/test_river" 68 | cfg.DumpExec = "mysqldump" 69 | 70 | cfg.StatAddr = "127.0.0.1:12800" 71 | cfg.BulkSize = 1 72 | cfg.FlushBulkTime = TomlDuration{3 * time.Millisecond} 73 | 74 | os.RemoveAll(cfg.DataDir) 75 | 76 | cfg.Sources = []SourceConfig{SourceConfig{Schema: "test", Tables: []string{"test_river", "test_river_[0-9]{4}", "test_for_id"}}} 77 | 78 | cfg.Rules = []*Rule{ 79 | &Rule{Schema: "test", 80 | Table: "test_river", 81 | Database: "river", 82 | Collection: "river", 83 | FieldMapping: map[string]string{"title": "mongo_title", "mylist": "mongo_mylist,list"}, 84 | }, 85 | 86 | &Rule{Schema: "test", 87 | Table: "test_for_id", 88 | Database: "river", 89 | Collection: "river", 90 | ID: []string{"id", "title"}, 91 | FieldMapping: map[string]string{"title": "mongo_title", "mylist": "mongo_mylist,list"}, 92 | }, 93 | 94 | &Rule{Schema: "test", 95 | Table: "test_river_[0-9]{4}", 96 | Database: "river", 97 | Collection: "river", 98 | FieldMapping: map[string]string{"title": "mongo_title", "mylist": "mongo_mylist,list"}, 99 | }, 100 | } 101 | 102 | s.r, err = NewRiver(cfg) 103 | c.Assert(err, IsNil) 104 | } 105 | 106 | func (s *riverTestSuite) TearDownSuite(c *C) { 107 | s.testCleanData(c) 108 | 109 | if s.c != nil { 110 | s.c.Close() 111 | } 112 | 113 | if s.r != nil { 114 | s.r.Close() 115 | } 116 | } 117 | 118 | func (s *riverTestSuite) TestConfig(c *C) { 119 | str := ` 120 | my_addr = "127.0.0.1:3306" 121 | my_user = "root" 122 | my_pass = "" 123 | my_charset = "utf8" 124 | mongo_addr = "127.0.0.1:27017" 125 | data_dir = "./var" 126 | [[source]] 127 | schema = "test" 128 | tables = ["test_river", "test_river_[0-9]{4}", "test_for_id"] 129 | [[rule]] 130 | schema = "test" 131 | table = "test_river" 132 | database = "river" 133 | collection = "river" 134 | [rule.field] 135 | title = "mongo_title" 136 | mylist = "mongo_mylist,list" 137 | [[rule]] 138 | schema = "test" 139 | table = "test_for_id" 140 | database = "river" 141 | collection = "river" 142 | id = ["id", "title"] 143 | [rule.field] 144 | title = "mongo_title" 145 | mylist = "mongo_mylist,list" 146 | [[rule]] 147 | schema = "test" 148 | table = "test_river_[0-9]{4}" 149 | database = "river" 150 | collection = "river" 151 | [rule.field] 152 | title = "mongo_title" 153 | mylist = "mongo_mylist,list" 154 | ` 155 | 156 | cfg, err := NewConfig(str) 157 | c.Assert(err, IsNil) 158 | c.Assert(cfg.Sources, HasLen, 1) 159 | c.Assert(cfg.Sources[0].Tables, HasLen, 3) 160 | c.Assert(cfg.Rules, HasLen, 3) 161 | } 162 | 163 | func (s *riverTestSuite) testExecute(c *C, query string, args ...interface{}) { 164 | fmt.Println(query, args) 165 | _, err := s.c.Execute(query, args...) 166 | c.Assert(err, IsNil) 167 | } 168 | 169 | func (s *riverTestSuite) testPrepareData(c *C) { 170 | s.testExecute(c, "INSERT INTO test_river (id, title, content, tenum, tset) VALUES (?, ?, ?, ?, ?)", 1, "first", "hello go 1", "e1", "a,b") 171 | s.testExecute(c, "INSERT INTO test_river (id, title, content, tenum, tset) VALUES (?, ?, ?, ?, ?)", 2, "second", "hello mysql 2", "e2", "b,c") 172 | s.testExecute(c, "INSERT INTO test_river (id, title, content, tenum, tset) VALUES (?, ?, ?, ?, ?)", 3, "third", "hello mongodb 3", "e3", "c") 173 | s.testExecute(c, "INSERT INTO test_river (id, title, content, tenum, tset) VALUES (?, ?, ?, ?, ?)", 4, "fouth", "hello go-mysql-mongodb 4", "e1", "a,b,c") 174 | s.testExecute(c, "INSERT INTO test_for_id (id, title, content, tenum, tset) VALUES (?, ?, ?, ?, ?)", 1, "first", "hello go 1", "e1", "a,b") 175 | 176 | for i := 0; i < 10; i++ { 177 | table := fmt.Sprintf("test_river_%04d", i) 178 | s.testExecute(c, fmt.Sprintf("INSERT INTO %s (id, title, content, tenum, tset) VALUES (?, ?, ?, ?, ?)", table), 5+i, "abc", "hello", "e1", "a,b,c") 179 | } 180 | } 181 | 182 | func (s *riverTestSuite) testCleanData(c *C) { 183 | for i := 1; i <= 4; i++ { 184 | s.testExecute(c, "DELETE FROM test_river WHERE id = ?", i) 185 | } 186 | s.testExecute(c, "DELETE FROM test_for_id WHERE id = ?", 1) 187 | 188 | for i := 0; i < 10; i++ { 189 | table := fmt.Sprintf("test_river_%04d", i) 190 | s.testExecute(c, fmt.Sprintf("DELETE FROM %s WHERE id = ?", table), 5+i) 191 | } 192 | } 193 | 194 | func (s *riverTestSuite) testMongoGet(c *C, id string) *mongodb.Response { 195 | database := "river" 196 | collection := "river" 197 | 198 | r, err := s.r.mongo.Get(database, collection, id) 199 | c.Assert(err, IsNil) 200 | 201 | return r 202 | } 203 | 204 | func testWaitSyncDone(c *C, r *River) { 205 | <-r.canal.WaitDumpDone() 206 | 207 | err := r.canal.CatchMasterPos(10 * time.Second) 208 | c.Assert(err, IsNil) 209 | 210 | for i := 0; i < 1000; i++ { 211 | if len(r.syncCh) == 0 { 212 | return 213 | } 214 | 215 | time.Sleep(10 * time.Millisecond) 216 | } 217 | 218 | c.Fatalf("wait 1s but still have %d items to be synced", len(r.syncCh)) 219 | } 220 | 221 | func (s *riverTestSuite) TestRiver(c *C) { 222 | s.testPrepareData(c) 223 | 224 | s.r.Start() 225 | 226 | testWaitSyncDone(c, s.r) 227 | 228 | var r *mongodb.Response 229 | r = s.testMongoGet(c, "1") 230 | c.Assert(r.Found, Equals, true) 231 | c.Assert(r.Source["tenum"], Equals, "e1") 232 | c.Assert(r.Source["tset"], Equals, "a,b") 233 | 234 | r = s.testMongoGet(c, "1:first") 235 | c.Assert(r.Found, Equals, true) 236 | 237 | r = s.testMongoGet(c, "100") 238 | c.Assert(r.Found, Equals, false) 239 | 240 | for i := 0; i < 10; i++ { 241 | r = s.testMongoGet(c, fmt.Sprintf("%d", 5+i)) 242 | c.Assert(r.Found, Equals, true) 243 | c.Assert(r.Source["mongo_title"], Equals, "abc") 244 | } 245 | 246 | s.testExecute(c, "UPDATE test_river SET title = ?, tenum = ?, tset = ?, mylist = ? WHERE id = ?", "second 2", "e3", "a,b,c", "a,b,c", 2) 247 | s.testExecute(c, "DELETE FROM test_river WHERE id = ?", 1) 248 | s.testExecute(c, "UPDATE test_river SET title = ?, id = ? WHERE id = ?", "second 30", 30, 3) 249 | 250 | // so we can insert invalid data 251 | s.testExecute(c, `SET SESSION sql_mode="NO_ENGINE_SUBSTITUTION";`) 252 | 253 | // bad insert 254 | s.testExecute(c, "UPDATE test_river SET title = ?, tenum = ?, tset = ? WHERE id = ?", "second 2", "e5", "a,b,c,d", 4) 255 | 256 | for i := 0; i < 10; i++ { 257 | table := fmt.Sprintf("test_river_%04d", i) 258 | s.testExecute(c, fmt.Sprintf("UPDATE %s SET title = ? WHERE id = ?", table), "hello", 5+i) 259 | } 260 | 261 | testWaitSyncDone(c, s.r) 262 | 263 | r = s.testMongoGet(c, "1") 264 | c.Assert(r.Found, Equals, false) 265 | 266 | r = s.testMongoGet(c, "2") 267 | c.Assert(r.Found, Equals, true) 268 | c.Assert(r.Source["mongo_title"], Equals, "second 2") 269 | c.Assert(r.Source["tenum"], Equals, "e3") 270 | c.Assert(r.Source["tset"], Equals, "a,b,c") 271 | c.Assert(r.Source["mongo_mylist"], DeepEquals, []interface{}{"a", "b", "c"}) 272 | 273 | r = s.testMongoGet(c, "4") 274 | c.Assert(r.Found, Equals, true) 275 | c.Assert(r.Source["tenum"], Equals, "") 276 | c.Assert(r.Source["tset"], Equals, "a,b,c") 277 | 278 | r = s.testMongoGet(c, "3") 279 | c.Assert(r.Found, Equals, false) 280 | 281 | r = s.testMongoGet(c, "30") 282 | c.Assert(r.Found, Equals, true) 283 | c.Assert(r.Source["mongo_title"], Equals, "second 30") 284 | 285 | for i := 0; i < 10; i++ { 286 | r = s.testMongoGet(c, fmt.Sprintf("%d", 5+i)) 287 | c.Assert(r.Found, Equals, true) 288 | c.Assert(r.Source["mongo_title"], Equals, "hello") 289 | } 290 | } 291 | -------------------------------------------------------------------------------- /river/rule.go: -------------------------------------------------------------------------------- 1 | package river 2 | 3 | import ( 4 | "github.com/siddontang/go-mysql/schema" 5 | ) 6 | 7 | // If you want to sync MySQL data into MongoDB, you must set a rule to let use know how to do it. 8 | // The mapping rule may thi: schema + table <-> database + collection. 9 | // schema and table is for MySQL, database and collection type is for MongoDB. 10 | type Rule struct { 11 | Schema string `toml:"schema"` 12 | Table string `toml:"table"` 13 | Database string `toml:"database"` 14 | Collection string `toml:"collection"` 15 | ID []string `toml:"id"` 16 | 17 | // Default, a MySQL table field name is mapped to MongoDB field name. 18 | // Sometimes, you want to use different name, e.g, the MySQL file name is title, 19 | // but in Elasticsearch, you want to name it my_title. 20 | FieldMapping map[string]string `toml:"field"` 21 | 22 | // MySQL table information 23 | TableInfo *schema.Table 24 | 25 | //only MySQL fields in fileter will be synced , default sync all fields 26 | Fileter []string `toml:"filter"` 27 | } 28 | 29 | func newDefaultRule(schema string, table string) *Rule { 30 | r := new(Rule) 31 | 32 | r.Schema = schema 33 | r.Table = table 34 | r.Database = schema 35 | r.Collection = table 36 | r.FieldMapping = make(map[string]string) 37 | 38 | return r 39 | } 40 | 41 | func (r *Rule) prepare() error { 42 | if r.FieldMapping == nil { 43 | r.FieldMapping = make(map[string]string) 44 | } 45 | 46 | if len(r.Database) == 0 { 47 | r.Database = r.Table 48 | } 49 | 50 | if len(r.Collection) == 0 { 51 | r.Collection = r.Database 52 | } 53 | 54 | return nil 55 | } 56 | 57 | func (r *Rule) CheckFilter(field string) bool { 58 | if r.Fileter == nil { 59 | return true 60 | } 61 | 62 | for _, f := range r.Fileter { 63 | if f == field { 64 | return true 65 | } 66 | } 67 | return false 68 | } 69 | -------------------------------------------------------------------------------- /river/status.go: -------------------------------------------------------------------------------- 1 | package river 2 | 3 | import ( 4 | "bytes" 5 | "fmt" 6 | "net" 7 | "net/http" 8 | 9 | "github.com/ngaut/log" 10 | "github.com/siddontang/go/sync2" 11 | ) 12 | 13 | type stat struct { 14 | r *River 15 | 16 | l net.Listener 17 | 18 | InsertNum sync2.AtomicInt64 19 | UpdateNum sync2.AtomicInt64 20 | DeleteNum sync2.AtomicInt64 21 | } 22 | 23 | func (s *stat) ServeHTTP(w http.ResponseWriter, r *http.Request) { 24 | var buf bytes.Buffer 25 | 26 | rr, err := s.r.canal.Execute("SHOW MASTER STATUS") 27 | if err != nil { 28 | w.WriteHeader(http.StatusInternalServerError) 29 | w.Write([]byte(fmt.Sprintf("execute sql error %v", err))) 30 | return 31 | } 32 | 33 | binName, _ := rr.GetString(0, 0) 34 | binPos, _ := rr.GetUint(0, 1) 35 | 36 | pos := s.r.canal.SyncedPosition() 37 | 38 | buf.WriteString(fmt.Sprintf("server_current_binlog:(%s, %d)\n", binName, binPos)) 39 | buf.WriteString(fmt.Sprintf("read_binlog:%s\n", pos)) 40 | 41 | buf.WriteString(fmt.Sprintf("insert_num:%d\n", s.InsertNum.Get())) 42 | buf.WriteString(fmt.Sprintf("update_num:%d\n", s.UpdateNum.Get())) 43 | buf.WriteString(fmt.Sprintf("delete_num:%d\n", s.DeleteNum.Get())) 44 | 45 | w.Write(buf.Bytes()) 46 | } 47 | 48 | func (s *stat) Run(addr string) { 49 | if len(addr) == 0 { 50 | return 51 | } 52 | log.Infof("run status http server %s", addr) 53 | var err error 54 | s.l, err = net.Listen("tcp", addr) 55 | if err != nil { 56 | log.Errorf("listen stat addr %s err %v", addr, err) 57 | return 58 | } 59 | 60 | srv := http.Server{} 61 | mux := http.NewServeMux() 62 | mux.Handle("/stat", s) 63 | srv.Handler = mux 64 | 65 | srv.Serve(s.l) 66 | } 67 | 68 | func (s *stat) Close() { 69 | if s.l != nil { 70 | s.l.Close() 71 | } 72 | } 73 | -------------------------------------------------------------------------------- /river/sync.go: -------------------------------------------------------------------------------- 1 | package river 2 | 3 | import ( 4 | "bytes" 5 | "encoding/hex" 6 | "encoding/json" 7 | "fmt" 8 | "reflect" 9 | "strings" 10 | "time" 11 | 12 | "github.com/WangXiangUSTC/go-mysql-mongodb/mongodb" 13 | "github.com/juju/errors" 14 | "github.com/ngaut/log" 15 | "github.com/siddontang/go-mysql/canal" 16 | "github.com/siddontang/go-mysql/mysql" 17 | "github.com/siddontang/go-mysql/replication" 18 | "github.com/siddontang/go-mysql/schema" 19 | ) 20 | 21 | const ( 22 | syncInsertDoc = iota 23 | syncDeleteDoc 24 | syncUpdateDoc 25 | ) 26 | 27 | const ( 28 | fieldTypeList = "list" 29 | ) 30 | 31 | type posSaver struct { 32 | pos mysql.Position 33 | force bool 34 | } 35 | 36 | type eventHandler struct { 37 | r *River 38 | } 39 | 40 | func (h *eventHandler) OnRotate(e *replication.RotateEvent) error { 41 | pos := mysql.Position{ 42 | string(e.NextLogName), 43 | uint32(e.Position), 44 | } 45 | 46 | h.r.syncCh <- posSaver{pos, true} 47 | 48 | return h.r.ctx.Err() 49 | } 50 | 51 | func (h *eventHandler) OnDDL(nextPos mysql.Position, _ *replication.QueryEvent) error { 52 | h.r.syncCh <- posSaver{nextPos, true} 53 | return h.r.ctx.Err() 54 | } 55 | 56 | func (h *eventHandler) OnXID(nextPos mysql.Position) error { 57 | h.r.syncCh <- posSaver{nextPos, false} 58 | return h.r.ctx.Err() 59 | } 60 | 61 | func (h *eventHandler) OnRow(e *canal.RowsEvent) error { 62 | var err error 63 | rule, ok := h.r.rules[ruleKey(e.Table.Schema, e.Table.Name)] 64 | if !ok { 65 | if h.r.c.AllDB == "yes" { 66 | rule = newDefaultRule(e.Table.Schema, e.Table.Name) 67 | rule.TableInfo, err = h.r.canal.GetTable(e.Table.Schema, e.Table.Name) 68 | if err != nil { 69 | return nil 70 | } 71 | h.r.rules[ruleKey(e.Table.Schema, e.Table.Name)] = rule 72 | } else { 73 | return nil 74 | } 75 | } 76 | 77 | var reqs []*mongodb.BulkRequest 78 | switch e.Action { 79 | case canal.InsertAction: 80 | reqs, err = h.r.makeInsertRequest(rule, e.Rows) 81 | case canal.DeleteAction: 82 | reqs, err = h.r.makeDeleteRequest(rule, e.Rows) 83 | case canal.UpdateAction: 84 | reqs, err = h.r.makeUpdateRequest(rule, e.Rows) 85 | default: 86 | err = errors.Errorf("invalid rows action %s", e.Action) 87 | } 88 | 89 | if err != nil { 90 | h.r.cancel() 91 | log.Warnf("make %s MongoDB request err %v, close sync", e.Action, err) 92 | return errors.Errorf("make %s MongoDB request err %v, close sync", e.Action, err) 93 | } 94 | 95 | h.r.syncCh <- reqs 96 | 97 | return h.r.ctx.Err() 98 | } 99 | 100 | func (h *eventHandler) String() string { 101 | return "MongoRiverEventHandler" 102 | } 103 | 104 | func (r *River) syncLoop() { 105 | bulkSize := r.c.BulkSize 106 | if bulkSize == 0 { 107 | bulkSize = 512 108 | } 109 | 110 | interval := r.c.FlushBulkTime.Duration 111 | if interval == 0 { 112 | interval = 500 * time.Millisecond 113 | } 114 | 115 | ticker := time.NewTicker(interval) 116 | defer ticker.Stop() 117 | defer r.wg.Done() 118 | 119 | lastSavedTime := time.Now() 120 | reqs := make([]*mongodb.BulkRequest, 0, 2048) 121 | 122 | var pos mysql.Position 123 | 124 | for { 125 | needFlush := false 126 | needSavePos := false 127 | 128 | select { 129 | case v := <-r.syncCh: 130 | switch v := v.(type) { 131 | case posSaver: 132 | now := time.Now() 133 | if v.force || now.Sub(lastSavedTime) > 3*time.Second { 134 | lastSavedTime = now 135 | needFlush = true 136 | needSavePos = true 137 | pos = v.pos 138 | } 139 | case []*mongodb.BulkRequest: 140 | reqs = append(reqs, v...) 141 | needFlush = len(reqs) >= bulkSize 142 | } 143 | case <-ticker.C: 144 | needFlush = true 145 | case <-r.ctx.Done(): 146 | return 147 | } 148 | 149 | if needFlush { 150 | // TODO: retry some times? 151 | if err := r.doBulk(reqs); err != nil { 152 | log.Errorf("do MongoDB bulk err %v, close sync", err) 153 | r.cancel() 154 | return 155 | } 156 | reqs = reqs[0:0] 157 | } 158 | 159 | if needSavePos { 160 | if err := r.master.Save(pos); err != nil { 161 | log.Errorf("save sync position %s err %v, close sync", pos, err) 162 | r.cancel() 163 | return 164 | } 165 | } 166 | } 167 | } 168 | 169 | // for insert and delete 170 | func (r *River) makeRequest(rule *Rule, action string, rows [][]interface{}) ([]*mongodb.BulkRequest, error) { 171 | reqs := make([]*mongodb.BulkRequest, 0, len(rows)) 172 | 173 | for _, values := range rows { 174 | id, err := r.getDocID(rule, values) 175 | if err != nil { 176 | return nil, errors.Trace(err) 177 | } 178 | 179 | req := &mongodb.BulkRequest{Database: rule.Database, Collection: rule.Collection, ID: id} 180 | 181 | if action == canal.DeleteAction { 182 | req.Action = mongodb.ActionDelete 183 | r.st.DeleteNum.Add(1) 184 | } else { 185 | r.makeInsertReqData(req, rule, values) 186 | r.st.InsertNum.Add(1) 187 | } 188 | 189 | reqs = append(reqs, req) 190 | } 191 | 192 | return reqs, nil 193 | } 194 | 195 | func (r *River) makeInsertRequest(rule *Rule, rows [][]interface{}) ([]*mongodb.BulkRequest, error) { 196 | return r.makeRequest(rule, canal.InsertAction, rows) 197 | } 198 | 199 | func (r *River) makeDeleteRequest(rule *Rule, rows [][]interface{}) ([]*mongodb.BulkRequest, error) { 200 | return r.makeRequest(rule, canal.DeleteAction, rows) 201 | } 202 | 203 | func (r *River) makeUpdateRequest(rule *Rule, rows [][]interface{}) ([]*mongodb.BulkRequest, error) { 204 | if len(rows)%2 != 0 { 205 | return nil, errors.Errorf("invalid update rows event, must have 2x rows, but %d", len(rows)) 206 | } 207 | 208 | reqs := make([]*mongodb.BulkRequest, 0, len(rows)) 209 | 210 | for i := 0; i < len(rows); i += 2 { 211 | beforeID, err := r.getDocID(rule, rows[i]) 212 | if err != nil { 213 | return nil, errors.Trace(err) 214 | } 215 | 216 | afterID, err := r.getDocID(rule, rows[i+1]) 217 | 218 | if err != nil { 219 | return nil, errors.Trace(err) 220 | } 221 | 222 | req := &mongodb.BulkRequest{Database: rule.Database, Collection: rule.Collection, ID: beforeID} 223 | 224 | if beforeID != afterID { 225 | req.Action = mongodb.ActionDelete 226 | reqs = append(reqs, req) 227 | 228 | req = &mongodb.BulkRequest{Database: rule.Database, Collection: rule.Collection, ID: afterID} 229 | r.makeInsertReqData(req, rule, rows[i+1]) 230 | 231 | r.st.DeleteNum.Add(1) 232 | r.st.InsertNum.Add(1) 233 | } else { 234 | r.makeUpdateReqData(req, rule, rows[i], rows[i+1]) 235 | r.st.UpdateNum.Add(1) 236 | } 237 | 238 | reqs = append(reqs, req) 239 | } 240 | 241 | return reqs, nil 242 | } 243 | 244 | func (r *River) makeReqColumnData(col *schema.TableColumn, value interface{}) interface{} { 245 | switch col.Type { 246 | case schema.TYPE_ENUM: 247 | switch value := value.(type) { 248 | case int64: 249 | // for binlog, ENUM may be int64, but for dump, enum is string 250 | eNum := value - 1 251 | if eNum < 0 || eNum >= int64(len(col.EnumValues)) { 252 | // we insert invalid enum value before, so return empty 253 | log.Warnf("invalid binlog enum index %d, for enum %v", eNum, col.EnumValues) 254 | return "" 255 | } 256 | 257 | return col.EnumValues[eNum] 258 | } 259 | case schema.TYPE_SET: 260 | switch value := value.(type) { 261 | case int64: 262 | // for binlog, SET may be int64, but for dump, SET is string 263 | bitmask := value 264 | sets := make([]string, 0, len(col.SetValues)) 265 | for i, s := range col.SetValues { 266 | if bitmask&int64(1< 0 { 267 | sets = append(sets, s) 268 | } 269 | } 270 | return strings.Join(sets, ",") 271 | } 272 | case schema.TYPE_BIT: 273 | switch value := value.(type) { 274 | case string: 275 | // for binlog, BIT is int64, but for dump, BIT is string 276 | // for dump 0x01 is for 1, \0 is for 0 277 | if value == "\x01" { 278 | return int64(1) 279 | } 280 | 281 | return int64(0) 282 | } 283 | case schema.TYPE_STRING: 284 | switch value := value.(type) { 285 | case []byte: 286 | return string(value[:]) 287 | } 288 | case schema.TYPE_JSON: 289 | var f interface{} 290 | var err error 291 | switch v := value.(type) { 292 | case string: 293 | err = json.Unmarshal([]byte(v), &f) 294 | case []byte: 295 | err = json.Unmarshal(v, &f) 296 | } 297 | if err == nil && f != nil { 298 | return f 299 | } 300 | } 301 | 302 | return value 303 | } 304 | 305 | func (r *River) getFieldParts(k string, v string) (string, string, string) { 306 | composedField := strings.Split(v, ",") 307 | 308 | mysql := k 309 | mongodb := composedField[0] 310 | fieldType := "" 311 | 312 | if 0 == len(mongodb) { 313 | mongodb = mysql 314 | } 315 | if 2 == len(composedField) { 316 | fieldType = composedField[1] 317 | } 318 | 319 | return mysql, mongodb, fieldType 320 | } 321 | 322 | func (r *River) makeInsertReqData(req *mongodb.BulkRequest, rule *Rule, values []interface{}) { 323 | req.Data = make(map[string]interface{}, len(values)) 324 | req.Action = mongodb.ActionInsert 325 | 326 | for i, c := range rule.TableInfo.Columns { 327 | if !rule.CheckFilter(c.Name) { 328 | continue 329 | } 330 | mapped := false 331 | for k, v := range rule.FieldMapping { 332 | mysql, mongodb, fieldType := r.getFieldParts(k, v) 333 | if mysql == c.Name { 334 | mapped = true 335 | v := r.makeReqColumnData(&c, values[i]) 336 | if fieldType == fieldTypeList { 337 | if str, ok := v.(string); ok { 338 | req.Data[mongodb] = strings.Split(str, ",") 339 | } else { 340 | req.Data[mongodb] = v 341 | } 342 | } else { 343 | req.Data[mongodb] = v 344 | } 345 | } 346 | } 347 | if mapped == false { 348 | req.Data[c.Name] = r.makeReqColumnData(&c, values[i]) 349 | } 350 | } 351 | } 352 | 353 | func (r *River) makeUpdateReqData(req *mongodb.BulkRequest, rule *Rule, 354 | beforeValues []interface{}, afterValues []interface{}) { 355 | req.Data = make(map[string]interface{}, len(beforeValues)) 356 | 357 | // maybe dangerous if something wrong delete before? 358 | req.Action = mongodb.ActionUpdate 359 | 360 | for i, c := range rule.TableInfo.Columns { 361 | mapped := false 362 | if !rule.CheckFilter(c.Name) { 363 | continue 364 | } 365 | if reflect.DeepEqual(beforeValues[i], afterValues[i]) { 366 | //nothing changed 367 | continue 368 | } 369 | for k, v := range rule.FieldMapping { 370 | mysql, mongodb, fieldType := r.getFieldParts(k, v) 371 | if mysql == c.Name { 372 | mapped = true 373 | // has custom field mapping 374 | v := r.makeReqColumnData(&c, afterValues[i]) 375 | str, ok := v.(string) 376 | if ok == false { 377 | req.Data[c.Name] = v 378 | } else { 379 | if fieldType == fieldTypeList { 380 | req.Data[mongodb] = strings.Split(str, ",") 381 | } else { 382 | req.Data[mongodb] = str 383 | } 384 | } 385 | } 386 | } 387 | if mapped == false { 388 | req.Data[c.Name] = r.makeReqColumnData(&c, afterValues[i]) 389 | } 390 | 391 | } 392 | } 393 | 394 | // If id in toml file is none, get primary keys in one row and format them into a string, and PK must not be nil 395 | // Else get the ID's column in one row and format them into a string 396 | func (r *River) getDocID(rule *Rule, row []interface{}) (string, error) { 397 | var ( 398 | flag bool 399 | ids []interface{} 400 | ) 401 | if rule.ID == nil { 402 | ids = make([]interface{}, 0, len(rule.TableInfo.PKColumns)) 403 | 404 | if len(rule.TableInfo.PKColumns) == 0 { 405 | flag = true 406 | } 407 | for _, num := range rule.TableInfo.PKColumns { 408 | ids = append(ids, r.makeReqColumnData(&(rule.TableInfo.Columns[num]), row[num])) 409 | } 410 | } else { 411 | ids = make([]interface{}, 0, len(rule.ID)) 412 | for _, column := range rule.ID { 413 | index := rule.TableInfo.FindColumn(column) 414 | ids = append(ids, r.makeReqColumnData(&(rule.TableInfo.Columns[index]), row[index])) 415 | } 416 | } 417 | if flag { 418 | ids = make([]interface{}, 0, len(rule.TableInfo.Columns)) 419 | for i, column := range rule.TableInfo.Columns { 420 | ids = append(ids, r.makeReqColumnData(&column, row[i])) 421 | } 422 | } 423 | 424 | var buf bytes.Buffer 425 | 426 | sep := "" 427 | for i, value := range ids { 428 | if value == nil { 429 | value = "" 430 | if !flag { 431 | log.Warnf("Position: %d id or PK value is nil, row: %s", i, row) 432 | } 433 | } 434 | 435 | buf.WriteString(fmt.Sprintf("%s%v", sep, value)) 436 | sep = ":" 437 | } 438 | 439 | if flag { 440 | r.md5Ctx.Write(buf.Bytes()) 441 | cipherStr := r.md5Ctx.Sum(nil) 442 | r.md5Ctx.Reset() 443 | return hex.EncodeToString(cipherStr), nil 444 | } 445 | return buf.String(), nil 446 | } 447 | 448 | func (r *River) doBulk(reqs []*mongodb.BulkRequest) error { 449 | if len(reqs) == 0 { 450 | return nil 451 | } 452 | 453 | if err := r.mongo.Bulk(reqs); err != nil { 454 | log.Errorf("sync docs err %v after binlog %s", err, r.canal.SyncedPosition()) 455 | return errors.Trace(err) 456 | } 457 | 458 | return nil 459 | } 460 | -------------------------------------------------------------------------------- /tests/check_contains: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | 3 | set -eu 4 | 5 | if ! grep -Fq "$1" "$2"; then 6 | echo "TEST FAILED: $2 DOES NOT CONTAIN '$1'" 7 | echo "____________________________________" 8 | cat $2 9 | echo "^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^" 10 | exit 1 11 | fi -------------------------------------------------------------------------------- /tests/river.toml: -------------------------------------------------------------------------------- 1 | # MySQL address, user and password 2 | # user must have replication privilege in MySQL. 3 | my_addr = "127.0.0.1:3306" 4 | my_user = "test" 5 | my_pass = "secret" 6 | my_charset = "utf8" 7 | 8 | # If you want sync all database data, you need set my_alldb="yes" 9 | my_alldb = "no" 10 | 11 | # MongoDB address 12 | mongo_addr = "127.0.0.1:27017" 13 | mongo_user = "" 14 | mongo_pass = "" 15 | 16 | # Path to store data, like master.info, if not set or empty, 17 | # we must use this to support breakpoint resume syncing. 18 | # TODO: support other storage, like etcd. 19 | data_dir = "./var" 20 | 21 | # Inner Http status address 22 | stat_addr = "127.0.0.1:12800" 23 | 24 | # pseudo server id like a slave 25 | server_id = 1001 26 | 27 | # mysql or mariadb 28 | flavor = "mysql" 29 | 30 | # mysqldump execution path 31 | # if not set or empty, ignore mysqldump. 32 | mysqldump = "mysqldump" 33 | 34 | # minimal items to be inserted in one bulk 35 | bulk_size = 128 36 | 37 | # force flush the pending requests if we don't have enough items >= bulk_size 38 | flush_bulk_time = "200ms" 39 | 40 | # MySQL data source 41 | [[source]] 42 | schema = "go_mysql_mongodb_test" 43 | 44 | # Only below tables will be synced into MongoDB. 45 | # "t_[0-9]{4}" is a wildcard table format, you can use it if you have many sub tables, like table_0000 - table_1023 46 | # I don't think it is necessary to sync all tables in a database. 47 | tables = ["t_[0-9]{4}"] 48 | -------------------------------------------------------------------------------- /tests/run.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | set -eu 4 | 5 | function EXEC_SQL() { 6 | mysql -utest -h 127.0.0.1 -psecret -P3306 -e "$1" 7 | } 8 | 9 | echo "Prepare data in MySQL" 10 | EXEC_SQL "drop database if exists go_mysql_mongodb_test" 11 | EXEC_SQL "create database go_mysql_mongodb_test"; 12 | EXEC_SQL "create table go_mysql_mongodb_test.t_0001(id int primary key, name varchar(10));"; 13 | EXEC_SQL "insert into go_mysql_mongodb_test.t_0001 values(1, 'a');" 14 | 15 | echo "Start go-mysql-mongodb" 16 | ./bin/go-mysql-mongodb --config ./tests/river.toml > test.log 2>&1 & 17 | 18 | echo "Insert data into MySQL" 19 | EXEC_SQL "insert into go_mysql_mongodb_test.t_0001 values(2, 'b');" 20 | EXEC_SQL "insert into go_mysql_mongodb_test.t_0001 values(3, 'c');" 21 | 22 | echo "Check data in MongoDB" 23 | cat test.log 24 | mongo go_mysql_mongodb_test --quiet --eval 'db.t_0001.find().toArray()' > find.result 25 | # output: 26 | #[ 27 | # { 28 | # "_id" : "1", 29 | # "id" : NumberLong(1), 30 | # "name" : "a" 31 | # }, 32 | # { 33 | # "_id" : "2", 34 | # "id" : NumberLong(2), 35 | # "name" : "b" 36 | # }, 37 | # { 38 | # "_id" : "3", 39 | # "id" : NumberLong(3), 40 | # "name" : "c" 41 | # } 42 | #] 43 | ./tests/check_contains '"_id" : "1"' find.result 44 | ./tests/check_contains '"_id" : "2"' find.result 45 | ./tests/check_contains '"_id" : "3"' find.result 46 | ./tests/check_contains '"name" : "a"' find.result 47 | ./tests/check_contains '"name" : "b"' find.result 48 | ./tests/check_contains '"name" : "c"' find.result 49 | --------------------------------------------------------------------------------