├── .gitignore ├── LICENSE ├── Makefile ├── README.md ├── go.mod ├── go.sum ├── logo.png └── main.go /.gitignore: -------------------------------------------------------------------------------- 1 | # We don't want the actual build 2 | go-redis-migrator 3 | 4 | # Compiled Object files, Static and Dynamic libs (Shared Objects) 5 | *.o 6 | *.a 7 | *.so 8 | 9 | # Folders 10 | _obj 11 | _test 12 | 13 | # Architecture specific extensions/prefixes 14 | *.[568vq] 15 | [568vq].out 16 | 17 | *.cgo1.go 18 | *.cgo2.c 19 | _cgo_defun.c 20 | _cgo_gotypes.go 21 | _cgo_export.* 22 | 23 | _testmain.go 24 | 25 | *.exe 26 | *.test 27 | *.prof 28 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | This is free and unencumbered software released into the public domain. 2 | 3 | Anyone is free to copy, modify, publish, use, compile, sell, or 4 | distribute this software, either in source code form or as a compiled 5 | binary, for any purpose, commercial or non-commercial, and by any 6 | means. 7 | 8 | In jurisdictions that recognize copyright laws, the author or authors 9 | of this software dedicate any and all copyright interest in the 10 | software to the public domain. We make this dedication for the benefit 11 | of the public at large and to the detriment of our heirs and 12 | successors. We intend this dedication to be an overt act of 13 | relinquishment in perpetuity of all present and future rights to this 14 | software under copyright law. 15 | 16 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 17 | EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 18 | MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. 19 | IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR 20 | OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, 21 | ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR 22 | OTHER DEALINGS IN THE SOFTWARE. 23 | 24 | For more information, please refer to 25 | 26 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | .Phony: all 2 | 3 | SHELL := /bin/bash # Use bash syntax 4 | 5 | build: 6 | go build -v -o out/ 7 | 8 | build-mac: 9 | GOOS=darwin GOARCH=amd64 go build -v -o out/ 10 | 11 | build-linux: 12 | GOOS=linux GOARCH=amd64 go build -v -o out/ 13 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ![](https://raw.githubusercontent.com/integrii/go-redis-migrator/master/logo.png) 2 | 3 | A cluster aware Redis key migrator. Moves keys from one cluster or host to another cluster or host. 4 | 5 | 6 | ## Details 7 | It is possible to use the -keyFilter flag to match a subset of keys you wish to migrate. It is also possible to fetch a list of all keys with this program, create a file with a subset of keys to migrate, and then feed that list back into this program for selective migration. This is useful if you have some keys hanging around that you do not want to migrate and want to prune during the migration. 8 | 9 | go-redis-migrator is especially useful for populating test data in development instances, executing production migrations, or moving from a non-clustered to a clustered deployment. 10 | 11 | ## Setup 12 | 13 | - Make sure you have golang installed on your operating system. 14 | - Make sure you have your `$GOPATH` environment variable setup properly. 15 | - Download it: `go get github.com/integrii/go-redis-migrator` 16 | - Build and install it: `go install github.com/integrii/go-redis-migrator` 17 | - Finally, run the binary to see the help: `$GOPATH/bin/go-redis-migrator` 18 | 19 | ## Example 20 | 21 | ``` 22 | ./go-redis-migrator -copyData -sourceHosts=172.31.37.164:6379,172.31.37.162:6379,172.31.37.168:6379,172.31.37.170:6379,172.31.37.169:6379 -destinationHosts=172.31.34.231:6379,172.31.34.228:6379,172.31.34.227:6379,172.31.34.230:6379,172.31.34.229:6379,172.31.34.226:6379 -keyFile=keyList.txt 23 | Migrated 57 keys. 24 | ``` 25 | 26 | ## Speed Test 27 | 28 | ``` 29 | time ./go-redis-migrator -sourceHosts=127.0.0.1:6379 -destinationHosts=127.0.0.1:6380 -copyData 30 | Migrated 22000 keys. 31 | 32 | real 0m3.455s 33 | user 0m0.812s 34 | sys 0m1.475s 35 | ``` 36 | **6,367 keys/sec!** 37 | 38 | 39 | ## More Examples 40 | 41 | #### Fetching all keys from a host: 42 | 43 | `./go-redis-migrator -getKeys -sourceHosts=127.0.0.1:6379` 44 | 45 | #### Fetch a filtered key list from a host: 46 | 47 | `./go-redis-migrator -getKeys -keyFilter=sessions:\* -sourceHosts=127.0.0.1:6379` 48 | 49 | #### Fetch a filtered key list from a cluster: 50 | 51 | `./go-redis-migrator -getKeys -keyFilter=sessions:\* -sourceHosts=127.0.0.1:6379,127.0.0.1:6380` 52 | 53 | #### From a host to a host: 54 | 55 | `./go-redis-migrator -copyData -sourceHosts=127.0.0.1:6379 -destinationHosts=192.168.0.1:6379` 56 | 57 | #### From a cluster to a cluster: 58 | 59 | `./go-redis-migrator -copyData -sourceHosts=127.0.0.1:6379,127.0.0.1:6380 -destinationHosts=192.168.0.1:6379,192.168.0.1:6380` 60 | 61 | #### From a cluster to a single host: 62 | 63 | `./go-redis-migrator -copyData -sourceHosts=127.0.0.1:6379,127.0.0.1:6380 -destinationHosts=192.168.0.1:6379` 64 | 65 | #### From a single host to a cluster: 66 | 67 | `./go-redis-migrator -copyData -sourceHosts=127.0.0.1:6379 -destinationHosts=192.168.0.1:6379,192.168.0.1:6380` 68 | 69 | #### Copying keys specified in a file from a host to a host: 70 | 71 | `./go-redis-migrator -copydata=true -sourceHosts=127.0.0.1:6379 -destinationHosts=192.168.0.1:6379 -keyFile=./onlyMoveTheseKeys.txt` 72 | 73 | 74 | ## CLI help 75 | Simply run the binary to get the following help: 76 | ``` 77 | - Redis Key Migrator - 78 | https://github.com/integrii/go-redis-migrator 79 | 80 | Migrates all or some of the keys from a Redis host or cluster to a specified host or cluster. Supports migrating TTLs. 81 | go-redis-migrator can also be used to list the keys in a cluster. Run with the -getKey and -sourceHosts= flags to do so. 82 | 83 | You must run this program with an operation before it will do anything. 84 | 85 | Flags: 86 | -getKeys=false: Fetches and prints keys from the source host. 87 | -copyData=false: Copies all keys in a list specified by -keyFile= to the destination cluster from the source cluster. 88 | -keyFile="": The file path which contains the list of keys to migrate. One per line. 89 | -keyFilter="*": The filter for which keys to migrate. Does not work when -keyFile is also specified. 90 | -destinationHosts="": A list of source cluster host:port servers seperated by spaces. Use a single host without a ',' if there is no cluster. EX) 127.0.0.1:6379,127.0.0.1:6380 91 | -sourceHosts="": A list of source cluster host:port servers seperated by commas. Use a single host without a ',' if there is no cluster. EX) 127.0.0.1:6379,127.0.0.1:6380 92 | ``` 93 | -------------------------------------------------------------------------------- /go.mod: -------------------------------------------------------------------------------- 1 | module github.com/integrii/go-redis-migrator 2 | 3 | go 1.16 4 | 5 | require ( 6 | github.com/garyburd/redigo v1.6.2 // indirect 7 | github.com/onsi/ginkgo v1.16.4 // indirect 8 | github.com/onsi/gomega v1.13.0 // indirect 9 | gopkg.in/bsm/ratelimit.v1 v1.0.0-20160220154919-db14e161995a // indirect 10 | gopkg.in/redis.v3 v3.6.4 11 | ) 12 | -------------------------------------------------------------------------------- /go.sum: -------------------------------------------------------------------------------- 1 | github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= 2 | github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= 3 | github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo= 4 | github.com/fsnotify/fsnotify v1.4.9 h1:hsms1Qyu0jgnwNXIxa+/V/PDsU6CfLf6CNO8H7IWoS4= 5 | github.com/fsnotify/fsnotify v1.4.9/go.mod h1:znqG4EE+3YCdAaPaxE2ZRY/06pZUdp0tY4IgpuI1SZQ= 6 | github.com/garyburd/redigo v1.6.2 h1:yE/pwKCrbLpLpQICzYTeZ7JsTA/C53wFTJHaEtRqniM= 7 | github.com/garyburd/redigo v1.6.2/go.mod h1:NR3MbYisc3/PwhQ00EMzDiPmrwpPxAn5GI05/YaO1SY= 8 | github.com/go-task/slim-sprig v0.0.0-20210107165309-348f09dbbbc0/go.mod h1:fyg7847qk6SyHyPtNmDHnmrv/HOrqktSC+C9fM+CJOE= 9 | github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= 10 | github.com/golang/protobuf v1.4.0-rc.1/go.mod h1:ceaxUfeHdC40wWswd/P6IGgMaK3YpKi5j83Wpe3EHw8= 11 | github.com/golang/protobuf v1.4.0-rc.1.0.20200221234624-67d41d38c208/go.mod h1:xKAWHe0F5eneWXFV3EuXVDTCmh+JuBKY0li0aMyXATA= 12 | github.com/golang/protobuf v1.4.0-rc.2/go.mod h1:LlEzMj4AhA7rCAGe4KMBDvJI+AwstrUpVNzEA03Pprs= 13 | github.com/golang/protobuf v1.4.0-rc.4.0.20200313231945-b860323f09d0/go.mod h1:WU3c8KckQ9AFe+yFwt9sWVRKCVIyN9cPHBJSNnbL67w= 14 | github.com/golang/protobuf v1.4.0/go.mod h1:jodUvKwWbYaEsadDk5Fwe5c77LiNKVO9IDvqG2KuDX0= 15 | github.com/golang/protobuf v1.4.2/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI= 16 | github.com/golang/protobuf v1.5.0/go.mod h1:FsONVRAS9T7sI+LIUmWTfcYkHO4aIWwzhcaSAoJOfIk= 17 | github.com/golang/protobuf v1.5.2/go.mod h1:XVQd3VNwM+JqD3oG2Ue2ip4fOMUkwXdXDdiuN0vRsmY= 18 | github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU= 19 | github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU= 20 | github.com/google/go-cmp v0.4.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= 21 | github.com/google/go-cmp v0.5.5/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= 22 | github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU= 23 | github.com/nxadm/tail v1.4.4/go.mod h1:kenIhsEOeOJmVchQTgglprH7qJGnHDVpk1VPCcaMI8A= 24 | github.com/nxadm/tail v1.4.8 h1:nPr65rt6Y5JFSKQO7qToXr7pePgD6Gwiw05lkbyAQTE= 25 | github.com/nxadm/tail v1.4.8/go.mod h1:+ncqLTQzXmGhMZNUePPaPqPvBxHAIsmXswZKocGu+AU= 26 | github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE= 27 | github.com/onsi/ginkgo v1.12.1/go.mod h1:zj2OWP4+oCPe1qIXoGWkgMRwljMUYCdkwsT2108oapk= 28 | github.com/onsi/ginkgo v1.16.2/go.mod h1:CObGmKUOKaSC0RjmoAK7tKyn4Azo5P2IWuoMnvwxz1E= 29 | github.com/onsi/ginkgo v1.16.4 h1:29JGrr5oVBm5ulCWet69zQkzWipVXIol6ygQUe/EzNc= 30 | github.com/onsi/ginkgo v1.16.4/go.mod h1:dX+/inL/fNMqNlz0e9LfyB9TswhZpCVdJM/Z6Vvnwo0= 31 | github.com/onsi/gomega v1.7.1/go.mod h1:XdKZgCCFLUoM/7CFJVPcG8C1xQ1AJ0vpAezJrB7JYyY= 32 | github.com/onsi/gomega v1.10.1/go.mod h1:iN09h71vgCQne3DLsj+A5owkum+a2tYe+TOCB1ybHNo= 33 | github.com/onsi/gomega v1.13.0 h1:7lLHu94wT9Ij0o6EWWclhu0aOh32VxhkwEJvzuWPeak= 34 | github.com/onsi/gomega v1.13.0/go.mod h1:lRk9szgn8TxENtWd0Tp4c3wjlRfMTMH27I+3Je41yGY= 35 | github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= 36 | github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= 37 | github.com/stretchr/testify v1.5.1/go.mod h1:5W2xD1RspED5o8YsWQXVCued0rvSQ+mT+I5cxcmMvtA= 38 | github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= 39 | golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= 40 | golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= 41 | golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= 42 | golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= 43 | golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= 44 | golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= 45 | golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= 46 | golang.org/x/net v0.0.0-20200520004742-59133d7f0dd7/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A= 47 | golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU= 48 | golang.org/x/net v0.0.0-20210428140749-89ef3d95e781 h1:DzZ89McO9/gWPsQXS/FVKAlG02ZjaQ6AlZRBimEYOd0= 49 | golang.org/x/net v0.0.0-20210428140749-89ef3d95e781/go.mod h1:OJAsFXCWl8Ukc7SiCT/9KSuxbyM7479/AVlXFRxuMCk= 50 | golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= 51 | golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= 52 | golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= 53 | golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= 54 | golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= 55 | golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 56 | golang.org/x/sys v0.0.0-20190904154756-749cb33beabd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 57 | golang.org/x/sys v0.0.0-20191005200804-aed5e4c7ecf9/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 58 | golang.org/x/sys v0.0.0-20191120155948-bd437916bb0e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 59 | golang.org/x/sys v0.0.0-20200323222414-85ca7c5b95cd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 60 | golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 61 | golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 62 | golang.org/x/sys v0.0.0-20210112080510-489259a85091/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 63 | golang.org/x/sys v0.0.0-20210423082822-04245dca01da h1:b3NXsE2LusjYGGjL5bxEVZZORm/YEFFrWFjR8eFrw/c= 64 | golang.org/x/sys v0.0.0-20210423082822-04245dca01da/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 65 | golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo= 66 | golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= 67 | golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= 68 | golang.org/x/text v0.3.6 h1:aRYxNxv6iGQlyVaZmk6ZgYEDa+Jg18DxebPSrd6bg1M= 69 | golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= 70 | golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= 71 | golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= 72 | golang.org/x/tools v0.0.0-20201224043029-2b0845dc783e/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA= 73 | golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= 74 | golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= 75 | golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= 76 | golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= 77 | google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8= 78 | google.golang.org/protobuf v0.0.0-20200221191635-4d8936d0db64/go.mod h1:kwYJMbMJ01Woi6D6+Kah6886xMZcty6N08ah7+eCXa0= 79 | google.golang.org/protobuf v0.0.0-20200228230310-ab0ca4ff8a60/go.mod h1:cfTl7dwQJ+fmap5saPgwCLgHXTUD7jkjRqWcaiX5VyM= 80 | google.golang.org/protobuf v1.20.1-0.20200309200217-e05f789c0967/go.mod h1:A+miEFZTKqfCUM6K7xSMQL9OKL/b6hQv+e19PK+JZNE= 81 | google.golang.org/protobuf v1.21.0/go.mod h1:47Nbq4nVaFHyn7ilMalzfO3qCViNmqZ2kzikPIcrTAo= 82 | google.golang.org/protobuf v1.23.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU= 83 | google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw= 84 | google.golang.org/protobuf v1.26.0/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc= 85 | gopkg.in/bsm/ratelimit.v1 v1.0.0-20160220154919-db14e161995a h1:stTHdEoWg1pQ8riaP5ROrjS6zy6wewH/Q2iwnLCQUXY= 86 | gopkg.in/bsm/ratelimit.v1 v1.0.0-20160220154919-db14e161995a/go.mod h1:KF9sEfUPAXdG8Oev9e99iLGnl2uJMjc5B+4y3O7x610= 87 | gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= 88 | gopkg.in/fsnotify.v1 v1.4.7/go.mod h1:Tz8NjZHkW78fSQdbUxIjBTcgA1z1m8ZHf0WmKUhAMys= 89 | gopkg.in/redis.v3 v3.6.4 h1:u7XgPH1rWwsdZnR+azldXC6x9qDU2luydOIeU/l52fE= 90 | gopkg.in/redis.v3 v3.6.4/go.mod h1:6XeGv/CrsUFDU9aVbUdNykN7k1zVmoeg83KC9RbQfiU= 91 | gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7 h1:uRGJdciOHaEIrze2W8Q3AKkepLTh2hOroT7a+7czfdQ= 92 | gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7/go.mod h1:dt/ZhP58zS4L8KSrWDmTeBkI65Dw0HsyUHuEVlX15mw= 93 | gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= 94 | gopkg.in/yaml.v2 v2.2.4/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= 95 | gopkg.in/yaml.v2 v2.3.0/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= 96 | gopkg.in/yaml.v2 v2.4.0 h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY= 97 | gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ= 98 | -------------------------------------------------------------------------------- /logo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/integrii/go-redis-migrator/b2942bed5568d1d5602c0f0759b5cee9e3f6a560/logo.png -------------------------------------------------------------------------------- /main.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | // 4 | // Migrate data from one redis cluster to another. 5 | // 6 | 7 | import ( 8 | "bufio" 9 | "flag" 10 | "fmt" 11 | "log" 12 | "os" 13 | "strconv" 14 | "strings" 15 | "time" 16 | 17 | "gopkg.in/redis.v3" // http://godoc.org/gopkg.in/redis.v3 18 | ) 19 | 20 | // redis server connections 21 | var sourceCluster *redis.ClusterClient 22 | var destinationCluster *redis.ClusterClient 23 | var sourceHost *redis.Client 24 | var destinationHost *redis.Client 25 | 26 | // redis server arrays for connecting 27 | var sourceHostsArray []string 28 | var destinationHostsArray []string 29 | 30 | // cluster indication flags 31 | var sourceIsCluster = false 32 | var destinationIsCluster = false 33 | 34 | // connected states 35 | var sourceClusterConnected = false 36 | var destinationClusterConnected = false 37 | 38 | // counter for number of keys migrated 39 | var keysMigrated int64 40 | 41 | func main() { 42 | 43 | // 44 | // Command line argument handling 45 | // 46 | 47 | // parse command line arguments 48 | var sourceHosts = flag.String("sourceHosts", "", "A list of source cluster host:port servers seperated by commas. EX) 127.0.0.1:6379,127.0.0.1:6380") 49 | var destinationHosts = flag.String("destinationHosts", "", "A list of source cluster host:port servers seperated by spaces. EX) 127.0.0.1:6379,127.0.0.1:6380") 50 | var getKeys = flag.Bool("getKeys", false, "Fetches and prints keys from the source host.") 51 | var copyData = flag.Bool("copyData", false, "Copies all keys in a specified list to the destination cluster from the source cluster.") 52 | var keyFilePath = flag.String("keyFile", "", "The file path which contains the list of keys to migrate.") 53 | var keyFilter = flag.String("keyFilter", "*", "The pattern of keys to migrate if no key file path was specified.") 54 | 55 | // parse the args we are looking for 56 | flag.Parse() 57 | 58 | // Ensure a valid operation was passed 59 | if *getKeys == false && *copyData == false { 60 | showHelp() 61 | } 62 | 63 | // 64 | // Connect to redis servers or clusters 65 | // 66 | 67 | // connect to the host if servers found 68 | // break source hosts comma list into an array 69 | var sourceHostsString = *sourceHosts 70 | if len(sourceHostsString) > 0 { 71 | //log.Println("Source hosts detected: " + sourceHostsString) 72 | // break source hosts string at commas into a slice 73 | sourceHostsArray = strings.Split(sourceHostsString, ",") 74 | connectSourceCluster() 75 | } 76 | 77 | // connect to the host if servers found 78 | // break destination hosts comma list into an array 79 | var destinationHostsString = *destinationHosts 80 | if len(destinationHostsString) > 0 { 81 | //log.Println("Destination hosts detected: " + destinationHostsString) 82 | // break source hosts string at commas into a slice 83 | destinationHostsArray = strings.Split(destinationHostsString, ",") 84 | connectDestinationCluster() 85 | } 86 | 87 | // 88 | // Do the right thing depending on the operations passed from cli 89 | // 90 | 91 | // Get and display a key list 92 | if *getKeys == true { 93 | 94 | // ensure we are connected 95 | if sourceClusterConnected != true { 96 | log.Fatalln("Please specify a source cluster using -sourceCluster=127.0.0.1:6379.") 97 | } 98 | 99 | //log.Println("Getting full key list...") 100 | // iterate through each host in the destination cluster, connect, and 101 | // run KEYS search 102 | var allKeys = getSourceKeys(*keyFilter) 103 | 104 | // see how many keys we fetched 105 | if len(allKeys) > 0 { 106 | // loop through all keys and print them plainly one per line 107 | for i := 0; i < len(allKeys); i++ { 108 | fmt.Println(allKeys[i]) 109 | } 110 | } else { 111 | fmt.Println("No keys found in source cluster.") 112 | } 113 | } 114 | 115 | // Copy all or some keys to the new server/cluster 116 | if *copyData == true { 117 | 118 | // ensure we are connected 119 | if sourceClusterConnected != true { 120 | log.Fatalln("Please specify a source cluster using -sourceCluster=127.0.0.1:6379.") 121 | showHelp() 122 | } 123 | if destinationClusterConnected != true { 124 | log.Fatalln("Please specify a destination cluster using -destinationCluster=127.0.0.1:6379") 125 | showHelp() 126 | } 127 | 128 | // if the key file path was set, open the file 129 | if len(*keyFilePath) > 0 { 130 | 131 | // ensure that keyFile and keyFilter are not both specified 132 | if *keyFilter != "*" { 133 | log.Fatalln("Can not use -keyFilter= option with -keyFile= option.") 134 | } 135 | 136 | var keyFile, err = os.Open(*keyFilePath) 137 | if err != nil { 138 | log.Fatalln("Unable to open key file specified.") 139 | } 140 | // create a new scanner for parsing the io reader returned by the 141 | // os.Open call earlier 142 | var keyFileScanner = bufio.NewScanner(keyFile) 143 | 144 | // read the entire key file 145 | for keyFileScanner.Scan() { 146 | 147 | // fetch the text for this line 148 | var key = keyFileScanner.Text() 149 | 150 | // migrate the key from source to destination 151 | migrateKey(key) 152 | 153 | } 154 | } else { 155 | // This is what we do if no key file was specified 156 | 157 | var allKeys = getSourceKeys(*keyFilter) 158 | 159 | if len(allKeys) > 0 { 160 | // loop through all keys and print them plainly one per line 161 | for i := 0; i < len(allKeys); i++ { 162 | var key = allKeys[i] 163 | migrateKey(key) 164 | } 165 | } else { 166 | fmt.Println("No keys found in source cluster.") 167 | } 168 | } 169 | 170 | } 171 | 172 | // Finish up with some stats 173 | fmt.Println("Migrated " + strconv.FormatInt(keysMigrated, 10) + " keys.") 174 | } 175 | 176 | // ping testing functions 177 | func clusterPingTest(redisClient *redis.ClusterClient) { 178 | var pingTest = redisClient.Ping() 179 | var pingMessage, pingError = pingTest.Result() 180 | if pingError != nil { 181 | log.Fatalln("Error when pinging a Redis connection:" + pingMessage) 182 | } 183 | } 184 | func hostPingTest(redisClient *redis.Client) { 185 | var pingTest = redisClient.Ping() 186 | var pingMessage, pingError = pingTest.Result() 187 | if pingError != nil { 188 | log.Fatalln("Error when pinging a Redis connection:" + pingMessage) 189 | } 190 | } 191 | 192 | // Connects to the source host/cluster 193 | func connectSourceCluster() { 194 | 195 | // connect to source cluster and ping it 196 | if len(sourceHostsArray) == 1 { 197 | sourceHost = redis.NewClient(&redis.Options{ 198 | Addr: sourceHostsArray[0], 199 | }) 200 | sourceIsCluster = false 201 | //log.Println("Source is a single host.") 202 | hostPingTest(sourceHost) 203 | } else { 204 | sourceCluster = redis.NewClusterClient(&redis.ClusterOptions{ 205 | Addrs: sourceHostsArray, 206 | }) 207 | sourceIsCluster = true 208 | //log.Println("Source is a cluster.") 209 | clusterPingTest(sourceCluster) 210 | } 211 | sourceClusterConnected = true 212 | //log.Println("Source connected") 213 | } 214 | 215 | // Connects to the destination host/cluster 216 | func connectDestinationCluster() { 217 | 218 | // connect to destination cluster and ping it 219 | if len(destinationHostsArray) == 1 { 220 | destinationHost = redis.NewClient(&redis.Options{ 221 | Addr: destinationHostsArray[0], 222 | }) 223 | destinationIsCluster = false 224 | //log.Println("Destination is a single host.") 225 | hostPingTest(destinationHost) 226 | } else { 227 | destinationCluster = redis.NewClusterClient(&redis.ClusterOptions{ 228 | Addrs: destinationHostsArray, 229 | }) 230 | destinationIsCluster = true 231 | //log.Println("Destination is a cluster.") 232 | clusterPingTest(destinationCluster) 233 | } 234 | destinationClusterConnected = true 235 | //log.Println("Destination connected.") 236 | } 237 | 238 | // Gets all the keys from the source server/cluster 239 | func getSourceKeys(keyFilter string) []string { 240 | 241 | var allKeys *redis.StringSliceCmd 242 | if sourceIsCluster == true { 243 | allKeys = sourceCluster.Keys(keyFilter) 244 | } else { 245 | allKeys = sourceHost.Keys(keyFilter) 246 | } 247 | 248 | return allKeys.Val() 249 | } 250 | 251 | // Migrates a key from the source cluster to the deestination one 252 | func migrateKey(key string) { 253 | 254 | //log.Println("migrating key:" + key) 255 | 256 | keysMigrated = keysMigrated + 1 257 | 258 | // init a value to hold the key data 259 | var data string 260 | 261 | // init a value to hold the key ttl 262 | var ttl time.Duration 263 | 264 | // get the key from the source 265 | if sourceIsCluster == true { 266 | data = sourceCluster.Dump(key).Val() 267 | ttl = sourceCluster.PTTL(key).Val() 268 | 269 | } else { 270 | data = sourceHost.Dump(key).Val() 271 | ttl = sourceHost.PTTL(key).Val() 272 | } 273 | 274 | // set ttl to 0 due to restore requiring >= 0 for ttl 275 | if ttl == -1 { 276 | ttl = 0 277 | } 278 | 279 | // put the key in the destination cluster and set the ttl 280 | if destinationIsCluster == true { 281 | destinationCluster.Restore(key, ttl, data) 282 | } else { 283 | destinationHost.Restore(key, ttl, data) 284 | } 285 | 286 | return 287 | } 288 | 289 | // Displays the help content 290 | func showHelp() { 291 | fmt.Println(` 292 | - Redis Key Migrator - 293 | https://github.com/integrii/go-redis-migrator 294 | 295 | Migrates all or some of the keys from a Redis host or cluster to a specified host or cluster. Supports migrating TTLs. 296 | go-redis-migrator can also be used to list the keys in a cluster. Run with the -getKey=true and -sourceHosts= flags to do so. 297 | 298 | You must run this program with an operation before it will do anything. 299 | 300 | Flags: 301 | -getKeys=false: Fetches and prints keys from the source host. 302 | -copyData=false: Copies all keys in a list specified by -keyFile= to the destination cluster from the source cluster. 303 | -keyFile="": The file path which contains the list of keys to migrate. One per line. 304 | -keyFilter="*": The filter for which keys to migrate. Does not work when -keyFile is also specified. 305 | -destinationHosts="": A list of source cluster host:port servers seperated by spaces. Use a single host without a ',' if there is no cluster. EX) 127.0.0.1:6379,127.0.0.1:6380 306 | -sourceHosts="": A list of source cluster host:port servers seperated by commas. Use a single host without a ',' if there is no cluster. EX) 127.0.0.1:6379,127.0.0.1:6380 307 | `) 308 | 309 | os.Exit(0) 310 | } 311 | --------------------------------------------------------------------------------