├── .dockerignore ├── Dockerfile ├── LICENSE ├── README.md ├── ali_decoding ├── Makefile ├── ali_decoding--0.0.1.sql ├── ali_decoding--unpackaged--0.0.1.sql ├── ali_decoding.c ├── ali_decoding.control └── test │ └── decode_test.sql ├── dbsync ├── Makefile ├── dbsync-mysql2pgsql.c ├── dbsync-pgsql2pgsql.c ├── demo.cpp ├── ini.c ├── ini.h ├── misc.c ├── misc.h ├── my.cfg ├── mysql2pgsql.c ├── pg_logicaldecode.c ├── pg_logicaldecode.h ├── pgsync.c ├── pgsync.h ├── pqformat.c ├── readcfg.cpp ├── readcfg.h ├── stringinfo.c ├── test │ └── decode_test.sql ├── utils.c └── utils.h └── doc ├── design.md ├── mysql2gp.md ├── mysql2pgsql_ch.md ├── mysql2pgsql_en.md ├── pgsql2pgsql_ch.md └── pgsql2pgsql_en.md /.dockerignore: -------------------------------------------------------------------------------- 1 | .DS_Store 2 | .dockerignore 3 | .gitlab-ci.yml 4 | .gitignore 5 | .git/ 6 | -------------------------------------------------------------------------------- /Dockerfile: -------------------------------------------------------------------------------- 1 | FROM centos:7 2 | 3 | ENV PATH=$PATH:/dbsync/bin 4 | 5 | COPY . /tmp/aliyun/ 6 | 7 | RUN set -ex \ 8 | && { \ 9 | echo '[mysql57-community]'; \ 10 | echo 'name=mysql57-community'; \ 11 | echo 'baseurl=http://repo.mysql.com/yum/mysql-5.7-community/el/$releasever/$basearch/'; \ 12 | echo 'enabled=1'; \ 13 | echo 'gpgcheck=0'; \ 14 | echo '[pgdg10]'; \ 15 | echo 'name=pgdg10'; \ 16 | echo 'baseurl=https://download.postgresql.org/pub/repos/yum/10/redhat/rhel-$releasever-$basearch'; \ 17 | echo 'enabled=1'; \ 18 | echo 'gpgcheck=0'; \ 19 | } > /etc/yum.repos.d/dbsync_deps.repo \ 20 | && cp -ra /var/log/yum.log /tmp/yum.log.old \ 21 | && yum install mysql-community-client mysql-community-devel postgresql10-devel gcc gcc-c++ make unzip -y \ 22 | && update-alternatives --install /usr/bin/pg_config pgsql-pg_config /usr/pgsql-10/bin/pg_config 300 \ 23 | && ( \ 24 | cd /tmp/aliyun/dbsync \ 25 | && make \ 26 | && install -D -d /dbsync/bin /dbsync/lib \ 27 | && install -p -D -m 0755 *2pgsql /dbsync/bin \ 28 | && install -p -D -m 0755 ali_recvlogical.so /dbsync/lib \ 29 | && install -p -D -m 0644 my.cfg ../LICENSE ../README.md /dbsync \ 30 | && ln -sf /usr/share/mysql /dbsync/share \ 31 | ) \ 32 | && update-alternatives --remove pgsql-pg_config /usr/pgsql-10/bin/pg_config \ 33 | && mkdir -p /tmp/extbin \ 34 | && curl -L https://github.com/aliyun/rds_dbsync/files/1555186/mysql2pgsql.bin.el7.20171213.zip -o /tmp/extbin/bin.zip \ 35 | && (cd /tmp/extbin && unzip -o bin.zip && install -p -D -m 0755 mysql2pgsql.bin*/bin/binlog_* /dbsync/bin) \ 36 | && yum remove -y mysql-community-devel postgresql10-devel unzip gcc gcc-c++ make cpp glibc-devel glibc-headers libicu-devel libstdc++-devel kernel-headers \ 37 | && yum clean all && mv /tmp/yum.log.old /var/log/yum.log \ 38 | && rm -rf /tmp/aliyun /tmp/extbin /var/cache/yum/* /etc/yum.repos.d/dbsync_deps.repo \ 39 | && ls -alhR /dbsync && ldd /dbsync/bin/* && mysql --version && psql --version && mysql2pgsql -h 40 | 41 | WORKDIR /dbsync 42 | 43 | CMD ["mysql2pgsql", "-h"] 44 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Apache License 2 | Version 2.0, January 2004 3 | http://www.apache.org/licenses/ 4 | 5 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 6 | 7 | 1. Definitions. 8 | 9 | "License" shall mean the terms and conditions for use, reproduction, 10 | and distribution as defined by Sections 1 through 9 of this document. 11 | 12 | "Licensor" shall mean the copyright owner or entity authorized by 13 | the copyright owner that is granting the License. 14 | 15 | "Legal Entity" shall mean the union of the acting entity and all 16 | other entities that control, are controlled by, or are under common 17 | control with that entity. For the purposes of this definition, 18 | "control" means (i) the power, direct or indirect, to cause the 19 | direction or management of such entity, whether by contract or 20 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 21 | outstanding shares, or (iii) beneficial ownership of such entity. 22 | 23 | "You" (or "Your") shall mean an individual or Legal Entity 24 | exercising permissions granted by this License. 25 | 26 | "Source" form shall mean the preferred form for making modifications, 27 | including but not limited to software source code, documentation 28 | source, and configuration files. 29 | 30 | "Object" form shall mean any form resulting from mechanical 31 | transformation or translation of a Source form, including but 32 | not limited to compiled object code, generated documentation, 33 | and conversions to other media types. 34 | 35 | "Work" shall mean the work of authorship, whether in Source or 36 | Object form, made available under the License, as indicated by a 37 | copyright notice that is included in or attached to the work 38 | (an example is provided in the Appendix below). 39 | 40 | "Derivative Works" shall mean any work, whether in Source or Object 41 | form, that is based on (or derived from) the Work and for which the 42 | editorial revisions, annotations, elaborations, or other modifications 43 | represent, as a whole, an original work of authorship. For the purposes 44 | of this License, Derivative Works shall not include works that remain 45 | separable from, or merely link (or bind by name) to the interfaces of, 46 | the Work and Derivative Works thereof. 47 | 48 | "Contribution" shall mean any work of authorship, including 49 | the original version of the Work and any modifications or additions 50 | to that Work or Derivative Works thereof, that is intentionally 51 | submitted to Licensor for inclusion in the Work by the copyright owner 52 | or by an individual or Legal Entity authorized to submit on behalf of 53 | the copyright owner. For the purposes of this definition, "submitted" 54 | means any form of electronic, verbal, or written communication sent 55 | to the Licensor or its representatives, including but not limited to 56 | communication on electronic mailing lists, source code control systems, 57 | and issue tracking systems that are managed by, or on behalf of, the 58 | Licensor for the purpose of discussing and improving the Work, but 59 | excluding communication that is conspicuously marked or otherwise 60 | designated in writing by the copyright owner as "Not a Contribution." 61 | 62 | "Contributor" shall mean Licensor and any individual or Legal Entity 63 | on behalf of whom a Contribution has been received by Licensor and 64 | subsequently incorporated within the Work. 65 | 66 | 2. Grant of Copyright License. Subject to the terms and conditions of 67 | this License, each Contributor hereby grants to You a perpetual, 68 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 69 | copyright license to reproduce, prepare Derivative Works of, 70 | publicly display, publicly perform, sublicense, and distribute the 71 | Work and such Derivative Works in Source or Object form. 72 | 73 | 3. Grant of Patent License. Subject to the terms and conditions of 74 | this License, each Contributor hereby grants to You a perpetual, 75 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 76 | (except as stated in this section) patent license to make, have made, 77 | use, offer to sell, sell, import, and otherwise transfer the Work, 78 | where such license applies only to those patent claims licensable 79 | by such Contributor that are necessarily infringed by their 80 | Contribution(s) alone or by combination of their Contribution(s) 81 | with the Work to which such Contribution(s) was submitted. If You 82 | institute patent litigation against any entity (including a 83 | cross-claim or counterclaim in a lawsuit) alleging that the Work 84 | or a Contribution incorporated within the Work constitutes direct 85 | or contributory patent infringement, then any patent licenses 86 | granted to You under this License for that Work shall terminate 87 | as of the date such litigation is filed. 88 | 89 | 4. Redistribution. You may reproduce and distribute copies of the 90 | Work or Derivative Works thereof in any medium, with or without 91 | modifications, and in Source or Object form, provided that You 92 | meet the following conditions: 93 | 94 | (a) You must give any other recipients of the Work or 95 | Derivative Works a copy of this License; and 96 | 97 | (b) You must cause any modified files to carry prominent notices 98 | stating that You changed the files; and 99 | 100 | (c) You must retain, in the Source form of any Derivative Works 101 | that You distribute, all copyright, patent, trademark, and 102 | attribution notices from the Source form of the Work, 103 | excluding those notices that do not pertain to any part of 104 | the Derivative Works; and 105 | 106 | (d) If the Work includes a "NOTICE" text file as part of its 107 | distribution, then any Derivative Works that You distribute must 108 | include a readable copy of the attribution notices contained 109 | within such NOTICE file, excluding those notices that do not 110 | pertain to any part of the Derivative Works, in at least one 111 | of the following places: within a NOTICE text file distributed 112 | as part of the Derivative Works; within the Source form or 113 | documentation, if provided along with the Derivative Works; or, 114 | within a display generated by the Derivative Works, if and 115 | wherever such third-party notices normally appear. The contents 116 | of the NOTICE file are for informational purposes only and 117 | do not modify the License. You may add Your own attribution 118 | notices within Derivative Works that You distribute, alongside 119 | or as an addendum to the NOTICE text from the Work, provided 120 | that such additional attribution notices cannot be construed 121 | as modifying the License. 122 | 123 | You may add Your own copyright statement to Your modifications and 124 | may provide additional or different license terms and conditions 125 | for use, reproduction, or distribution of Your modifications, or 126 | for any such Derivative Works as a whole, provided Your use, 127 | reproduction, and distribution of the Work otherwise complies with 128 | the conditions stated in this License. 129 | 130 | 5. Submission of Contributions. Unless You explicitly state otherwise, 131 | any Contribution intentionally submitted for inclusion in the Work 132 | by You to the Licensor shall be under the terms and conditions of 133 | this License, without any additional terms or conditions. 134 | Notwithstanding the above, nothing herein shall supersede or modify 135 | the terms of any separate license agreement you may have executed 136 | with Licensor regarding such Contributions. 137 | 138 | 6. Trademarks. This License does not grant permission to use the trade 139 | names, trademarks, service marks, or product names of the Licensor, 140 | except as required for reasonable and customary use in describing the 141 | origin of the Work and reproducing the content of the NOTICE file. 142 | 143 | 7. Disclaimer of Warranty. Unless required by applicable law or 144 | agreed to in writing, Licensor provides the Work (and each 145 | Contributor provides its Contributions) on an "AS IS" BASIS, 146 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 147 | implied, including, without limitation, any warranties or conditions 148 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 149 | PARTICULAR PURPOSE. You are solely responsible for determining the 150 | appropriateness of using or redistributing the Work and assume any 151 | risks associated with Your exercise of permissions under this License. 152 | 153 | 8. Limitation of Liability. In no event and under no legal theory, 154 | whether in tort (including negligence), contract, or otherwise, 155 | unless required by applicable law (such as deliberate and grossly 156 | negligent acts) or agreed to in writing, shall any Contributor be 157 | liable to You for damages, including any direct, indirect, special, 158 | incidental, or consequential damages of any character arising as a 159 | result of this License or out of the use or inability to use the 160 | Work (including but not limited to damages for loss of goodwill, 161 | work stoppage, computer failure or malfunction, or any and all 162 | other commercial damages or losses), even if such Contributor 163 | has been advised of the possibility of such damages. 164 | 165 | 9. Accepting Warranty or Additional Liability. While redistributing 166 | the Work or Derivative Works thereof, You may choose to offer, 167 | and charge a fee for, acceptance of support, warranty, indemnity, 168 | or other liability obligations and/or rights consistent with this 169 | License. However, in accepting such obligations, You may act only 170 | on Your own behalf and on Your sole responsibility, not on behalf 171 | of any other Contributor, and only if You agree to indemnify, 172 | defend, and hold each Contributor harmless for any liability 173 | incurred by, or claims asserted against, such Contributor by reason 174 | of your accepting any such warranty or additional liability. 175 | 176 | END OF TERMS AND CONDITIONS 177 | 178 | APPENDIX: How to apply the Apache License to your work. 179 | 180 | To apply the Apache License to your work, attach the following 181 | boilerplate notice, with the fields enclosed by brackets "{}" 182 | replaced with your own identifying information. (Don't include 183 | the brackets!) The text should be enclosed in the appropriate 184 | comment syntax for the file format. We also recommend that a 185 | file or class name and description of purpose be included on the 186 | same "printed page" as the copyright notice for easier 187 | identification within third-party archives. 188 | 189 | Copyright {yyyy} {name of copyright owner} 190 | 191 | Licensed under the Apache License, Version 2.0 (the "License"); 192 | you may not use this file except in compliance with the License. 193 | You may obtain a copy of the License at 194 | 195 | http://www.apache.org/licenses/LICENSE-2.0 196 | 197 | Unless required by applicable law or agreed to in writing, software 198 | distributed under the License is distributed on an "AS IS" BASIS, 199 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 200 | See the License for the specific language governing permissions and 201 | limitations under the License. 202 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # dbsync 项目 2 | 3 | dbsync 项目目标是围绕 PostgreSQL Greenplum ,实现易用的数据的互迁功能。 4 | 5 | ## 支持的功能 6 | 7 | 1. PostgreSQL -> PostgreSQL pgsql2pgsql 8 | 9 | 功能 pg->pg 全量+增量数据同步 10 | 11 | 状态：已开源 [文档](doc/pgsql2pgsql_ch.md) 12 | 13 | 2. MySQL -> PostgreSQL/Greenplum（binlog_minner binlog_loader） 14 | 15 | 功能：基于 MySQL binlog 解析的增量数据同步 16 | 17 | 状态：已开放二进制 [文档](doc/mysql2gp.md) 18 | 19 | 3. PostgreSQL -> PostgreSQL/Greenplum pgsql2gp 20 | 21 | 功能：基于 PostgreSQL 逻辑日志的增量数据同步 22 | 23 | 状态：未开发完成 24 | 25 | 4. MySQL -> PostgreSQL/Greenplum mysql2pgsql 26 | 27 | 功能：以表为单位的多线程全量数据迁移 28 | 29 | 状态：已开源 [文档](doc/mysql2pgsql_ch.md) 30 | 31 | 32 | ## 项目成员 33 | 该项目由阿里云 PostgreSQL 小组开发，为 PostgreSQL 世界贡献一份力量 34 | 35 | 1. PM & 架构设计 曾文旌（义从） 36 | 2. PD 萧少聪（铁庵） 37 | 3. TESTER & 技术支持 周正中(德歌) 38 | 4. DEV 张广舟（明虚）曾文旌（义从） 39 | 40 | ## 使用方法 41 | 1. 修改配置文件 my.cfg 中相关的项，例如需求 MySQL -> PostgreSQL 全量迁移，不需要增量，则只需要配置 src.mysql 和 desc.pgsql ，其他的项不用管。 42 | 2. 执行对应二进制，在二进制所在目录执行 ./mysql2pgsql 43 | 44 | ## 编译步骤 45 | 46 | ### 从零开始 47 | 48 | 1. 下载代码 49 | 50 | `git clone https://github.com/aliyun/rds_dbsync.git` 51 | 52 | 2. 下载安装mysql的开发包 53 | 54 | 下载repo的rpm： `wget http://dev.mysql.com/get/mysql57-community-release-el6-9.noarch.rpm` 55 | 56 | 安装repo：rpm -Uvh mysql57-community-release-el6-9.noarch.rpm 57 | 58 | 编辑 /etc/yum.repos.d/mysql-community.repo，把除mysql 57外的其他repo的enable设为0 59 | 60 | 查看可安装的mysql报：yum list mysql-community-* 61 | 62 | 安装mysql的开发包： yum install mysql-community-devel.x86_64 63 | 64 | 3. 下载安装pg的安装包 65 | 66 | 下载repo的rpm： `wget https://download.postgresql.org/pub/repos/yum/9.6/redhat/rhel-6-x86_64/pgdg-centos96-9.6-3.noarch.rpm` 67 | 68 | 安装repo：rpm -ivh pgdg-centos96-9.6-3.noarch.rpm 69 | 70 | 编辑/etc/yum.repos.d/pgdg-96-centos.repo，可能需要把https改成http 71 | 72 | 查看可安装的pg包：yum list postgresql96* 73 | 74 | 安装pg的server和开发包：yum install postgresql96-devel.x86_64 postgresql96-server.x86_64 75 | 76 | 4. 执行make 77 | 78 | 5. 打包二进制 make package 将生成一个install目录，里面有二进制和lib 79 | 80 | 6. 执行dbsync：cd install; bin/mysql2pgsql ; bin/pgsql2pgsql ; bin/demo 81 | 82 | ### 打包docker镜像 83 | 以上手动步骤，已集成进 [Dockerfile](Dockerfile)，运行 `docker build .` 无意外即完成编译过程，镜像内同时包含 `binlog_minner` `binlog_loader` 两个二进制文件。 84 | 85 | ## 问题反馈 86 | 有任何问题，请反馈到 https://github.com/aliyun/rds_dbsync issues 或联系 158306855@qq.com 87 | -------------------------------------------------------------------------------- /ali_decoding/Makefile: -------------------------------------------------------------------------------- 1 | # contrib/ali_decoding/Makefile 2 | MODULE_big = ali_decoding 3 | MODULES = ali_decoding 4 | OBJS = ali_decoding.o 5 | 6 | DATA = ali_decoding--0.0.1.sql ali_decoding--unpackaged--0.0.1.sql 7 | 8 | EXTENSION = ali_decoding 9 | 10 | NAME = ali_decoding 11 | 12 | #subdir = contrib/ali_decoding 13 | #top_builddir = ../.. 14 | #include $(top_builddir)/src/Makefile.global 15 | #include $(top_srcdir)/contrib/contrib-global.mk 16 | 17 | PG_CONFIG = pg_config 18 | pgsql_lib_dir := $(shell $(PG_CONFIG) --libdir) 19 | PGXS := $(shell $(PG_CONFIG) --pgxs) 20 | include $(PGXS) 21 | 22 | -------------------------------------------------------------------------------- /ali_decoding/ali_decoding--0.0.1.sql: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aliyun/rds_dbsync/edc3f92e71830e9d41b75c721a2ef7729835afcf/ali_decoding/ali_decoding--0.0.1.sql -------------------------------------------------------------------------------- /ali_decoding/ali_decoding--unpackaged--0.0.1.sql: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aliyun/rds_dbsync/edc3f92e71830e9d41b75c721a2ef7729835afcf/ali_decoding/ali_decoding--unpackaged--0.0.1.sql -------------------------------------------------------------------------------- /ali_decoding/ali_decoding.c: -------------------------------------------------------------------------------- 1 | /*------------------------------------------------------------------------- 2 | * 3 | * ali_decoding.c 4 | * ali logical decoding output plugin 5 | * 6 | * 7 | * 8 | * IDENTIFICATION 9 | * contrib/ali_decoding/ali_decoding.c 10 | * 11 | *------------------------------------------------------------------------- 12 | */ 13 | #include "postgres.h" 14 | 15 | #include "access/sysattr.h" 16 | 17 | #include "catalog/pg_class.h" 18 | #include "catalog/pg_type.h" 19 | 20 | #include "nodes/parsenodes.h" 21 | 22 | #include "replication/output_plugin.h" 23 | #include "replication/logical.h" 24 | 25 | #include "utils/builtins.h" 26 | #include "utils/lsyscache.h" 27 | #include "utils/memutils.h" 28 | #include "utils/rel.h" 29 | #include "utils/relcache.h" 30 | #include "utils/syscache.h" 31 | #include "utils/typcache.h" 32 | 33 | #include "libpq/pqformat.h" 34 | #include "access/tuptoaster.h" 35 | #include "mb/pg_wchar.h" 36 | #include "utils/guc.h" 37 | 38 | 39 | PG_MODULE_MAGIC; 40 | 41 | /* These must be available to pg_dlsym() */ 42 | extern void _PG_init(void); 43 | extern void _PG_output_plugin_init(OutputPluginCallbacks *cb); 44 | 45 | typedef struct 46 | { 47 | MemoryContext context; 48 | 49 | bool allow_binary_protocol; 50 | bool allow_sendrecv_protocol; 51 | bool int_datetime_mismatch; 52 | 53 | uint32 client_version; 54 | 55 | size_t client_sizeof_int; 56 | size_t client_sizeof_long; 57 | size_t client_sizeof_datum; 58 | size_t client_maxalign; 59 | bool client_bigendian; 60 | bool client_float4_byval; 61 | bool client_float8_byval; 62 | bool client_int_datetime; 63 | 64 | char *client_encoding; 65 | 66 | bool output_key_info; 67 | bool output_column_info; 68 | bool output_type_as_name; 69 | } Ali_OutputData; 70 | 71 | static void pg_decode_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt, 72 | bool is_init); 73 | static void pg_decode_shutdown(LogicalDecodingContext *ctx); 74 | static void pg_decode_begin_txn(LogicalDecodingContext *ctx, 75 | ReorderBufferTXN *txn); 76 | static void pg_decode_commit_txn(LogicalDecodingContext *ctx, 77 | ReorderBufferTXN *txn, XLogRecPtr commit_lsn); 78 | static void pg_decode_change(LogicalDecodingContext *ctx, 79 | ReorderBufferTXN *txn, Relation rel, 80 | ReorderBufferChange *change); 81 | static void write_rel(StringInfo out, Relation rel, Ali_OutputData *data, int action); 82 | static void write_tuple(Ali_OutputData *data, StringInfo out, Relation rel, 83 | HeapTuple tuple); 84 | static void write_colum_info(StringInfo out, Relation rel, Ali_OutputData *data, int action); 85 | static void parse_notnull(DefElem *elem, const char *paramtype); 86 | static void parse_uint32(DefElem *elem, uint32 *res); 87 | 88 | void 89 | _PG_init(void) 90 | { 91 | /* other plugins can perform things here */ 92 | } 93 | 94 | /* specify output plugin callbacks */ 95 | void 96 | _PG_output_plugin_init(OutputPluginCallbacks *cb) 97 | { 98 | AssertVariableIsOfType(&_PG_output_plugin_init, LogicalOutputPluginInit); 99 | 100 | cb->startup_cb = pg_decode_startup; 101 | cb->begin_cb = pg_decode_begin_txn; 102 | cb->change_cb = pg_decode_change; 103 | cb->commit_cb = pg_decode_commit_txn; 104 | cb->shutdown_cb = pg_decode_shutdown; 105 | } 106 | 107 | 108 | /* initialize this plugin */ 109 | static void 110 | pg_decode_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt, 111 | bool is_init) 112 | { 113 | ListCell *option; 114 | Ali_OutputData *data; 115 | 116 | data = palloc0(sizeof(Ali_OutputData)); 117 | data->context = AllocSetContextCreate(ctx->context, 118 | "ali decode conversion context", 119 | ALLOCSET_DEFAULT_MINSIZE, 120 | ALLOCSET_DEFAULT_INITSIZE, 121 | ALLOCSET_DEFAULT_MAXSIZE); 122 | 123 | ctx->output_plugin_private = data; 124 | 125 | opt->output_type = OUTPUT_PLUGIN_BINARY_OUTPUT; 126 | 127 | foreach(option, ctx->output_plugin_options) 128 | { 129 | DefElem *elem = lfirst(option); 130 | 131 | Assert(elem->arg == NULL || IsA(elem->arg, String)); 132 | 133 | if (strcmp(elem->defname, "version") == 0) 134 | parse_uint32(elem, &data->client_version); 135 | else if (strcmp(elem->defname, "encoding") == 0) 136 | data->client_encoding = pstrdup(strVal(elem->arg)); 137 | else 138 | { 139 | ereport(ERROR, 140 | (errcode(ERRCODE_INVALID_PARAMETER_VALUE), 141 | errmsg("option \"%s\" = \"%s\" is unknown", 142 | elem->defname, 143 | elem->arg ? strVal(elem->arg) : "(null)"))); 144 | } 145 | } 146 | 147 | if (!is_init) 148 | { 149 | /* Set defaults values */ 150 | if(data->client_encoding == NULL) 151 | { 152 | data->client_encoding = pstrdup(GetDatabaseEncodingName()); 153 | } 154 | 155 | /* fix me */ 156 | data->allow_binary_protocol = false; 157 | data->allow_sendrecv_protocol = false; 158 | data->int_datetime_mismatch = false; 159 | 160 | data->output_key_info = true; 161 | data->output_column_info = true; 162 | data->output_type_as_name = true; 163 | 164 | if (strcmp(data->client_encoding, GetDatabaseEncodingName()) != 0) 165 | elog(ERROR, "mismatching encodings are not yet supported"); 166 | 167 | if (extra_float_digits < 3) 168 | (void) set_config_option("extra_float_digits", "3", 169 | PGC_USERSET, PGC_S_SESSION, 170 | GUC_ACTION_SAVE, true, 0, false); 171 | } 172 | 173 | return; 174 | } 175 | 176 | /* cleanup this plugin's resources */ 177 | static void 178 | pg_decode_shutdown(LogicalDecodingContext *ctx) 179 | { 180 | Ali_OutputData *data = ctx->output_plugin_private; 181 | 182 | if(data->client_encoding != NULL) 183 | { 184 | pfree(data->client_encoding); 185 | data->client_encoding = NULL; 186 | } 187 | 188 | /* cleanup our own resources via memory context reset */ 189 | MemoryContextDelete(data->context); 190 | 191 | pfree(ctx->output_plugin_private); 192 | ctx->output_plugin_private = NULL; 193 | } 194 | 195 | /* 196 | * BEGIN callback 197 | */ 198 | static void 199 | pg_decode_begin_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn) 200 | { 201 | int flags = 0; 202 | MemoryContext old; 203 | Ali_OutputData *data; 204 | 205 | data = ctx->output_plugin_private; 206 | old = MemoryContextSwitchTo(data->context); 207 | 208 | AssertVariableIsOfType(&pg_decode_begin_txn, LogicalDecodeBeginCB); 209 | 210 | OutputPluginPrepareWrite(ctx, true); 211 | pq_sendbyte(ctx->out, 'B'); /* BEGIN */ 212 | 213 | /* send the flags field its self */ 214 | pq_sendint(ctx->out, flags, 4); 215 | 216 | /* fixed fields */ 217 | pq_sendint64(ctx->out, txn->final_lsn); 218 | pq_sendint64(ctx->out, txn->commit_time); 219 | pq_sendint(ctx->out, txn->xid, 4); 220 | 221 | OutputPluginWrite(ctx, true); 222 | MemoryContextSwitchTo(old); 223 | 224 | return; 225 | } 226 | 227 | /* 228 | * COMMIT callback 229 | * 230 | * Send the LSN at the time of the commit, the commit time, and the end LSN. 231 | * 232 | * The presence of additional records is controlled by a flag field, with 233 | * records that're present appearing strictly in the order they're listed 234 | * here. There is no sub-record header or other structure beyond the flags 235 | * field. 236 | * 237 | * If you change this, you'll need to change process_remote_commit(...) 238 | * too. Make sure to keep any flags in sync. 239 | */ 240 | static void 241 | pg_decode_commit_txn(LogicalDecodingContext *ctx, ReorderBufferTXN *txn, 242 | XLogRecPtr commit_lsn) 243 | { 244 | int flags = 0; 245 | MemoryContext old; 246 | Ali_OutputData *data; 247 | 248 | data = ctx->output_plugin_private; 249 | old = MemoryContextSwitchTo(data->context); 250 | 251 | OutputPluginPrepareWrite(ctx, true); 252 | pq_sendbyte(ctx->out, 'C'); /* sending COMMIT */ 253 | 254 | /* send the flags field its self */ 255 | pq_sendint(ctx->out, flags, 4); 256 | 257 | /* Send fixed fields */ 258 | pq_sendint64(ctx->out, commit_lsn); 259 | pq_sendint64(ctx->out, txn->end_lsn); 260 | pq_sendint64(ctx->out, txn->commit_time); 261 | 262 | OutputPluginWrite(ctx, true); 263 | MemoryContextSwitchTo(old); 264 | 265 | return; 266 | } 267 | 268 | void 269 | pg_decode_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn, 270 | Relation relation, ReorderBufferChange *change) 271 | { 272 | Ali_OutputData *data; 273 | MemoryContext old; 274 | 275 | data = ctx->output_plugin_private; 276 | 277 | /* Avoid leaking memory by using and resetting our own context */ 278 | old = MemoryContextSwitchTo(data->context); 279 | 280 | OutputPluginPrepareWrite(ctx, true); 281 | 282 | switch (change->action) 283 | { 284 | case REORDER_BUFFER_CHANGE_INSERT: 285 | pq_sendbyte(ctx->out, 'I'); /* action INSERT */ 286 | write_rel(ctx->out, relation, data, change->action); 287 | pq_sendbyte(ctx->out, 'N'); /* new tuple follows */ 288 | write_tuple(data, ctx->out, relation, &change->data.tp.newtuple->tuple); 289 | break; 290 | case REORDER_BUFFER_CHANGE_UPDATE: 291 | pq_sendbyte(ctx->out, 'U'); /* action UPDATE */ 292 | write_rel(ctx->out, relation, data, change->action); 293 | if (change->data.tp.oldtuple != NULL) 294 | { 295 | pq_sendbyte(ctx->out, 'K'); /* old key follows */ 296 | write_tuple(data, ctx->out, relation, 297 | &change->data.tp.oldtuple->tuple); 298 | } 299 | pq_sendbyte(ctx->out, 'N'); /* new tuple follows */ 300 | write_tuple(data, ctx->out, relation, 301 | &change->data.tp.newtuple->tuple); 302 | break; 303 | case REORDER_BUFFER_CHANGE_DELETE: 304 | pq_sendbyte(ctx->out, 'D'); /* action DELETE */ 305 | write_rel(ctx->out, relation, data, change->action); 306 | if (change->data.tp.oldtuple != NULL) 307 | { 308 | pq_sendbyte(ctx->out, 'K'); /* old key follows */ 309 | write_tuple(data, ctx->out, relation, 310 | &change->data.tp.oldtuple->tuple); 311 | } 312 | else 313 | pq_sendbyte(ctx->out, 'E'); /* empty */ 314 | break; 315 | default: 316 | Assert(false); 317 | } 318 | OutputPluginWrite(ctx, true); 319 | 320 | MemoryContextSwitchTo(old); 321 | MemoryContextReset(data->context); 322 | 323 | } 324 | 325 | /* 326 | * Write schema.relation to the output stream. 327 | */ 328 | static void 329 | write_rel(StringInfo out, Relation rel, Ali_OutputData *data, int action) 330 | { 331 | const char *nspname; 332 | int64 nspnamelen; 333 | const char *relname; 334 | int64 relnamelen; 335 | 336 | nspname = get_namespace_name(rel->rd_rel->relnamespace); 337 | if (nspname == NULL) 338 | elog(ERROR, "cache lookup failed for namespace %u", 339 | rel->rd_rel->relnamespace); 340 | nspnamelen = strlen(nspname) + 1; 341 | 342 | relname = NameStr(rel->rd_rel->relname); 343 | relnamelen = strlen(relname) + 1; 344 | 345 | pq_sendint(out, nspnamelen, 2); /* schema name length */ 346 | appendBinaryStringInfo(out, nspname, nspnamelen); 347 | 348 | pq_sendint(out, relnamelen, 2); /* table name length */ 349 | appendBinaryStringInfo(out, relname, relnamelen); 350 | 351 | if (data->output_column_info == true) 352 | { 353 | write_colum_info(out, rel, data, action); 354 | } 355 | } 356 | 357 | /* 358 | * Write a tuple to the outputstream, in the most efficient format possible. 359 | */ 360 | static void 361 | write_tuple(Ali_OutputData *data, StringInfo out, Relation rel, 362 | HeapTuple tuple) 363 | { 364 | TupleDesc desc; 365 | Datum values[MaxTupleAttributeNumber]; 366 | bool isnull[MaxTupleAttributeNumber]; 367 | int i; 368 | 369 | desc = RelationGetDescr(rel); 370 | 371 | pq_sendbyte(out, 'T'); /* tuple follows */ 372 | 373 | pq_sendint(out, desc->natts, 4); /* number of attributes */ 374 | 375 | /* try to allocate enough memory from the get go */ 376 | enlargeStringInfo(out, tuple->t_len + 377 | desc->natts * ( 1 + 4)); 378 | 379 | /* 380 | * XXX: should this prove to be a relevant bottleneck, it might be 381 | * interesting to inline heap_deform_tuple() here, we don't actually need 382 | * the information in the form we get from it. 383 | */ 384 | heap_deform_tuple(tuple, desc, values, isnull); 385 | 386 | for (i = 0; i < desc->natts; i++) 387 | { 388 | HeapTuple typtup; 389 | Form_pg_type typclass; 390 | char *outputstr = NULL; 391 | int len = 0; 392 | 393 | Form_pg_attribute att = desc->attrs[i]; 394 | 395 | if (isnull[i] || att->attisdropped) 396 | { 397 | pq_sendbyte(out, 'n'); /* null column */ 398 | continue; 399 | } 400 | else if (att->attlen == -1 && VARATT_IS_EXTERNAL_ONDISK(values[i])) 401 | { 402 | pq_sendbyte(out, 'u'); /* unchanged toast column */ 403 | continue; 404 | } 405 | 406 | typtup = SearchSysCache1(TYPEOID, ObjectIdGetDatum(att->atttypid)); 407 | if (!HeapTupleIsValid(typtup)) 408 | elog(ERROR, "cache lookup failed for type %u", att->atttypid); 409 | typclass = (Form_pg_type) GETSTRUCT(typtup); 410 | 411 | pq_sendbyte(out, 't'); /* 'text' data follows */ 412 | 413 | outputstr = 414 | OidOutputFunctionCall(typclass->typoutput, values[i]); 415 | len = strlen(outputstr) + 1; 416 | pq_sendint(out, len, 4); /* length */ 417 | appendBinaryStringInfo(out, outputstr, len); /* data */ 418 | pfree(outputstr); 419 | 420 | ReleaseSysCache(typtup); 421 | } 422 | } 423 | 424 | static void 425 | write_colum_info(StringInfo out, Relation rel, Ali_OutputData *data, int action) 426 | { 427 | TupleDesc desc; 428 | int i; 429 | 430 | desc = RelationGetDescr(rel); 431 | 432 | pq_sendbyte(out, 'C'); /* tuple follows */ 433 | 434 | pq_sendint(out, desc->natts, 2); /* number of attributes */ 435 | 436 | for (i = 0; i < desc->natts; i++) 437 | { 438 | int attlen; 439 | const char *attname = NULL; 440 | int typelen; 441 | char *typname = NULL; 442 | 443 | Form_pg_attribute att = desc->attrs[i]; 444 | 445 | if (att->attisdropped) 446 | { 447 | pq_sendint(out, 0, 2); 448 | continue; 449 | } 450 | 451 | if (att->attnum < 0) 452 | { 453 | pq_sendint(out, 0, 2); 454 | continue; 455 | } 456 | 457 | attname = quote_identifier(NameStr(att->attname)); 458 | attlen = strlen(attname) + 1; 459 | 460 | typname = format_type_be(att->atttypid); 461 | typelen = strlen(typname) + 1; 462 | 463 | pq_sendint(out, attlen, 2); 464 | appendBinaryStringInfo(out, attname, attlen); 465 | 466 | if (data->output_type_as_name) 467 | { 468 | pq_sendint(out, typelen, 2); 469 | appendBinaryStringInfo(out, typname, typelen); 470 | } 471 | 472 | } 473 | 474 | if ((action == REORDER_BUFFER_CHANGE_UPDATE || 475 | action == REORDER_BUFFER_CHANGE_DELETE) && 476 | data->output_key_info == true) 477 | { 478 | Oid idxoid; 479 | Relation idxrel; 480 | TupleDesc idx_desc; 481 | int idxnatt; 482 | List *latt = NULL; 483 | ListCell *cell = NULL; 484 | 485 | if (rel->rd_indexvalid == 0) 486 | RelationGetIndexList(rel); 487 | idxoid = rel->rd_replidindex; 488 | if (!OidIsValid(idxoid)) 489 | { 490 | pq_sendbyte(out, 'P'); 491 | return; 492 | } 493 | 494 | idxrel = RelationIdGetRelation(idxoid); 495 | idx_desc = RelationGetDescr(idxrel); 496 | for (idxnatt = 0; idxnatt < idx_desc->natts; idxnatt++) 497 | { 498 | int attno = idxrel->rd_index->indkey.values[idxnatt]; 499 | char *attname; 500 | 501 | if (attno < 0) 502 | { 503 | if (attno == ObjectIdAttributeNumber) 504 | continue; 505 | elog(ERROR, "system column in index"); 506 | } 507 | 508 | attname = get_relid_attribute_name(RelationGetRelid(rel), attno); 509 | latt = lappend(latt, makeString(attname)); 510 | } 511 | 512 | RelationClose(idxrel); 513 | 514 | pq_sendbyte(out, 'M'); 515 | idxnatt = list_length(latt); 516 | pq_sendint(out, idxnatt, 2); 517 | foreach(cell, latt) 518 | { 519 | char *col = strVal(lfirst(cell)); 520 | int len = strlen(col) + 1; 521 | 522 | pq_sendint(out, len, 2); 523 | appendBinaryStringInfo(out, col, len); 524 | } 525 | 526 | list_free_deep(latt); 527 | } 528 | else 529 | { 530 | pq_sendbyte(out, 'P'); 531 | } 532 | 533 | return; 534 | } 535 | 536 | static void 537 | parse_uint32(DefElem *elem, uint32 *res) 538 | { 539 | parse_notnull(elem, "uint32"); 540 | errno = 0; 541 | *res = strtoul(strVal(elem->arg), NULL, 0); 542 | 543 | if (errno != 0) 544 | ereport(ERROR, 545 | (errcode(ERRCODE_INVALID_PARAMETER_VALUE), 546 | errmsg("could not parse uint32 value \"%s\" for parameter \"%s\": %m", 547 | strVal(elem->arg), elem->defname))); 548 | } 549 | 550 | static void 551 | parse_notnull(DefElem *elem, const char *paramtype) 552 | { 553 | if (elem->arg == NULL || strVal(elem->arg) == NULL) 554 | ereport(ERROR, 555 | (errcode(ERRCODE_INVALID_PARAMETER_VALUE), 556 | errmsg("%s parameter \"%s\" had no value", 557 | paramtype, elem->defname))); 558 | } 559 | -------------------------------------------------------------------------------- /ali_decoding/ali_decoding.control: -------------------------------------------------------------------------------- 1 | comment = 'ali_decoding' 2 | default_version = '0.0.1' 3 | module_pathname = '$libdir/ali_decoding' 4 | relocatable = true 5 | -------------------------------------------------------------------------------- /ali_decoding/test/decode_test.sql: -------------------------------------------------------------------------------- 1 | 2 | create schema test_case; 3 | set search_path=test_case; 4 | 5 | -- no index 6 | create table a(a int ,b text, c timestamptz); 7 | 8 | insert into a values(1,'test','1999-01-08 04:05:06'); 9 | insert into a values(2,'test','1999-01-08 04:05:06'); 10 | insert into a values(3,'test','1999-01-08 04:05:06'); 11 | 12 | update a set b = 'test1'; 13 | update a set b = 'test2' where a = 3; 14 | 15 | delete from a where a = 2; 16 | delete from a; 17 | 18 | -- primary key 19 | create table b(a int primary key ,b text, c timestamptz); 20 | 21 | insert into b values(1,'test','1999-01-08 04:05:06'); 22 | insert into b values(2,'test','1999-01-08 04:05:06'); 23 | insert into b values(3,'test','1999-01-08 04:05:06'); 24 | 25 | update b set b = 'test1'; 26 | update b set a = 5 where a = 1; 27 | update b set b = 'test2' where a = 3; 28 | update b set c = '1999-01-08 04:05:06' where a = 5; 29 | update b set a = 6, c = '1999-01-08 04:05:06' where a = 5; 30 | update b set a = 5, b = 't',c = '1999-01-08 04:05:06' where a = 6; 31 | 32 | delete from b where a = 2; 33 | delete from b; 34 | 35 | -- mprimary key 36 | create table c(a int ,b text, c timestamptz, d bigint, primary key(a,d)); 37 | 38 | insert into c values(1,'test','1999-01-08 04:05:06',3); 39 | insert into c values(2,'test','1999-01-08 04:05:06',2); 40 | insert into c values(3,'test','1999-01-08 04:05:06',1); 41 | 42 | update c set b = 'test1'; 43 | update c set a = 5 where a = 1; 44 | update c set b = null where a = 3; 45 | delete from c where a = 2; 46 | 47 | -- REPLICA index 48 | create table d(a int ,b text, c timestamptz, d bigint); 49 | CREATE UNIQUE INDEX idx_d_a_d ON d(a,d); 50 | alter table d ALTER COLUMN a set not null; 51 | alter table d ALTER COLUMN d set not null; 52 | alter table d REPLICA IDENTITY USING INDEX idx_d_a_d; 53 | 54 | insert into d values(1,'test','1999-01-08 04:05:06',3); 55 | insert into d values(2,'test','1999-01-08 04:05:06',2); 56 | insert into d values(3,'test','1999-01-08 04:05:06',1); 57 | 58 | update d set b = 'test1'; 59 | update d set a = 5 where a = 1; 60 | update d set b = 'test2' where a = 3; 61 | update d set a = 5, b = 't',c = '1999-01-08 04:05:06' where a = 3; 62 | delete from d; 63 | 64 | -- full data 65 | create table e(a int ,b text, c timestamptz, d bigint); 66 | alter table e REPLICA IDENTITY FULL; 67 | 68 | insert into e values(1,'test','1999-01-08 04:05:06',3); 69 | insert into e values(2,'test','1999-01-08 04:05:06',2); 70 | insert into e values(3,'test','1999-01-08 04:05:06',1); 71 | 72 | update e set b = 'test1'; 73 | update e set a = 5 where a = 1; 74 | update e set b = 'test2' where a = 3; 75 | update e set a = 5, b = 't',c = '1999-01-08 04:05:06' where a = 3; 76 | 77 | delete from e; 78 | 79 | 80 | -- full data and primary key 81 | create table f(a int primary key,b text, c timestamptz, d bigint); 82 | alter table f REPLICA IDENTITY FULL; 83 | 84 | insert into f values(1,'test','1999-01-08 04:05:06',3); 85 | insert into f values(2,'test','1999-01-08 04:05:06',2); 86 | insert into f values(3,'test','1999-01-08 04:05:06',1); 87 | 88 | update f set b = 'test1'; 89 | update f set a = 5 where a = 1; 90 | 91 | alter table f REPLICA IDENTITY DEFAULT; 92 | update f set a = 7 where a = 2; 93 | 94 | update f set b = 'test2' where a = 3; 95 | update f set a = 6, b = 't',c = '1999-01-08 04:05:06' where a = 3; 96 | 97 | delete from f; 98 | 99 | -- data type 100 | create table test_data_type_1(a smallint,b integer,c bigint,d decimal,e numeric); 101 | insert into test_data_type_1 values(-32768, 2147483647, 9223372036854775807, 111.111, 111.111); 102 | 103 | create table test_data_type_2(a real,b double precision,c smallserial,d serial,e bigserial); 104 | insert into test_data_type_2 values(111.111, 111.111, 32767, 2147483647, 9223372036854775807); 105 | 106 | create table test_data_type_3(a money,b character varying(20),c character(20),d text,e char(20)); 107 | insert into test_data_type_3 values('12.34', '12.34', '12.34', '12.34', '12.34'); 108 | 109 | create table test_data_type_4(a bytea,b bytea,c bytea,d bytea,e bytea); 110 | insert into test_data_type_4 values('\\xDEADBEEF', '\\000', '0', '\\134', '\\176'); 111 | 112 | create table test_data_type_5(a timestamp without time zone ,b timestamp with time zone,c timestamp,d time,e time with time zone); 113 | insert into test_data_type_5 values('1999-01-08 04:05:06', '1999-01-08 04:05:06 +8:00', '1999-01-08 04:05:06 -8:00', '1999-01-08 04:05:06 -8:00', '1999-01-08 04:05:06 -8:00'); 114 | 115 | CREATE TYPE mood AS ENUM ('sad', 'ok', 'happy'); 116 | create table test_data_type_6(a boolean,b mood,c point,d line,e lseg); 117 | insert into test_data_type_6 values(TRUE, 'happy', '(1,1)', '{1,2,1}', '[(1,2),(2,1)]'); 118 | 119 | create table test_data_type_7(a path,b path,c polygon,d circle,e circle); 120 | insert into test_data_type_7 values('[(1,3),(2,2)]', '((1,3),(2,2))', '((1,3),(2,2))', '<(2,2),2>', '((2,3),1)'); 121 | 122 | create table test_data_type_8(a cidr,b cidr,c inet,d macaddr,e macaddr); 123 | insert into test_data_type_8 values('192.168.100.128/25', '2001:4f8:3:ba:2e0:81ff:fe22:d1f1/128', '1.2.3.4', '08-00-2b-01-02-03', '08:00:2b:01:02:03'); 124 | 125 | CREATE TABLE test_9 (a BIT(3), b BIT VARYING(5)); 126 | INSERT INTO test_9 VALUES (B'101', B'00'); 127 | INSERT INTO test_9 VALUES (B'10'::bit(3), B'101'); 128 | 129 | CREATE TABLE test_10 (a tsvector, b tsvector,c tsquery, d tsquery); 130 | INSERT INTO test_10 VALUES ('a fat cat sat on a mat and ate a fat rat', 'a:1 fat:2 cat:3 sat:4 on:5 a:6 mat:7 and:8 ate:9 a:10 fat:11 rat:12','fat & rat','Fat:ab & Cats'); 131 | 132 | create extension "uuid-ossp"; 133 | CREATE TABLE test_11 (a uuid, b uuid,c uuid, d uuid); 134 | INSERT INTO test_11 VALUES ('25285134-7314-11e5-8e45-d89d672b3560', 'c12a3d5f-53bb-4223-9fca-0af78b4d269f', 'cf16fe52-3365-3a1f-8572-288d8d2aaa46', '252852d8-7314-11e5-8e45-2f1f0837ccca'); 135 | 136 | CREATE TABLE test_12 (a xml, b xml,c xml); 137 | INSERT INTO test_12 VALUES (xml 'bar', XMLPARSE (DOCUMENT 'Manual...'), XMLPARSE (CONTENT 'abcbarfoo')); 138 | 139 | CREATE TABLE test_13 (a xml, b xml,c xml); 140 | INSERT INTO test_13 VALUES ('{"reading": 1.230e-5}', 141 | '[1, 2, "foo", null]', 142 | '{"bar": "baz", "balance": 7.77, "active":false}'); 143 | 144 | CREATE TABLE sal_emp_14 ( 145 | name text, 146 | pay_by_quarter integer[], 147 | schedule text[][] 148 | ); 149 | 150 | INSERT INTO sal_emp_14 151 | VALUES ('Bill', 152 | '{10000, 10000, 10000, 10000}', 153 | '{{"meeting", "lunch"}, {"training", "presentation"}}'); 154 | 155 | INSERT INTO sal_emp_14 156 | VALUES ('Carol', 157 | '{20000, 25000, 25000, 25000}', 158 | '{{"breakfast", "consulting"}, {"meeting", "lunch"}}'); 159 | 160 | INSERT INTO sal_emp_14 161 | VALUES ('Carol', 162 | ARRAY[20000, 25000, 25000, 25000], 163 | ARRAY[['breakfast', 'consulting'], ['meeting', 'lunch']]); 164 | 165 | CREATE TYPE complex AS ( 166 | r double precision, 167 | i double precision 168 | ); 169 | 170 | CREATE TYPE inventory_item AS ( 171 | name text, 172 | supplier_id integer, 173 | price numeric 174 | ); 175 | 176 | CREATE TABLE on_hand_15 ( 177 | item inventory_item, 178 | count integer 179 | ); 180 | 181 | INSERT INTO on_hand_15 VALUES (ROW('fuzzy dice', 42, 1.99), 1000); 182 | 183 | CREATE TABLE reservation_16 (room int, during tsrange); 184 | INSERT INTO reservation_16 VALUES 185 | (1108, '[2010-01-01 14:30, 2010-01-01 15:30)'); 186 | 187 | CREATE TYPE floatrange AS RANGE ( 188 | subtype = float8, 189 | subtype_diff = float8mi 190 | ); 191 | 192 | create table t_range_16(a floatrange); 193 | insert into t_range_16 values('[1.234, 5.678]'); 194 | 195 | create extension hstore; 196 | create table hstore_test_17(item_id serial, data hstore); 197 | INSERT INTO hstore_test_17 (data) VALUES ('"key1"=>"value1", "key2"=>"value2", "key3"=>"value3"'); 198 | UPDATE hstore_test_17 SET data = delete(data, 'key2'); 199 | UPDATE hstore_test_17 SET data = data || '"key4"=>"some value"'::hstore; 200 | 201 | CREATE EXTENSION postgis; 202 | CREATE EXTENSION postgis_topology; 203 | CREATE EXTENSION fuzzystrmatch; 204 | CREATE EXTENSION postgis_tiger_geocoder; 205 | 206 | create table test_18 (myID int4, pt geometry, myName varchar ); 207 | insert into test_18 values (1, 'POINT(0 0)', 'beijing' ); 208 | insert into test_18 values (2, 'MULTIPOINT(1 1, 3 4, -1 3)', 'shanghai' ); 209 | insert into test_18 values (3, 'LINESTRING(1 1, 2 2, 3 4)', 'tianjin' ); 210 | insert into test_18 values (3, 'POLYGON((0 0, 0 1, 1 1, 1 0, 0 0))', 'tianjin' ); 211 | insert into test_18 values (3, 'MULTIPOLYGON(((0 0,4 0,4 4,0 4,0 0),(1 1,2 1,2 2,1 2,1 1)), ((-1 -1,-1 -2,-2 -2,-2 -1,-1 -1)))', 'tianjin' ); 212 | insert into test_18 values (3, 'MULTILINESTRING((1 1, 2 2, 3 4),(2 2, 3 3, 4 5))', 'tianjin' ); 213 | 214 | insert into test_18 values (3, '01060000000200000001030000000200000005000000000000000000000000000000000000000000000000001040000000000000000000000000000010400000000000001040000000000000000000000000000010400000000000000000000000000000000005000000000000000000F03F000000000000F03F0000000000000040000000000000F03F00000000000000400000000000000040000000000000F03F0000000000000040000000000000F03F000000000000F03F01030000000100000005000000000000000000F0BF000000000000F0BF000000000000F0BF00000000000000C000000000000000C000000000000000C000000000000000C0000000000000F0BF000000000000F0BF000000000000F0BF', 'tianjin' ); 215 | 216 | -- m sql in a tran 217 | create table msql(a int primary key,b text, c timestamptz); 218 | 219 | begin; 220 | insert into msql values(1,'test','1999-01-08 04:05:06'); 221 | insert into msql values(2,'test','1999-01-08 04:05:06'); 222 | insert into msql values(3,'test','1999-01-08 04:05:06'); 223 | update msql set b = 'test' where a = 1; 224 | delete from msql where a = 3; 225 | commit; 226 | 227 | -- alter table 228 | create table msql_1(a int primary key,b text, c timestamptz); 229 | 230 | insert into msql_1 values(1,'test','1999-01-08 04:05:06'); 231 | alter table msql_1 add COLUMN d int; 232 | insert into msql_1 values(2,'test','1999-01-08 04:05:06',1); 233 | alter table msql_1 drop COLUMN b; 234 | insert into msql_1 values(3,'1999-01-08 04:05:06',2); 235 | 236 | update msql_1 set c = '1999-01-08 04:05:07'; 237 | delete from msql_1; 238 | 239 | -- alter table in a tran 240 | create table msql_2(a int primary key,b text, c timestamptz); 241 | begin; 242 | insert into msql_2 values(1,'test','1999-01-08 04:05:06'); 243 | alter table msql_2 add COLUMN d int; 244 | insert into msql_2 values(2,'test','1999-01-08 04:05:06',1); 245 | alter table msql_2 drop COLUMN b; 246 | insert into msql_2 values(3,'1999-01-08 04:05:06',2); 247 | update msql_2 set c = '1999-01-08 04:05:07'; 248 | commit; 249 | 250 | 251 | -- alter table drop pk 252 | create table msql_3(a int primary key,b text, c timestamptz); 253 | begin; 254 | insert into msql_3 values(1,'test','1999-01-08 04:05:06'); 255 | insert into msql_3 values(5,'test','1999-01-08 04:05:06'); 256 | alter table msql_3 add COLUMN d int; 257 | insert into msql_3 values(2,'test','1999-01-08 04:05:06',1); 258 | alter table msql_3 drop COLUMN a; 259 | insert into msql_3 values('test','1999-01-08 04:05:06',2); 260 | delete from msql_3; 261 | commit; 262 | 263 | -- SERIAL 264 | CREATE TABLE seq_test 265 | ( 266 | id SERIAL primary key , 267 | name text 268 | ) ; 269 | 270 | insert into seq_test (name) values('test'); 271 | 272 | -- toast 273 | create table t_kenyon(id int,vname varchar(48),remark text); 274 | select oid,relname,reltoastrelid from pg_class where relname = 't_kenyon'; 275 | insert into t_kenyon select generate_series(1,2000),repeat('kenyon here'||'^_^',2),repeat('^_^ Kenyon is not God',500); 276 | insert into t_kenyon select generate_series(1,2),repeat('kenyon here'||'^_^',2),repeat('^_^ Kenyon is not God,Remark here!!',2000); 277 | insert into t_kenyon select generate_series(3,4),repeat('kenyon here'||'^_^',2),repeat('^_^ Kenyon is not God,Remark here!!',4000); 278 | insert into t_kenyon select generate_series(5,6),repeat('kenyon here'||'^_^',2),repeat('^_^ Kenyon is not God,Remark here!!',5500); 279 | insert into t_kenyon select generate_series(1,2),repeat('kenyon here'||'^_^',2),repeat('^_^ Kenyon is not God,Remark here!!',10000); 280 | insert into t_kenyon select generate_series(7,8),repeat('kenyon here'||'^_^',2),repeat('^_^ Kenyon is not God,Remark here!!',20000); 281 | 282 | -- utf8 283 | create table chinese_text(t text); 284 | insert into chinese_text values('微软 Surface Pro 4 中国开放预售 价格公布'); 285 | insert into chinese_text values('\'); 286 | insert into chinese_text values('\\'); 287 | insert into chinese_text values('\\\'); 288 | insert into chinese_text values('///'); 289 | insert into chinese_text values('//'); 290 | insert into chinese_text values('/'); 291 | insert into chinese_text values(''''); 292 | insert into chinese_text values('"''"'); 293 | 294 | -- bug extra_float_digits default 3 295 | create table tf(c1 float4, c2 float8 ,c3 numeric); 296 | insert into tf values (1.5555555555555555555555,1.5555555555555555555555,1.5555555555555555555555); 297 | 298 | drop schema test_case cascade; 299 | -------------------------------------------------------------------------------- /dbsync/Makefile: -------------------------------------------------------------------------------- 1 | # Minimal requriement: Mysql >= 5.6, Pgsql >= 9.3 2 | MYSQL_CONFIG = mysql_config 3 | mysql_include_dir = $(shell $(MYSQL_CONFIG) --variable=pkgincludedir) 4 | mysql_lib_dir = $(shell $(MYSQL_CONFIG) --variable=pkglibdir) 5 | 6 | PGFILEDESC = "ali_recvlogical" 7 | NAME = ali_recvlogical 8 | MODULE_big = ali_recvlogical 9 | MODULES = ali_recvlogical 10 | 11 | OBJS = pg_logicaldecode.o pqformat.o stringinfo.o utils.o misc.o pgsync.o ini.o 12 | 13 | PG_CPPFLAGS = -DFRONTEND -I$(srcdir) -I$(libpq_srcdir) -I$(mysql_include_dir) 14 | PG_FLAGS = -DFRONTEND -I$(srcdir) -I$(libpq_srcdir) -I$(mysql_include_dir) 15 | 16 | PG_CONFIG = pg_config 17 | pgsql_lib_dir := $(shell $(PG_CONFIG) --libdir) 18 | PGXS := $(shell $(PG_CONFIG) --pgxs) 19 | include $(PGXS) 20 | 21 | RPATH_LDFLAGS='-Wl,-rpath,$$ORIGIN,-rpath,$$ORIGIN/lib,-rpath,$$ORIGIN/../lib,-rpath,$(mysql_lib_dir),-rpath,$(pgsql_lib_dir)' 22 | export RPATH_LDFLAGS 23 | 24 | LIBS = -lpthread 25 | 26 | all: demo.o dbsync-pgsql2pgsql.o mysql2pgsql.o dbsync-mysql2pgsql.o readcfg.o 27 | $(CXX) $(CFLAGS) demo.o $(OBJS) $(libpq_pgport) $(RPATH_LDFLAGS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o demo 28 | $(CXX) $(CFLAGS) readcfg.o dbsync-pgsql2pgsql.o $(OBJS) $(libpq_pgport) $(RPATH_LDFLAGS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o pgsql2pgsql 29 | $(CXX) $(CFLAGS) readcfg.o ini.o mysql2pgsql.o dbsync-mysql2pgsql.o misc.o stringinfo.o $(libpq_pgport) $(RPATH_LDFLAGS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -L$(mysql_lib_dir) -lmysqlclient -o mysql2pgsql 30 | 31 | clean: 32 | rm -rf *.o pgsql2pgsql mysql2pgsql demo ali_recvlogical.so 33 | 34 | package: 35 | mkdir -p install 36 | mkdir -p install/bin 37 | mkdir -p install/lib 38 | cp -fr pgsql2pgsql install/bin 39 | cp -fr demo install/bin 40 | cp -fr ali_recvlogical.so install/lib 41 | cp -fr mysql2pgsql install/bin 42 | cp -fr $(mysql_lib_dir)/libmysqlclient.so* install/lib 43 | cp -fr $(pgsql_lib_dir)/libpq.so* install/lib 44 | -------------------------------------------------------------------------------- /dbsync/dbsync-mysql2pgsql.c: -------------------------------------------------------------------------------- 1 | 2 | #include "postgres_fe.h" 3 | #include "libpq-fe.h" 4 | 5 | #include "pgsync.h" 6 | #include "ini.h" 7 | #include "mysql.h" 8 | #include 9 | 10 | extern bool get_ddl_only; 11 | extern bool simple_wo_part; 12 | extern bool first_col_as_dist_key; 13 | extern int buffer_size; 14 | 15 | static int load_table_list_file(const char *filename, char*** p_tables, char*** p_queries); 16 | 17 | 18 | int 19 | main(int argc, char **argv) 20 | { 21 | char *desc = NULL; 22 | mysql_conn_info src = {0}; 23 | int num_thread = 5; //Default to 5 24 | void *cfg = NULL; 25 | char *sport = NULL; 26 | //char *tabname = NULL; 27 | int res_getopt = 0; 28 | char *target_schema = NULL; 29 | char *table_list_file = NULL; 30 | char *cfg_file="my.cfg"; 31 | char **tables = NULL, **queries = NULL; 32 | char *ignore_copy_error_count_each_table_str = NULL; 33 | uint32 ignore_copy_error_count_each_table = 0; 34 | 35 | while ((res_getopt = getopt(argc, argv, ":l:c:j:dnfhs:b:")) != -1) 36 | { 37 | switch (res_getopt) 38 | { 39 | case 'l': 40 | table_list_file = optarg; 41 | break; 42 | case 'c': 43 | cfg_file = optarg; 44 | break; 45 | case 'j': 46 | num_thread = atoi(optarg); 47 | break; 48 | case 'b': 49 | buffer_size = 1024 * atoi(optarg); 50 | break; 51 | case 's': 52 | target_schema = optarg; 53 | break; 54 | case ':': 55 | fprintf(stderr, "No value specified for -%c\n", optopt); 56 | break; 57 | case 'd': 58 | get_ddl_only = true; 59 | break; 60 | case 'n': 61 | simple_wo_part = true; 62 | break; 63 | case 'f': 64 | first_col_as_dist_key = true; 65 | break; 66 | case 'h': 67 | fprintf(stderr, "Usage: -l -j -d -n -f -s -b -h\n"); 68 | fprintf(stderr, "\n -l specifies a file with table listed;\n -j specifies number of threads to do the job;\n -d means get DDL only without fetching data;\n -n means no partion info in DDLs;\n -f means taking first column as distribution key;\n -s specifies the target schema;\n -b specifies the buffer size in KB used to sending copy data to target db, the default is 0\n -h display this usage manual\n"); 69 | return 0; 70 | case '?': 71 | fprintf(stderr, "Unsupported option: %c", optopt); 72 | break; 73 | default: 74 | fprintf(stderr, "Parameter parsing error: %c", res_getopt); 75 | return -1; 76 | 77 | } 78 | } 79 | 80 | cfg = init_config(cfg_file); 81 | if (cfg == NULL) 82 | { 83 | fprintf(stderr, "read config file error, insufficient permissions or my.cfg does not exist"); 84 | return 1; 85 | } 86 | 87 | memset(&src, 0, sizeof(mysql_conn_info)); 88 | get_config(cfg, "src.mysql", "host", &src.host); 89 | get_config(cfg, "src.mysql", "port", &sport); 90 | get_config(cfg, "src.mysql", "user", &src.user); 91 | get_config(cfg, "src.mysql", "password", &src.passwd); 92 | get_config(cfg, "src.mysql", "db", &src.db); 93 | get_config(cfg, "src.mysql", "encodingdir", &src.encodingdir); 94 | get_config(cfg, "src.mysql", "encoding", &src.encoding); 95 | get_config(cfg, "desc.pgsql", "connect_string", &desc); 96 | get_config(cfg, "desc.pgsql", "ignore_copy_error_count_each_table", &ignore_copy_error_count_each_table_str); 97 | 98 | if (src.host == NULL || sport == NULL || 99 | src.user == NULL || src.passwd == NULL || 100 | src.db == NULL || src.encodingdir == NULL || 101 | src.encoding == NULL || desc == NULL) 102 | { 103 | fprintf(stderr, "parameter error, the necessary parameter is empty\n"); 104 | return 1; 105 | } 106 | 107 | src.port = atoi(sport); 108 | 109 | if (ignore_copy_error_count_each_table_str) 110 | { 111 | ignore_copy_error_count_each_table = atoi(ignore_copy_error_count_each_table_str); 112 | } 113 | 114 | fprintf(stderr, "ignore copy error count %u each table\n", ignore_copy_error_count_each_table); 115 | 116 | if (table_list_file != NULL) 117 | { 118 | if (load_table_list_file(table_list_file, &tables, &queries)) 119 | { 120 | fprintf(stderr, "Error occurs while loading table list file %s \n", table_list_file); 121 | return -1; 122 | } 123 | 124 | src.tabnames = (char **) tables; 125 | src.queries = (char**) queries; 126 | } 127 | 128 | /* Only one thread is needed when just generating DDL */ 129 | if (get_ddl_only) 130 | num_thread = 1; 131 | 132 | return mysql2pgsql_sync_main(desc , num_thread, &src, target_schema, ignore_copy_error_count_each_table); 133 | } 134 | 135 | 136 | int load_table_list_file(const char *filename, char*** p_tables, char*** p_queries) { 137 | FILE *fp = NULL; 138 | int n, sz, num_lines = 0; 139 | char *table_list = NULL; 140 | char **table_array = NULL; 141 | char **query_array = NULL; 142 | char *p = NULL; 143 | char *tail = NULL; 144 | char *table_begin = NULL; 145 | char *table_end = NULL; 146 | char *query_begin = NULL; 147 | int cur_table = 0; 148 | 149 | /* Open file */ 150 | fp = fopen(filename, "rb"); 151 | if (!fp) { 152 | fprintf(stderr, "Error opening file %s", filename); 153 | goto fail; 154 | } 155 | 156 | /* Get file size */ 157 | fseek(fp, 0, SEEK_END); 158 | sz = ftell(fp); 159 | if(sz < 0) { 160 | fprintf(stderr, "Error ftell file %s", filename); 161 | goto fail; 162 | } 163 | rewind(fp); 164 | 165 | /* Load file content into memory, null terminate, init end var */ 166 | table_list = (char*) palloc0(sz + 1); 167 | if (!table_list) 168 | { 169 | fprintf(stderr, "Error malloc mem for file %s", filename); 170 | goto fail; 171 | } 172 | 173 | table_list[sz] = '\0'; 174 | p = table_list; 175 | tail = table_list + sz; 176 | n = fread(table_list, 1, sz, fp); 177 | if (n != sz) { 178 | fprintf(stderr, "Error reading file %s", filename); 179 | goto fail; 180 | } 181 | 182 | /* Count lines of the file */ 183 | while(p < tail) 184 | { 185 | switch (*p) 186 | { 187 | case '\0': 188 | fprintf(stderr, "Unexpected terminator encountered in file %s \n", filename); 189 | goto fail; 190 | case '\n': 191 | num_lines++; 192 | break; 193 | 194 | default: 195 | break; 196 | } 197 | p++; 198 | } 199 | 200 | /* Add the last line */ 201 | num_lines++; 202 | 203 | /* Get memory for table array, with the last element being NULL */ 204 | table_array = (char **) palloc0((num_lines + 1) * sizeof(char*)); 205 | query_array = (char **) palloc0((num_lines + 1) * sizeof(char*)); 206 | 207 | /* Parse data */ 208 | p = table_list; 209 | table_begin = table_list; 210 | cur_table = 0; 211 | while(p <= tail) 212 | { 213 | if (*p == '\n' || p == tail) 214 | { 215 | /* Get the table name without leanding and trailing blanks 216 | * E.g. following line will generate a table name "tab 1" 217 | * | tab 1 : select * from tab | 218 | */ 219 | while (*table_begin == ' ' || *table_begin == '\t') 220 | table_begin++; 221 | 222 | table_end = table_begin; 223 | while (*table_end != ':' && table_end != p) 224 | table_end++; 225 | 226 | query_begin = table_end + 1; 227 | table_end--; 228 | while (table_end >= table_begin && 229 | (*table_end == ' ' || *table_end == '\t')) 230 | table_end--; 231 | 232 | if (table_end < table_begin) 233 | table_begin = NULL; 234 | else 235 | *(table_end+1) = '\0'; 236 | 237 | while ((*query_begin == ' ' || *query_begin == '\t') && query_begin < p) 238 | query_begin++; 239 | 240 | if (query_begin >= p) 241 | query_begin = NULL; 242 | else 243 | *p = '\0'; 244 | 245 | if (table_begin) 246 | { 247 | table_array[cur_table] = table_begin; 248 | query_array[cur_table] = query_begin; 249 | cur_table++; 250 | fprintf(stderr, "-- Adding table: %s\n", table_begin); 251 | } 252 | 253 | table_begin = p + 1; 254 | } 255 | 256 | p++; 257 | } 258 | 259 | /* Clean up and return */ 260 | fclose(fp); 261 | *p_tables = table_array; 262 | *p_queries = query_array; 263 | return 0; 264 | 265 | fail: 266 | if (fp) 267 | fclose(fp); 268 | if (table_list) 269 | free(table_list); 270 | if (table_array) 271 | free(table_array); 272 | if (query_array) 273 | free(query_array); 274 | 275 | return -1; 276 | } 277 | 278 | -------------------------------------------------------------------------------- /dbsync/dbsync-pgsql2pgsql.c: -------------------------------------------------------------------------------- 1 | 2 | #include "postgres_fe.h" 3 | #include "lib/stringinfo.h" 4 | 5 | #include 6 | 7 | #include "common/fe_memutils.h" 8 | #include "libpq-fe.h" 9 | #include "libpq/pqsignal.h" 10 | #include "pqexpbuffer.h" 11 | #include "libpq/pqformat.h" 12 | 13 | #include "pg_logicaldecode.h" 14 | #include "pgsync.h" 15 | #include "ini.h" 16 | 17 | int 18 | main(int argc, char **argv) 19 | { 20 | char *src = NULL; 21 | char *desc = NULL; 22 | char *local = NULL; 23 | void *cfg = NULL; 24 | 25 | cfg = init_config("my.cfg"); 26 | if (cfg == NULL) 27 | { 28 | fprintf(stderr, "read config file error, insufficient permissions or my.cfg does not exist"); 29 | return 1; 30 | } 31 | 32 | get_config(cfg, "src.pgsql", "connect_string", &src); 33 | get_config(cfg, "local.pgsql", "connect_string", &local); 34 | get_config(cfg, "desc.pgsql", "connect_string", &desc); 35 | 36 | if (src == NULL || desc == NULL || local == NULL) 37 | { 38 | fprintf(stderr, "parameter error, the necessary parameter is empty"); 39 | return 1; 40 | } 41 | 42 | return db_sync_main(src, desc, local ,5); 43 | } 44 | 45 | -------------------------------------------------------------------------------- /dbsync/demo.cpp: -------------------------------------------------------------------------------- 1 | /*------------------------------------------------------------------------- 2 | * 3 | * pg_recvlogical.c - receive data from a logical decoding slot in a streaming 4 | * fashion and write it to a local file. 5 | * 6 | * Portions Copyright (c) 1996-2014, PostgreSQL Global Development Group 7 | * 8 | * IDENTIFICATION 9 | * src/bin/pg_basebackup/pg_recvlogical.c 10 | *------------------------------------------------------------------------- 11 | */ 12 | 13 | #include "postgres_fe.h" 14 | 15 | #include 16 | #include 17 | #include 18 | 19 | #include "lib/stringinfo.h" 20 | 21 | #include "access/xlog_internal.h" 22 | #include "common/fe_memutils.h" 23 | #include "getopt_long.h" 24 | #include "libpq-fe.h" 25 | #include "libpq/pqsignal.h" 26 | #include "pqexpbuffer.h" 27 | #include "libpq/pqformat.h" 28 | 29 | #include "pg_logicaldecode.h" 30 | #include "catalog/catversion.h" 31 | 32 | 33 | /* Time to sleep between reconnection attempts */ 34 | #define RECONNECT_SLEEP_TIME 5 35 | 36 | static int standby_message_timeout = 10 * 1000; /* 10 sec = default */ 37 | 38 | static volatile bool time_to_abort = false; 39 | 40 | /* 41 | * Unfortunately we can't do sensible signal handling on windows... 42 | */ 43 | #ifndef WIN32 44 | 45 | /* 46 | * When sigint is called, just tell the system to exit at the next possible 47 | * moment. 48 | */ 49 | static void 50 | sigint_handler(int signum) 51 | { 52 | time_to_abort = true; 53 | } 54 | 55 | #endif 56 | 57 | 58 | int 59 | main(int argc, char **argv) 60 | { 61 | Decoder_handler *hander; 62 | int rc = 0; 63 | bool init = false; 64 | 65 | hander = init_hander(); 66 | hander->connection_string = (char *)"host=192.168.1.1 port=3001 dbname=test user=test password=123456"; 67 | XLogRecPtr lsn; 68 | 69 | init_logfile(hander); 70 | rc = check_handler_parameters(hander); 71 | if(rc != 0) 72 | { 73 | exit(1); 74 | } 75 | 76 | rc = initialize_connection(hander); 77 | if(rc != 0) 78 | { 79 | exit(1); 80 | } 81 | init_streaming(hander); 82 | init = true; 83 | 84 | #ifndef WIN32 85 | signal(SIGINT, sigint_handler); 86 | #endif 87 | 88 | if (hander->do_drop_slot) 89 | { 90 | rc = drop_replication_slot(hander); 91 | return 0; 92 | } 93 | 94 | if (hander->do_create_slot) 95 | { 96 | if(create_replication_slot(hander, &lsn, (char *)"test") == NULL) 97 | { 98 | exit(1); 99 | } 100 | } 101 | 102 | if (!hander->do_start_slot) 103 | { 104 | disconnect(hander); 105 | exit(0); 106 | } 107 | 108 | while (true) 109 | { 110 | ALI_PG_DECODE_MESSAGE *msg = NULL; 111 | 112 | if (time_to_abort) 113 | { 114 | if (hander->copybuf != NULL) 115 | { 116 | PQfreemem(hander->copybuf); 117 | hander->copybuf = NULL; 118 | } 119 | if (hander->conn) 120 | { 121 | PQfinish(hander->conn); 122 | hander->conn = NULL; 123 | } 124 | break; 125 | } 126 | 127 | if (!init) 128 | { 129 | initialize_connection(hander); 130 | init_streaming(hander); 131 | init = true; 132 | } 133 | 134 | msg = exec_logical_decoder(hander, &time_to_abort); 135 | if (msg != NULL) 136 | { 137 | out_put_decode_message(hander, msg, hander->outfd); 138 | hander->flushpos = hander->recvpos; 139 | } 140 | else 141 | { 142 | //printf("%s: disconnected; waiting %d seconds to try again\n",hander->progname, RECONNECT_SLEEP_TIME); 143 | pg_sleep(RECONNECT_SLEEP_TIME * 1000000); 144 | init = false; 145 | } 146 | } 147 | } 148 | 149 | -------------------------------------------------------------------------------- /dbsync/ini.c: -------------------------------------------------------------------------------- 1 | 2 | #include 3 | #include 4 | #include 5 | #include 6 | 7 | #include "postgres_fe.h" 8 | #include "port.h" 9 | #include "ini.h" 10 | 11 | struct ini_t { 12 | char *data; 13 | char *end; 14 | }; 15 | 16 | /* Case insensitive string compare */ 17 | static int strcmpci(const char *a, const char *b) { 18 | for (;;) { 19 | int d = tolower(*a) - tolower(*b); 20 | if (d != 0 || !*a) { 21 | return d; 22 | } 23 | a++, b++; 24 | } 25 | } 26 | 27 | /* Returns the next string in the split data */ 28 | static char* next(ini_t *ini, char *p) { 29 | p += strlen(p); 30 | while (p < ini->end && *p == '\0') { 31 | p++; 32 | } 33 | return p; 34 | } 35 | 36 | static void trim_back(ini_t *ini, char *p) { 37 | while (p >= ini->data && (*p == ' ' || *p == '\t' || *p == '\r')) { 38 | *p-- = '\0'; 39 | } 40 | } 41 | 42 | static char* discard_line(ini_t *ini, char *p) { 43 | while (p < ini->end && *p != '\n') { 44 | *p++ = '\0'; 45 | } 46 | return p; 47 | } 48 | 49 | static char *unescape_quoted_value(ini_t *ini, char *p) { 50 | /* Use `q` as write-head and `p` as read-head, `p` is always ahead of `q` 51 | * as escape sequences are always larger than their resultant data */ 52 | char *q = p; 53 | p++; 54 | while (p < ini->end && *p != '"' && *p != '\r' && *p != '\n') { 55 | if (*p == '\\') { 56 | /* Handle escaped char */ 57 | p++; 58 | switch (*p) { 59 | default : *q = *p; break; 60 | case 'r' : *q = '\r'; break; 61 | case 'n' : *q = '\n'; break; 62 | case 't' : *q = '\t'; break; 63 | case '\r' : 64 | case '\n' : 65 | case '\0' : goto end; 66 | } 67 | 68 | } else { 69 | /* Handle normal char */ 70 | *q = *p; 71 | } 72 | q++, p++; 73 | } 74 | end: 75 | return q; 76 | } 77 | 78 | /* Splits data in place into strings containing section-headers, keys and 79 | * values using one or more '\0' as a delimiter. Unescapes quoted values */ 80 | static void split_data(ini_t *ini) { 81 | char *value_start, *line_start; 82 | char *p = ini->data; 83 | 84 | while (p < ini->end) { 85 | switch (*p) { 86 | case '\r': 87 | case '\n': 88 | case '\t': 89 | case ' ': 90 | *p = '\0'; 91 | /* Fall through */ 92 | 93 | case '\0': 94 | p++; 95 | break; 96 | 97 | case '[': 98 | p += strcspn(p, "]\n"); 99 | *p = '\0'; 100 | break; 101 | 102 | case ';': 103 | p = discard_line(ini, p); 104 | break; 105 | 106 | default: 107 | line_start = p; 108 | p += strcspn(p, "=\n"); 109 | 110 | /* Is line missing a '='? */ 111 | if (*p != '=') { 112 | p = discard_line(ini, line_start); 113 | break; 114 | } 115 | trim_back(ini, p - 1); 116 | 117 | /* Replace '=' and whitespace after it with '\0' */ 118 | do { 119 | *p++ = '\0'; 120 | } while (*p == ' ' || *p == '\r' || *p == '\t'); 121 | 122 | /* Is a value after '=' missing? */ 123 | if (*p == '\n' || *p == '\0') { 124 | p = discard_line(ini, line_start); 125 | break; 126 | } 127 | 128 | if (*p == '"') { 129 | /* Handle quoted string value */ 130 | value_start = p; 131 | p = unescape_quoted_value(ini, p); 132 | 133 | /* Was the string empty? */ 134 | if (p == value_start) { 135 | p = discard_line(ini, line_start); 136 | break; 137 | } 138 | 139 | /* Discard the rest of the line after the string value */ 140 | p = discard_line(ini, p); 141 | 142 | } else { 143 | /* Handle normal value */ 144 | p += strcspn(p, "\n"); 145 | trim_back(ini, p - 1); 146 | } 147 | break; 148 | } 149 | } 150 | } 151 | 152 | ini_t* ini_load(const char *filename) { 153 | ini_t *ini = NULL; 154 | FILE *fp = NULL; 155 | int n, sz; 156 | 157 | /* Init ini struct */ 158 | ini = (ini_t*) malloc(sizeof(*ini)); 159 | if (!ini) { 160 | goto fail; 161 | } 162 | memset(ini, 0, sizeof(*ini)); 163 | 164 | /* Open file */ 165 | fp = fopen(filename, "rb"); 166 | if (!fp) { 167 | goto fail; 168 | } 169 | 170 | /* Get file size */ 171 | fseek(fp, 0, SEEK_END); 172 | sz = ftell(fp); 173 | if(sz < 0) { 174 | goto fail; 175 | } 176 | rewind(fp); 177 | 178 | /* Load file content into memory, null terminate, init end var */ 179 | ini->data = (char*) malloc(sz + 1); 180 | ini->data[sz] = '\0'; 181 | ini->end = ini->data + sz; 182 | n = fread(ini->data, 1, sz, fp); 183 | if (n != sz) { 184 | goto fail; 185 | } 186 | 187 | /* Prepare data */ 188 | split_data(ini); 189 | 190 | /* Clean up and return */ 191 | fclose(fp); 192 | return ini; 193 | 194 | fail: 195 | if (fp) fclose(fp); 196 | if (ini) ini_free(ini); 197 | return NULL; 198 | } 199 | 200 | void ini_free(ini_t *ini) { 201 | free(ini->data); 202 | free(ini); 203 | } 204 | 205 | const char* ini_get(ini_t *ini, const char *section, const char *key) { 206 | const char *current_section = ""; 207 | char *val; 208 | char *p = ini->data; 209 | 210 | if (*p == '\0') { 211 | p = next(ini, p); 212 | } 213 | 214 | while (p < ini->end) { 215 | if (*p == '[') { 216 | /* Handle section */ 217 | current_section = p + 1; 218 | 219 | } else { 220 | /* Handle key */ 221 | val = next(ini, p); 222 | if (!section || !strcmpci(section, current_section)) { 223 | if (!strcmpci(p, key)) { 224 | return val; 225 | } 226 | } 227 | p = val; 228 | } 229 | 230 | p = next(ini, p); 231 | } 232 | 233 | return NULL; 234 | } 235 | 236 | int ini_sget( 237 | ini_t *ini, const char *section, const char *value, 238 | const char *scanfmt, void *dst 239 | ) { 240 | const char *val = ini_get(ini, section, value); 241 | if (!val) { 242 | return 0; 243 | } 244 | if (scanfmt) { 245 | sscanf(val, scanfmt, dst); 246 | } else { 247 | *((const char**) dst) = val; 248 | } 249 | return 1; 250 | } 251 | -------------------------------------------------------------------------------- /dbsync/ini.h: -------------------------------------------------------------------------------- 1 | /** 2 | * Copyright (c) 2015 rxi 3 | * 4 | * This library is free software; you can redistribute it and/or modify it 5 | * under the terms of the MIT license. See LICENSE for details. 6 | */ 7 | 8 | #ifndef INI_H 9 | #define INI_H 10 | 11 | #define INI_VERSION "0.1.1" 12 | 13 | #ifdef __cplusplus 14 | extern "C" { 15 | #endif 16 | 17 | typedef struct ini_t ini_t; 18 | 19 | extern ini_t* ini_load(const char *filename); 20 | extern void ini_free(ini_t *ini); 21 | extern const char* ini_get(ini_t *ini, const char *section, const char *key); 22 | extern int ini_sget(ini_t *ini, const char *section, const char *value, 23 | const char *scanfmt, void *dst); 24 | extern void *init_config(char *cfgpath); 25 | extern int get_config(void *cfg, char *sec, char* key, char **value); 26 | 27 | #ifdef __cplusplus 28 | } /* extern "C" */ 29 | #endif 30 | 31 | #endif 32 | -------------------------------------------------------------------------------- /dbsync/misc.c: -------------------------------------------------------------------------------- 1 | 2 | #include "postgres_fe.h" 3 | #include "lib/stringinfo.h" 4 | #include "common/fe_memutils.h" 5 | 6 | #include "libpq-fe.h" 7 | 8 | #include "access/transam.h" 9 | #include "libpq/pqformat.h" 10 | #include "pqexpbuffer.h" 11 | 12 | #include "misc.h" 13 | 14 | #include 15 | 16 | 17 | #ifndef WIN32 18 | #include 19 | #include 20 | #endif 21 | 22 | bool 23 | WaitThreadEnd(int n, Thread *th) 24 | { 25 | ThreadHandle *hanlde = NULL; 26 | int i; 27 | 28 | hanlde = (ThreadHandle *)malloc(sizeof(ThreadHandle) * n); 29 | for(i = 0; i < n; i++) 30 | { 31 | hanlde[i]=th[i].os_handle; 32 | } 33 | 34 | #ifdef WIN32 35 | WaitForMultipleObjects(n, hanlde, TRUE, INFINITE); 36 | #else 37 | for(i = 0; i < n; i++) 38 | pthread_join(hanlde[i], NULL); 39 | #endif 40 | 41 | free(hanlde); 42 | 43 | return true; 44 | } 45 | 46 | int 47 | ThreadCreate(Thread *th, 48 | void *(*start)(void *arg), 49 | void *arg) 50 | { 51 | int rc = -1; 52 | #ifdef WIN32 53 | th->os_handle = (HANDLE)_beginthreadex(NULL, 54 | 0, 55 | (unsigned(__stdcall*)(void*)) start, 56 | arg, 57 | 0, 58 | &th->thid); 59 | 60 | /* error for returned value 0 */ 61 | if (th->os_handle == (HANDLE) 0) 62 | th->os_handle = INVALID_HANDLE_VALUE; 63 | else 64 | rc = 1; 65 | #else 66 | rc = pthread_create(&th->os_handle, 67 | NULL, 68 | start, 69 | arg); 70 | #endif 71 | return rc; 72 | } 73 | 74 | void 75 | ThreadExit(int code) 76 | { 77 | #ifdef WIN32 78 | _endthreadex((unsigned) code); 79 | #else 80 | pthread_exit((void *)NULL); 81 | return; 82 | #endif 83 | } 84 | 85 | 86 | PGconn * 87 | pglogical_connect(const char *connstring, const char *connname) 88 | { 89 | PGconn *conn; 90 | StringInfoData dsn; 91 | 92 | initStringInfo(&dsn); 93 | appendStringInfo(&dsn, 94 | "%s fallback_application_name='%s'", 95 | connstring, connname); 96 | 97 | conn = PQconnectdb(dsn.data); 98 | if (PQstatus(conn) != CONNECTION_OK) 99 | { 100 | fprintf(stderr,"could not connect to the postgresql server: %s dsn was: %s", 101 | PQerrorMessage(conn), dsn.data); 102 | return NULL; 103 | } 104 | 105 | return conn; 106 | } 107 | 108 | bool 109 | is_greenplum(PGconn *conn) 110 | { 111 | char *query = "select version from version()"; 112 | bool is_greenplum = false; 113 | char *result; 114 | PGresult *res; 115 | 116 | res = PQexec(conn, query); 117 | if (PQresultStatus(res) != PGRES_TUPLES_OK) 118 | { 119 | fprintf(stderr, "init sql run failed: %s", PQresultErrorMessage(res)); 120 | return false; 121 | } 122 | result = PQgetvalue(res, 0, 0); 123 | if (strstr(result, "Greenplum") != NULL) 124 | { 125 | is_greenplum = true; 126 | } 127 | 128 | PQclear(res); 129 | 130 | return is_greenplum; 131 | } 132 | 133 | size_t 134 | quote_literal_internal(char *dst, const char *src, size_t len) 135 | { 136 | const char *s; 137 | char *savedst = dst; 138 | 139 | for (s = src; s < src + len; s++) 140 | { 141 | if (*s == '\\') 142 | { 143 | *dst++ = ESCAPE_STRING_SYNTAX; 144 | break; 145 | } 146 | } 147 | 148 | *dst++ = '\''; 149 | while (len-- > 0) 150 | { 151 | if (SQL_STR_DOUBLE(*src, true)) 152 | *dst++ = *src; 153 | *dst++ = *src++; 154 | } 155 | *dst++ = '\''; 156 | 157 | return dst - savedst; 158 | } 159 | 160 | int 161 | start_copy_origin_tx(PGconn *conn, const char *snapshot, int pg_version, bool is_greenplum) 162 | { 163 | PGresult *res; 164 | const char *setup_query = NULL; 165 | StringInfoData query; 166 | 167 | if (is_greenplum == false) 168 | { 169 | setup_query = "BEGIN TRANSACTION ISOLATION LEVEL REPEATABLE READ, READ ONLY;\n"; 170 | } 171 | else 172 | { 173 | setup_query = "BEGIN"; 174 | } 175 | 176 | initStringInfo(&query); 177 | appendStringInfoString(&query, setup_query); 178 | 179 | if (snapshot) 180 | appendStringInfo(&query, "SET TRANSACTION SNAPSHOT '%s';\n", snapshot); 181 | 182 | res = PQexec(conn, query.data); 183 | if (PQresultStatus(res) != PGRES_COMMAND_OK) 184 | { 185 | fprintf(stderr, "BEGIN on origin node failed: %s", 186 | PQresultErrorMessage(res)); 187 | return 1; 188 | } 189 | 190 | PQclear(res); 191 | 192 | setup_connection(conn, pg_version, is_greenplum); 193 | 194 | return 0; 195 | } 196 | 197 | int 198 | start_copy_target_tx(PGconn *conn, int pg_version, bool is_greenplum) 199 | { 200 | PGresult *res; 201 | const char *setup_query = 202 | "BEGIN TRANSACTION ISOLATION LEVEL READ COMMITTED;\n"; 203 | 204 | res = PQexec(conn, setup_query); 205 | if (PQresultStatus(res) != PGRES_COMMAND_OK) 206 | { 207 | fprintf(stderr, "BEGIN on target node failed: %s", 208 | PQresultErrorMessage(res)); 209 | return 1; 210 | } 211 | 212 | PQclear(res); 213 | 214 | setup_connection(conn, pg_version, is_greenplum); 215 | 216 | return 0; 217 | } 218 | 219 | int 220 | finish_copy_origin_tx(PGconn *conn) 221 | { 222 | PGresult *res; 223 | 224 | /* Close the transaction and connection on origin node. */ 225 | res = PQexec(conn, "ROLLBACK"); 226 | if (PQresultStatus(res) != PGRES_COMMAND_OK) 227 | { 228 | fprintf(stderr, "ROLLBACK on origin node failed: %s", 229 | PQresultErrorMessage(res)); 230 | return 1; 231 | } 232 | 233 | PQclear(res); 234 | //PQfinish(conn); 235 | return 0; 236 | } 237 | 238 | int 239 | finish_copy_target_tx(PGconn *conn) 240 | { 241 | PGresult *res; 242 | 243 | /* Close the transaction and connection on target node. */ 244 | res = PQexec(conn, "COMMIT"); 245 | if (PQresultStatus(res) != PGRES_COMMAND_OK) 246 | { 247 | fprintf(stderr, "COMMIT on target node failed: %s", 248 | PQresultErrorMessage(res)); 249 | return 1; 250 | } 251 | 252 | PQclear(res); 253 | //PQfinish(conn); 254 | return 0; 255 | } 256 | 257 | int 258 | setup_connection(PGconn *conn, int remoteVersion, bool is_greenplum) 259 | { 260 | char *dumpencoding = "utf8"; 261 | 262 | /* 263 | * Set the client encoding if requested. If dumpencoding == NULL then 264 | * either it hasn't been requested or we're a cloned connection and then 265 | * this has already been set in CloneArchive according to the original 266 | * connection encoding. 267 | */ 268 | if (PQsetClientEncoding(conn, dumpencoding) < 0) 269 | { 270 | fprintf(stderr, "invalid client encoding \"%s\" specified\n", 271 | dumpencoding); 272 | return 1; 273 | } 274 | 275 | /* 276 | * Get the active encoding and the standard_conforming_strings setting, so 277 | * we know how to escape strings. 278 | */ 279 | //AH->encoding = PQclientEncoding(conn); 280 | 281 | //std_strings = PQparameterStatus(conn, "standard_conforming_strings"); 282 | //AH->std_strings = (std_strings && strcmp(std_strings, "on") == 0); 283 | 284 | /* Set the datestyle to ISO to ensure the dump's portability */ 285 | ExecuteSqlStatement(conn, "SET DATESTYLE = ISO"); 286 | 287 | /* Likewise, avoid using sql_standard intervalstyle */ 288 | if (remoteVersion >= 80400) 289 | ExecuteSqlStatement(conn, "SET INTERVALSTYLE = POSTGRES"); 290 | 291 | /* 292 | * If supported, set extra_float_digits so that we can dump float data 293 | * exactly (given correctly implemented float I/O code, anyway) 294 | */ 295 | if (remoteVersion >= 90000) 296 | ExecuteSqlStatement(conn, "SET extra_float_digits TO 3"); 297 | else if (remoteVersion >= 70400) 298 | ExecuteSqlStatement(conn, "SET extra_float_digits TO 2"); 299 | 300 | /* 301 | * If synchronized scanning is supported, disable it, to prevent 302 | * unpredictable changes in row ordering across a dump and reload. 303 | */ 304 | if (remoteVersion >= 80300 && !is_greenplum) 305 | ExecuteSqlStatement(conn, "SET synchronize_seqscans TO off"); 306 | 307 | /* 308 | * Disable timeouts if supported. 309 | */ 310 | if (remoteVersion >= 70300) 311 | ExecuteSqlStatement(conn, "SET statement_timeout = 0"); 312 | if (remoteVersion >= 90300) 313 | ExecuteSqlStatement(conn, "SET lock_timeout = 0"); 314 | 315 | return 0; 316 | } 317 | 318 | int 319 | ExecuteSqlStatement(PGconn *conn, const char *query) 320 | { 321 | PGresult *res; 322 | int rc = 0; 323 | 324 | res = PQexec(conn, query); 325 | if (PQresultStatus(res) != PGRES_COMMAND_OK) 326 | { 327 | fprintf(stderr, "set %s failed: %s", 328 | query, PQerrorMessage(conn)); 329 | rc = 1; 330 | } 331 | PQclear(res); 332 | 333 | return rc; 334 | } 335 | 336 | #ifndef FRONTEND 337 | #error "This file is not expected to be compiled for backend code" 338 | #endif 339 | 340 | void * 341 | pg_malloc(size_t size) 342 | { 343 | void *tmp; 344 | 345 | /* Avoid unportable behavior of malloc(0) */ 346 | if (size == 0) 347 | size = 1; 348 | tmp = malloc(size); 349 | if (!tmp) 350 | { 351 | fprintf(stderr, _("out of memory\n")); 352 | exit(EXIT_FAILURE); 353 | } 354 | return tmp; 355 | } 356 | 357 | void * 358 | pg_malloc0(size_t size) 359 | { 360 | void *tmp; 361 | 362 | tmp = pg_malloc(size); 363 | MemSet(tmp, 0, size); 364 | return tmp; 365 | } 366 | 367 | void * 368 | palloc(Size size) 369 | { 370 | return pg_malloc(size); 371 | } 372 | 373 | void * 374 | palloc0(Size size) 375 | { 376 | return pg_malloc0(size); 377 | } 378 | 379 | char * 380 | pstrdup(const char *in) 381 | { 382 | return pg_strdup(in); 383 | } 384 | 385 | void * 386 | repalloc(void *pointer, Size size) 387 | { 388 | return pg_realloc(pointer, size); 389 | } 390 | 391 | void * 392 | pg_realloc(void *ptr, size_t size) 393 | { 394 | void *tmp; 395 | 396 | /* Avoid unportable behavior of realloc(NULL, 0) */ 397 | if (ptr == NULL && size == 0) 398 | size = 1; 399 | tmp = realloc(ptr, size); 400 | if (!tmp) 401 | { 402 | fprintf(stderr, _("out of memory\n")); 403 | exit(EXIT_FAILURE); 404 | } 405 | return tmp; 406 | } 407 | 408 | char * 409 | pg_strdup(const char *in) 410 | { 411 | char *tmp; 412 | 413 | if (!in) 414 | { 415 | fprintf(stderr, 416 | _("cannot duplicate null pointer (internal error)\n")); 417 | exit(EXIT_FAILURE); 418 | } 419 | tmp = strdup(in); 420 | if (!tmp) 421 | { 422 | fprintf(stderr, _("out of memory\n")); 423 | exit(EXIT_FAILURE); 424 | } 425 | return tmp; 426 | } 427 | 428 | void 429 | pfree(void *pointer) 430 | { 431 | pg_free(pointer); 432 | } 433 | 434 | void 435 | pg_free(void *ptr) 436 | { 437 | if (ptr != NULL) 438 | free(ptr); 439 | } 440 | 441 | char * 442 | psprintf(const char *fmt,...) 443 | { 444 | size_t len = 128; /* initial assumption about buffer size */ 445 | 446 | for (;;) 447 | { 448 | char *result; 449 | va_list args; 450 | size_t newlen; 451 | 452 | /* 453 | * Allocate result buffer. Note that in frontend this maps to malloc 454 | * with exit-on-error. 455 | */ 456 | result = (char *) palloc(len); 457 | 458 | /* Try to format the data. */ 459 | va_start(args, fmt); 460 | newlen = pvsnprintf(result, len, fmt, args); 461 | va_end(args); 462 | 463 | if (newlen < len) 464 | return result; /* success */ 465 | 466 | /* Release buffer and loop around to try again with larger len. */ 467 | pfree(result); 468 | len = newlen; 469 | } 470 | } 471 | 472 | /* 473 | * pvsnprintf 474 | * 475 | * Attempt to format text data under the control of fmt (an sprintf-style 476 | * format string) and insert it into buf (which has length len, len > 0). 477 | * 478 | * If successful, return the number of bytes emitted, not counting the 479 | * trailing zero byte. This will always be strictly less than len. 480 | * 481 | * If there's not enough space in buf, return an estimate of the buffer size 482 | * needed to succeed (this *must* be more than the given len, else callers 483 | * might loop infinitely). 484 | * 485 | * Other error cases do not return, but exit via elog(ERROR) or exit(). 486 | * Hence, this shouldn't be used inside libpq. 487 | * 488 | * This function exists mainly to centralize our workarounds for 489 | * non-C99-compliant vsnprintf implementations. Generally, any call that 490 | * pays any attention to the return value should go through here rather 491 | * than calling snprintf or vsnprintf directly. 492 | * 493 | * Note that the semantics of the return value are not exactly C99's. 494 | * First, we don't promise that the estimated buffer size is exactly right; 495 | * callers must be prepared to loop multiple times to get the right size. 496 | * Second, we return the recommended buffer size, not one less than that; 497 | * this lets overflow concerns be handled here rather than in the callers. 498 | */ 499 | size_t 500 | pvsnprintf(char *buf, size_t len, const char *fmt, va_list args) 501 | { 502 | int nprinted; 503 | 504 | Assert(len > 0); 505 | 506 | errno = 0; 507 | 508 | /* 509 | * Assert check here is to catch buggy vsnprintf that overruns the 510 | * specified buffer length. Solaris 7 in 64-bit mode is an example of a 511 | * platform with such a bug. 512 | */ 513 | #ifdef USE_ASSERT_CHECKING 514 | buf[len - 1] = '\0'; 515 | #endif 516 | 517 | nprinted = vsnprintf(buf, len, fmt, args); 518 | 519 | Assert(buf[len - 1] == '\0'); 520 | 521 | /* 522 | * If vsnprintf reports an error other than ENOMEM, fail. The possible 523 | * causes of this are not user-facing errors, so elog should be enough. 524 | */ 525 | if (nprinted < 0 && errno != 0 && errno != ENOMEM) 526 | { 527 | #ifndef FRONTEND 528 | elog(ERROR, "vsnprintf failed: %m"); 529 | #else 530 | fprintf(stderr, "vsnprintf failed: %s\n", strerror(errno)); 531 | exit(EXIT_FAILURE); 532 | #endif 533 | } 534 | 535 | /* 536 | * Note: some versions of vsnprintf return the number of chars actually 537 | * stored, not the total space needed as C99 specifies. And at least one 538 | * returns -1 on failure. Be conservative about believing whether the 539 | * print worked. 540 | */ 541 | if (nprinted >= 0 && (size_t) nprinted < len - 1) 542 | { 543 | /* Success. Note nprinted does not include trailing null. */ 544 | return (size_t) nprinted; 545 | } 546 | 547 | if (nprinted >= 0 && (size_t) nprinted > len) 548 | { 549 | /* 550 | * This appears to be a C99-compliant vsnprintf, so believe its 551 | * estimate of the required space. (If it's wrong, the logic will 552 | * still work, but we may loop multiple times.) Note that the space 553 | * needed should be only nprinted+1 bytes, but we'd better allocate 554 | * one more than that so that the test above will succeed next time. 555 | * 556 | * In the corner case where the required space just barely overflows, 557 | * fall through so that we'll error out below (possibly after 558 | * looping). 559 | */ 560 | if ((size_t) nprinted <= MaxAllocSize - 2) 561 | return nprinted + 2; 562 | } 563 | 564 | /* 565 | * Buffer overrun, and we don't know how much space is needed. Estimate 566 | * twice the previous buffer size, but not more than MaxAllocSize; if we 567 | * are already at MaxAllocSize, choke. Note we use this palloc-oriented 568 | * overflow limit even when in frontend. 569 | */ 570 | if (len >= MaxAllocSize) 571 | { 572 | #ifndef FRONTEND 573 | ereport(ERROR, 574 | (errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED), 575 | errmsg("out of memory"))); 576 | #else 577 | fprintf(stderr, _("out of memory\n")); 578 | exit(EXIT_FAILURE); 579 | #endif 580 | } 581 | 582 | if (len >= MaxAllocSize / 2) 583 | return MaxAllocSize; 584 | 585 | return len * 2; 586 | } 587 | 588 | 589 | -------------------------------------------------------------------------------- /dbsync/misc.h: -------------------------------------------------------------------------------- 1 | 2 | 3 | #ifndef PG_MISC_H 4 | #define PG_MISC_H 5 | 6 | #include "postgres_fe.h" 7 | 8 | #include "lib/stringinfo.h" 9 | #include "lib/stringinfo.h" 10 | #include "common/fe_memutils.h" 11 | 12 | #include "libpq-fe.h" 13 | 14 | #include "access/transam.h" 15 | #include "libpq/pqformat.h" 16 | #include "pqexpbuffer.h" 17 | 18 | #ifdef WIN32 19 | typedef CRITICAL_SECTION pthread_mutex_t; 20 | typedef HANDLE ThreadHandle; 21 | typedef DWORD ThreadId; 22 | typedef unsigned thid_t; 23 | 24 | typedef struct Thread 25 | { 26 | ThreadHandle os_handle; 27 | thid_t thid; 28 | }Thread; 29 | 30 | typedef CRITICAL_SECTION pthread_mutex_t; 31 | typedef DWORD pthread_t; 32 | #define pthread_mutex_lock(A) (EnterCriticalSection(A),0) 33 | #define pthread_mutex_trylock(A) win_pthread_mutex_trylock((A)) 34 | #define pthread_mutex_unlock(A) (LeaveCriticalSection(A), 0) 35 | #define pthread_mutex_init(A,B) (InitializeCriticalSection(A),0) 36 | #define pthread_mutex_lock(A) (EnterCriticalSection(A),0) 37 | #define pthread_mutex_trylock(A) win_pthread_mutex_trylock((A)) 38 | #define pthread_mutex_unlock(A) (LeaveCriticalSection(A), 0) 39 | #define pthread_mutex_destroy(A) (DeleteCriticalSection(A), 0) 40 | 41 | 42 | #else 43 | typedef pthread_t ThreadHandle; 44 | typedef pthread_t ThreadId; 45 | typedef pthread_t thid_t; 46 | 47 | typedef struct Thread 48 | { 49 | pthread_t os_handle; 50 | } Thread; 51 | 52 | #define SIGALRM 14 53 | #endif 54 | 55 | #define MaxAllocSize ((Size) 0x3fffffff) /* 1 gigabyte - 1 */ 56 | 57 | extern bool WaitThreadEnd(int n, Thread *th); 58 | extern void ThreadExit(int code); 59 | extern int ThreadCreate(Thread *th, void *(*start)(void *arg), void *arg); 60 | 61 | extern PGconn *pglogical_connect(const char *connstring, const char *connname); 62 | extern bool is_greenplum(PGconn *conn); 63 | extern size_t quote_literal_internal(char *dst, const char *src, size_t len); 64 | extern int start_copy_origin_tx(PGconn *conn, const char *snapshot, int pg_version, bool is_greenplum); 65 | extern int finish_copy_origin_tx(PGconn *conn); 66 | extern int start_copy_target_tx(PGconn *conn, int pg_version, bool is_greenplum); 67 | extern int finish_copy_target_tx(PGconn *conn); 68 | extern int ExecuteSqlStatement(PGconn *conn, const char *query); 69 | extern int setup_connection(PGconn *conn, int remoteVersion, bool is_greenplum); 70 | 71 | #endif 72 | 73 | 74 | -------------------------------------------------------------------------------- /dbsync/my.cfg: -------------------------------------------------------------------------------- 1 | [src.mysql] 2 | host = "192.168.1.2" 3 | port = "3306" 4 | user = "test" 5 | password = "123456" 6 | db = "test" 7 | encodingdir = "share" 8 | encoding = "utf8" 9 | binlogfile = "mysql-bin.000001" 10 | binlogfile_offset = "0" 11 | 12 | [src.pgsql] 13 | connect_string = "host=192.168.1.1 dbname=test port=5432 user=gptest password=123456" 14 | 15 | [local.pgsql] 16 | connect_string = "host=192.168.1.1 dbname=test port=5433 user=gptest password=123456" 17 | 18 | [desc.pgsql] 19 | connect_string = "host=192.168.1.1 dbname=test port=5434 user=gptest password=123456" 20 | ignore_copy_error_count_each_table = "0" 21 | target_schema = "public" 22 | 23 | [binlogloader] 24 | loader_table_list = "loader_table_list.txt" 25 | load_batch = "10" 26 | load_batch_gap = "10" 27 | -------------------------------------------------------------------------------- /dbsync/mysql2pgsql.c: -------------------------------------------------------------------------------- 1 | /* 2 | * mysql2pgsql.c 3 | * 4 | */ 5 | 6 | #include "postgres_fe.h" 7 | #include "lib/stringinfo.h" 8 | #include "common/fe_memutils.h" 9 | 10 | #include "libpq-fe.h" 11 | 12 | #include "access/transam.h" 13 | #include "libpq/pqformat.h" 14 | #include "pqexpbuffer.h" 15 | #include "pgsync.h" 16 | #include "libpq/pqsignal.h" 17 | #include "catalog/pg_type.h" 18 | 19 | #include 20 | 21 | #ifndef WIN32 22 | #include 23 | #endif 24 | 25 | #include "mysql.h" 26 | #include "utils.h" 27 | #include 28 | 29 | static volatile bool time_to_abort = false; 30 | bool get_ddl_only = false; 31 | bool simple_wo_part = false; 32 | bool first_col_as_dist_key = false; 33 | /* buffer for sending copy data to target, unit is Byte */ 34 | int buffer_size = 0; 35 | 36 | #define STMT_SHOW_TABLES "show full tables in `%s` where table_type='BASE TABLE'" 37 | 38 | #define STMT_SELECT "select * from `%s`.`%s`" 39 | 40 | static MYSQL *connect_to_mysql(mysql_conn_info* hd); 41 | static void *mysql2pgsql_copy_data(void *arg); 42 | static void quote_literal_local_withoid(StringInfo s, const char *rawstr, Oid type, PQExpBuffer buffer); 43 | static int setup_connection_from_mysql(PGconn *conn); 44 | static void sigint_handler(int signum); 45 | 46 | #ifndef WIN32 47 | static void 48 | sigint_handler(int signum) 49 | { 50 | time_to_abort = true; 51 | } 52 | #endif 53 | 54 | static Oid * 55 | fetch_colmum_info(char *schemaname, char *tabname, MYSQL_RES *my_res, bool is_target_gp) 56 | { 57 | MYSQL_FIELD *field; 58 | int col_num = 0; 59 | Oid *col_type = NULL; 60 | int i = 0; 61 | PQExpBuffer ddl; 62 | /* Mysql column name len should be no more than 64 */ 63 | char first_col_name[256] = {0}; 64 | 65 | ddl = createPQExpBuffer(); 66 | 67 | if (!get_ddl_only) 68 | { 69 | appendPQExpBufferStr(ddl, "-- Reference DDL to create the target table:\n"); 70 | } 71 | 72 | appendPQExpBuffer(ddl, "CREATE TABLE %s%s%s%s (", 73 | is_target_gp ? "" : "IF NOT EXISTS ", schemaname ? schemaname : "", schemaname ? "." : "", tabname); 74 | 75 | col_num = mysql_num_fields(my_res); 76 | col_type = palloc0(sizeof(Oid) * col_num); 77 | for (i = 0; i < col_num; i++) 78 | { 79 | int type; 80 | 81 | if (i == 0) 82 | { 83 | } 84 | else 85 | { 86 | appendPQExpBufferStr(ddl, ", "); 87 | } 88 | 89 | field = mysql_fetch_field(my_res); 90 | 91 | if (i == 0) 92 | { 93 | sprintf(first_col_name, "%s", field->org_name); 94 | } 95 | 96 | type = field->type; 97 | switch(type) 98 | { 99 | case MYSQL_TYPE_VARCHAR: 100 | case MYSQL_TYPE_VAR_STRING: 101 | case MYSQL_TYPE_STRING: 102 | case MYSQL_TYPE_BIT: 103 | case MYSQL_TYPE_BLOB: 104 | case MYSQL_TYPE_LONG_BLOB: 105 | appendPQExpBuffer(ddl, "%s %s", field->org_name, "text"); 106 | col_type[i] = TEXTOID; 107 | break; 108 | 109 | case MYSQL_TYPE_TIMESTAMP: 110 | appendPQExpBuffer(ddl, "%s %s", field->org_name, "timestamptz"); 111 | col_type[i] = TIMESTAMPTZOID; 112 | break; 113 | case MYSQL_TYPE_DATE: 114 | case MYSQL_TYPE_TIME: 115 | case MYSQL_TYPE_DATETIME: 116 | case MYSQL_TYPE_YEAR: 117 | case MYSQL_TYPE_NEWDATE: 118 | appendPQExpBuffer(ddl, "%s %s", field->org_name, "timestamp"); 119 | col_type[i] = TIMESTAMPOID; 120 | break; 121 | 122 | case MYSQL_TYPE_SHORT: 123 | appendPQExpBuffer(ddl, "%s %s", field->org_name, "int2"); 124 | col_type[i] = INT2OID; 125 | break; 126 | 127 | case MYSQL_TYPE_TINY: 128 | appendPQExpBuffer(ddl, "%s %s", field->org_name, "int4"); 129 | col_type[i] = INT4OID; 130 | break; 131 | 132 | case MYSQL_TYPE_LONG: 133 | appendPQExpBuffer(ddl, "%s %s", field->org_name, "int4"); 134 | col_type[i] = INT4OID; 135 | break; 136 | 137 | case MYSQL_TYPE_LONGLONG: 138 | appendPQExpBuffer(ddl, "%s %s", field->org_name, "int8"); 139 | col_type[i] = INT8OID; 140 | break; 141 | 142 | case MYSQL_TYPE_FLOAT: 143 | appendPQExpBuffer(ddl, "%s %s", field->org_name, "float4"); 144 | col_type[i] = FLOAT4OID; 145 | break; 146 | 147 | case MYSQL_TYPE_DOUBLE: 148 | appendPQExpBuffer(ddl, "%s %s", field->org_name, "float8"); 149 | col_type[i] = FLOAT8OID; 150 | break; 151 | 152 | case MYSQL_TYPE_DECIMAL: 153 | case MYSQL_TYPE_NEWDECIMAL: 154 | appendPQExpBuffer(ddl, "%s %s", field->org_name, "numeric"); 155 | col_type[i] = NUMERICOID; 156 | break; 157 | case MYSQL_TYPE_INT24: 158 | appendPQExpBuffer(ddl, "%s %s", field->org_name, "int4"); 159 | col_type[i] = INT4OID; 160 | break; 161 | 162 | default: 163 | fprintf(stderr, "unsupported col %s type %d\n", field->org_name, type); 164 | return NULL; 165 | } 166 | } 167 | 168 | if (is_target_gp) 169 | { 170 | appendPQExpBuffer(ddl, ") with (APPENDONLY=true, ORIENTATION=column, COMPRESSTYPE=zlib, COMPRESSLEVEL=1, BLOCKSIZE=1048576, OIDS=false) DISTRIBUTED BY (%s)", first_col_as_dist_key ? first_col_name : ""); 171 | 172 | if (!simple_wo_part) 173 | appendPQExpBuffer(ddl," PARTITION BY RANGE () (START (date '') INCLUSIVE END (date '') EXCLUSIVE EVERY (INTERVAL '<1 month>' ))"); 174 | } 175 | else 176 | { 177 | appendPQExpBuffer(ddl, ")"); 178 | } 179 | 180 | fprintf(stderr, "%s;\n\n", ddl->data); 181 | 182 | destroyPQExpBuffer(ddl); 183 | 184 | return col_type; 185 | } 186 | 187 | 188 | static MYSQL * 189 | connect_to_mysql(mysql_conn_info* hd) 190 | { 191 | int ret = -1; 192 | bool m_reConn = true; 193 | 194 | //my_init(); 195 | //mysql_thread_init(); 196 | MYSQL *m_mysqlConnection = mysql_init(NULL); 197 | if (hd->encoding != NULL) 198 | { 199 | ret = mysql_options(m_mysqlConnection, MYSQL_SET_CHARSET_NAME, hd->encoding); 200 | if (ret != 0) 201 | { 202 | fprintf(stderr, "set CHARSET_NAME to %s error: %s\n", hd->encoding, mysql_error(m_mysqlConnection)); 203 | return NULL; 204 | } 205 | } 206 | if (hd->encodingdir != NULL) 207 | { 208 | ret = mysql_options(m_mysqlConnection, MYSQL_SET_CHARSET_DIR, hd->encodingdir); 209 | if (ret != 0) 210 | { 211 | fprintf(stderr, "set CHARSET_DIR to %s error: %s\n", hd->encodingdir, mysql_error(m_mysqlConnection)); 212 | return NULL; 213 | } 214 | } 215 | 216 | if (1) 217 | { 218 | int opt_local_infile = 1; 219 | mysql_options(m_mysqlConnection, MYSQL_OPT_LOCAL_INFILE, (char*) &opt_local_infile); 220 | } 221 | 222 | // printf("set reconnect %s", m_reConn ? "true" : "false"); 223 | ret = mysql_options(m_mysqlConnection, MYSQL_OPT_RECONNECT, &m_reConn); 224 | if (ret != 0) 225 | { 226 | fprintf(stderr, "set OPT_RECONNECT error: %s\n", mysql_error(m_mysqlConnection)); 227 | return NULL; 228 | } 229 | 230 | if (! mysql_real_connect(m_mysqlConnection, 231 | hd->host, 232 | hd->user, 233 | hd->passwd, 234 | hd->db, 235 | hd->port, 236 | NULL, 237 | CLIENT_MULTI_STATEMENTS|CLIENT_MULTI_RESULTS)) 238 | { 239 | fprintf(stderr, "connect error: %s\n", mysql_error(m_mysqlConnection)); 240 | 241 | return NULL; 242 | } 243 | 244 | ret = mysql_query(m_mysqlConnection, "set unique_checks = 0;"); 245 | if (ret != 0) 246 | { 247 | fprintf(stderr, "set unique_checks = 0 error: %s\n", mysql_error(m_mysqlConnection)); 248 | return NULL; 249 | } 250 | 251 | // hd->conn_hd = m_mysqlConnection; 252 | 253 | return m_mysqlConnection; 254 | } 255 | 256 | /* 257 | * Entry point for mysql2pgsql 258 | */ 259 | int 260 | mysql2pgsql_sync_main(char *desc, int nthread, mysql_conn_info *hd, char* target_schema, uint32 ignore_error_count) 261 | { 262 | int i = 0; 263 | Thread_hd th_hd; 264 | Thread *thread = NULL; 265 | PGresult *res = NULL; 266 | PGconn *desc_conn; 267 | long s_count = 0; 268 | long t_count = 0; 269 | bool have_err = false; 270 | TimevalStruct before, 271 | after; 272 | double elapsed_msec = 0; 273 | int ntask = 0; 274 | MYSQL *conn_src = NULL; 275 | MYSQL_RES *my_res = NULL; 276 | char **p = NULL; 277 | 278 | #ifndef WIN32 279 | signal(SIGINT, sigint_handler); 280 | #endif 281 | 282 | GETTIMEOFDAY(&before); 283 | 284 | memset(&th_hd, 0, sizeof(Thread_hd)); 285 | th_hd.nth = nthread; 286 | th_hd.desc = desc; 287 | th_hd.mysql_src = hd; 288 | th_hd.ignore_error_count = ignore_error_count; 289 | 290 | conn_src = connect_to_mysql(hd); 291 | if (conn_src == NULL) 292 | { 293 | fprintf(stderr, "init src conn failed.\n"); 294 | return 1; 295 | } 296 | 297 | desc_conn = pglogical_connect(desc, EXTENSION_NAME "_main"); 298 | if (desc_conn == NULL) 299 | { 300 | fprintf(stderr, "init desc conn failed: %s\n", PQerrorMessage(desc_conn)); 301 | return 1; 302 | } 303 | th_hd.desc_version = PQserverVersion(desc_conn); 304 | th_hd.desc_is_greenplum = is_greenplum(desc_conn); 305 | PQfinish(desc_conn); 306 | 307 | if (hd->tabnames == NULL) 308 | { 309 | PQExpBuffer query; 310 | MYSQL_ROW row; 311 | int ret = -1; 312 | 313 | query = createPQExpBuffer(); 314 | 315 | appendPQExpBuffer(query, STMT_SHOW_TABLES, hd->db); 316 | ret = mysql_real_query(conn_src, query->data, strlen(query->data)); 317 | if (ret != 0) 318 | { 319 | fprintf(stderr, "init desc conn failed: %s", mysql_error(conn_src)); 320 | return 1; 321 | } 322 | 323 | my_res = mysql_store_result(conn_src); 324 | if (my_res == NULL) 325 | { 326 | fprintf(stderr, "get src db table failed: %s", mysql_error(conn_src)); 327 | return 1; 328 | } 329 | 330 | ntask = mysql_num_rows(my_res); 331 | th_hd.ntask = ntask; 332 | if (th_hd.ntask >= 1) 333 | { 334 | th_hd.task = (Task_hd *)palloc0(sizeof(Task_hd) * th_hd.ntask); 335 | } 336 | 337 | /* 338 | * The linked-array th_hd.task serves as a task queue for all the worker threads to pick up tasks and consume 339 | */ 340 | for (i = 0; i < th_hd.ntask; i++) 341 | { 342 | row = mysql_fetch_row(my_res); 343 | th_hd.task[i].id = i; 344 | th_hd.task[i].schemaname = target_schema; 345 | th_hd.task[i].relname = pstrdup(row[0]); 346 | th_hd.task[i].query = NULL; 347 | th_hd.task[i].count = 0; 348 | th_hd.task[i].complete = false; 349 | 350 | /* Set the former entry's link to this entry. Last entry's next feild would remain NULL */ 351 | if (i != 0) 352 | { 353 | th_hd.task[i-1].next = &th_hd.task[i]; 354 | } 355 | } 356 | mysql_free_result(my_res); 357 | PQclear(res); 358 | destroyPQExpBuffer(query); 359 | } 360 | else 361 | { 362 | for (i = 0, p = hd->tabnames; *p != NULL; p++, i++) 363 | { 364 | } 365 | 366 | ntask = i; 367 | th_hd.ntask = ntask; 368 | th_hd.task = (Task_hd *)palloc0(sizeof(Task_hd) * th_hd.ntask); 369 | 370 | for (i = 0, p = hd->tabnames; *p != NULL; p++, i++) 371 | { 372 | th_hd.task[i].id = i; 373 | th_hd.task[i].schemaname = target_schema; 374 | th_hd.task[i].relname = *p; 375 | th_hd.task[i].query = hd->queries[i]; 376 | 377 | th_hd.task[i].count = 0; 378 | th_hd.task[i].complete = false; 379 | /* Set the former entry's link to this entry. Last entry's next feild would remain NULL */ 380 | if (i != 0) 381 | { 382 | th_hd.task[i-1].next = &th_hd.task[i]; 383 | } 384 | } 385 | } 386 | 387 | 388 | th_hd.l_task = &(th_hd.task[0]); 389 | 390 | th_hd.th = (ThreadArg *)palloc0(sizeof(ThreadArg) * th_hd.nth); 391 | for (i = 0; i < th_hd.nth; i++) 392 | { 393 | th_hd.th[i].id = i; 394 | th_hd.th[i].count = 0; 395 | th_hd.th[i].all_ok = false; 396 | 397 | th_hd.th[i].hd = &th_hd; 398 | } 399 | pthread_mutex_init(&th_hd.t_lock, NULL); 400 | 401 | if (!get_ddl_only) 402 | { 403 | fprintf(stderr, "Starting data sync\n"); 404 | } 405 | 406 | thread = (Thread *)palloc0(sizeof(Thread) * th_hd.nth); 407 | for (i = 0; i < th_hd.nth; i++) 408 | { 409 | ThreadCreate(&thread[i], mysql2pgsql_copy_data, &th_hd.th[i]); 410 | } 411 | 412 | WaitThreadEnd(th_hd.nth, thread); 413 | 414 | GETTIMEOFDAY(&after); 415 | DIFF_MSEC(&after, &before, elapsed_msec); 416 | 417 | for (i = 0; i < th_hd.nth; i++) 418 | { 419 | if(th_hd.th[i].all_ok) 420 | { 421 | s_count += th_hd.th[i].count; 422 | } 423 | else 424 | { 425 | have_err = true; 426 | } 427 | } 428 | 429 | for (i = 0; i < ntask; i++) 430 | { 431 | t_count += th_hd.task[i].count; 432 | } 433 | 434 | if (!get_ddl_only) 435 | { 436 | fprintf(stderr, "Number of rows migrated: %ld (number of source tables' rows: %ld) \n", s_count, t_count); 437 | fprintf(stderr, "Data sync time cost %.3f ms\n", elapsed_msec); 438 | } 439 | else 440 | { 441 | fprintf(stderr, "-- Number of tables: %d \n", ntask); 442 | } 443 | 444 | if (have_err) 445 | { 446 | fprintf(stderr, "errors occured during migration\n"); 447 | } 448 | 449 | return 0; 450 | } 451 | 452 | static void * 453 | mysql2pgsql_copy_data(void *arg) 454 | { 455 | ThreadArg *args = (ThreadArg *)arg; 456 | Thread_hd *hd = args->hd; 457 | MYSQL_RES *my_res; 458 | PGresult *res2; 459 | PQExpBuffer query; 460 | char *nspname; 461 | char *relname; 462 | Task_hd *curr = NULL; 463 | TimevalStruct before, 464 | after; 465 | double elapsed_msec = 0; 466 | MYSQL *origin_conn = NULL; 467 | PGconn *target_conn = NULL; 468 | int ret = -1; 469 | MYSQL_ROW row; 470 | Oid *column_oids = NULL; 471 | StringInfoData s_tmp; 472 | int i = 0; 473 | int target_version; 474 | bool isgp = false; 475 | 476 | initStringInfo(&s_tmp); 477 | origin_conn = connect_to_mysql(hd->mysql_src); 478 | if (origin_conn == NULL) 479 | { 480 | fprintf(stderr, "init src conn failed"); 481 | return NULL; 482 | } 483 | 484 | target_conn = pglogical_connect(hd->desc, EXTENSION_NAME "_copy"); 485 | if (target_conn == NULL) 486 | { 487 | fprintf(stderr, "init desc conn failed: %s", PQerrorMessage(target_conn)); 488 | return NULL; 489 | } 490 | target_version = PQserverVersion(target_conn); 491 | isgp = is_greenplum(target_conn); 492 | setup_connection(target_conn, target_version, isgp); 493 | setup_connection_from_mysql(target_conn); 494 | 495 | query = createPQExpBuffer(); 496 | 497 | if (get_ddl_only) 498 | fprintf(stderr, "\n-- Reference commands to create target tables %s: \n---------------\n\n", 499 | isgp ? "(Please choose a distribution key and replace it with for each table)" : ""); 500 | while(1) 501 | { 502 | int nlist = 0; 503 | int n_col = 0; 504 | int row_count = 0; 505 | 506 | GETTIMEOFDAY(&before); 507 | pthread_mutex_lock(&hd->t_lock); 508 | nlist = hd->ntask; 509 | if (nlist == 1) 510 | { 511 | curr = hd->l_task; 512 | hd->l_task = NULL; 513 | hd->ntask = 0; 514 | } 515 | else if (nlist > 1) 516 | { 517 | Task_hd *tmp = hd->l_task->next; 518 | curr = hd->l_task; 519 | hd->l_task = tmp; 520 | hd->ntask--; 521 | } 522 | else 523 | { 524 | curr = NULL; 525 | } 526 | 527 | pthread_mutex_unlock(&hd->t_lock); 528 | 529 | if(curr == NULL) 530 | { 531 | break; 532 | } 533 | 534 | if (!get_ddl_only) 535 | start_copy_target_tx(target_conn, hd->desc_version, hd->desc_is_greenplum); 536 | 537 | nspname = hd->mysql_src->db; 538 | relname = curr->relname; 539 | 540 | //fprintf(stderr, "relname %s, query %s \n", relname, curr->query ? curr->query : ""); 541 | 542 | if (curr && curr->query) 543 | { 544 | appendPQExpBufferStr(query, curr->query); 545 | } 546 | else 547 | { 548 | appendPQExpBuffer(query, STMT_SELECT, 549 | nspname, 550 | relname); 551 | } 552 | 553 | if (get_ddl_only) 554 | { 555 | appendPQExpBufferStr(query, " limit 1 "); 556 | } 557 | else 558 | { 559 | fprintf(stderr, "Query to get source data for target table %s: %s \n", relname, query->data); 560 | } 561 | 562 | ret = mysql_query(origin_conn, query->data); 563 | if (ret != 0) 564 | { 565 | fprintf(stderr, "run query error: %s\n", mysql_error(origin_conn)); 566 | goto exit; 567 | } 568 | my_res = mysql_use_result(origin_conn); 569 | column_oids = fetch_colmum_info(curr->schemaname, relname, my_res, isgp); 570 | if (column_oids == NULL) 571 | { 572 | fprintf(stderr, "get table %s column type error\n", relname); 573 | goto exit; 574 | } 575 | 576 | if (get_ddl_only) 577 | { 578 | curr->complete = true; 579 | mysql_free_result(my_res); 580 | resetPQExpBuffer(query); 581 | continue; 582 | } 583 | 584 | n_col = mysql_num_fields(my_res); 585 | 586 | resetPQExpBuffer(query); 587 | appendPQExpBuffer(query, "COPY %s%s%s FROM stdin DELIMITERS '|' with csv QUOTE ''''", 588 | curr->schemaname ? PQescapeIdentifier(target_conn, curr->schemaname, strlen(curr->schemaname)) : "", curr->schemaname ? "." : "", 589 | PQescapeIdentifier(target_conn, relname, 590 | strlen(relname))); 591 | 592 | if (isgp && hd->ignore_error_count > 0) 593 | { 594 | appendPQExpBuffer(query, " SEGMENT REJECT LIMIT %u", 595 | hd->ignore_error_count); 596 | 597 | } 598 | 599 | res2 = PQexec(target_conn, query->data); 600 | if (PQresultStatus(res2) != PGRES_COPY_IN) 601 | { 602 | fprintf(stderr,"table copy failed Query '%s': %s", 603 | query->data, PQerrorMessage(target_conn)); 604 | goto exit; 605 | } 606 | 607 | 608 | resetPQExpBuffer(query); 609 | while ((row = mysql_fetch_row(my_res)) != NULL) 610 | { 611 | unsigned long *lengths; 612 | 613 | lengths = mysql_fetch_lengths(my_res); 614 | for (i = 0; i < n_col; i++) 615 | { 616 | if (i != 0) 617 | { 618 | appendPQExpBufferStr(query, "|"); 619 | } 620 | 621 | /* value of the field is NULL if it is fact NULL */ 622 | if(lengths[i] >= 0 && row[i] != NULL) 623 | { 624 | quote_literal_local_withoid(&s_tmp, row[i], column_oids[i], query); 625 | } 626 | } 627 | 628 | appendPQExpBufferStr(query, "\n"); 629 | row_count++; 630 | 631 | if (query->len >= buffer_size) 632 | { 633 | if (PQputCopyData(target_conn, query->data, query->len) != 1) 634 | { 635 | fprintf(stderr,"writing to target table failed destination connection reported: %s", 636 | PQerrorMessage(target_conn)); 637 | goto exit; 638 | } 639 | 640 | /* Reset buffer for next use */ 641 | resetPQExpBuffer(query); 642 | } 643 | 644 | if (time_to_abort) 645 | { 646 | args->count = row_count; 647 | curr->count = row_count; 648 | fprintf(stderr, "receive shutdown sigint\n"); 649 | goto exit; 650 | } 651 | } 652 | 653 | if (query->len > 0) 654 | { 655 | if (PQputCopyData(target_conn, query->data, query->len) != 1) 656 | { 657 | fprintf(stderr,"writing to target table failed destination connection reported: %s", 658 | PQerrorMessage(target_conn)); 659 | goto exit; 660 | } 661 | 662 | resetPQExpBuffer(query); 663 | } 664 | 665 | args->count = row_count; 666 | curr->count = row_count; 667 | 668 | /* Send local finish */ 669 | if (PQputCopyEnd(target_conn, NULL) != 1) 670 | { 671 | fprintf(stderr,"sending copy-completion to destination connection failed destination connection reported: %s", 672 | PQerrorMessage(target_conn)); 673 | goto exit; 674 | } 675 | 676 | PQclear(res2); 677 | res2 = PQgetResult(target_conn); 678 | if (PQresultStatus(res2) != PGRES_COMMAND_OK) 679 | { 680 | fprintf(stderr, "COPY failed for table \"%s\": %s", 681 | relname, PQerrorMessage(target_conn)); 682 | goto exit; 683 | } 684 | 685 | finish_copy_target_tx(target_conn); 686 | curr->complete = true; 687 | PQclear(res2); 688 | resetPQExpBuffer(query); 689 | mysql_free_result(my_res); 690 | 691 | GETTIMEOFDAY(&after); 692 | DIFF_MSEC(&after, &before, elapsed_msec); 693 | fprintf(stderr,"thread %d migrate task %d table %s.%s %ld rows complete, time cost %.3f ms\n", 694 | args->id, curr->id, nspname, relname, curr->count, elapsed_msec); 695 | } 696 | 697 | args->all_ok = true; 698 | 699 | if (get_ddl_only) 700 | fprintf(stderr, "---------------\n\n"); 701 | 702 | exit: 703 | 704 | mysql_close(origin_conn); 705 | PQfinish(target_conn); 706 | ThreadExit(0); 707 | return NULL; 708 | } 709 | 710 | static void 711 | quote_literal_local_withoid(StringInfo s, const char *rawstr, Oid type, PQExpBuffer buffer) 712 | { 713 | char *result; 714 | int len; 715 | int newlen; 716 | bool need_process = true; 717 | 718 | if (INT2OID == type) 719 | need_process = false; 720 | else if (INT4OID == type) 721 | need_process = false; 722 | else if (INT8OID == type) 723 | need_process = false; 724 | else if (FLOAT4OID == type) 725 | need_process = false; 726 | else if (FLOAT8OID == type) 727 | need_process = false; 728 | else if (NUMERICOID == type) 729 | need_process = false; 730 | else if (TIMESTAMPOID == type) 731 | { 732 | if (strcmp(rawstr, "0000-00-00") == 0 || 733 | strcmp(rawstr, "00:00:00") == 0 || 734 | strcmp(rawstr, "0000-00-00 00:00:00") == 0 || 735 | strcmp(rawstr, "0000") == 0) 736 | { 737 | return; 738 | } 739 | 740 | need_process = false; 741 | } 742 | 743 | if (need_process == false) 744 | { 745 | appendPQExpBuffer(buffer, "%s", rawstr); 746 | return; 747 | } 748 | 749 | /*if (TIMESTAMPOID == type) 750 | need_process = false; 751 | 752 | if (need_process == false) 753 | { 754 | appendPQExpBuffer(buffer, "'"); 755 | appendPQExpBuffer(buffer, "%s", rawstr); 756 | appendPQExpBuffer(buffer, "'"); 757 | return; 758 | }*/ 759 | 760 | len = strlen(rawstr); 761 | resetStringInfo(s); 762 | appendStringInfoSpaces(s, len * 2 + 3); 763 | 764 | result = s->data; 765 | 766 | newlen = quote_literal_internal(result, rawstr, len); 767 | result[newlen] = '\0'; 768 | 769 | appendPQExpBufferStr(buffer, result); 770 | 771 | return; 772 | } 773 | 774 | static int 775 | setup_connection_from_mysql(PGconn *conn) 776 | { 777 | ExecuteSqlStatement(conn, "SET standard_conforming_strings TO off"); 778 | ExecuteSqlStatement(conn, "SET backslash_quote TO on"); 779 | 780 | return 0; 781 | } 782 | 783 | -------------------------------------------------------------------------------- /dbsync/pg_logicaldecode.c: -------------------------------------------------------------------------------- 1 | /* ------------------------------------------------------------------------- 2 | * 3 | * pg_logicaldecode.c 4 | * Replication 5 | * 6 | * Replication 7 | * 8 | * Copyright (C) 2012-2015, Alibaba Group 9 | * 10 | * IDENTIFICATION 11 | * pg_logicaldecode.c 12 | * 13 | * ------------------------------------------------------------------------- 14 | */ 15 | #include "postgres_fe.h" 16 | #include "lib/stringinfo.h" 17 | #include "common/fe_memutils.h" 18 | 19 | #include "libpq-fe.h" 20 | 21 | #include "access/transam.h" 22 | #include "libpq/pqformat.h" 23 | #include "pg_logicaldecode.h" 24 | #include "pqexpbuffer.h" 25 | 26 | #include 27 | 28 | 29 | 30 | static bool process_remote_begin(StringInfo s, ALI_PG_DECODE_MESSAGE *msg); 31 | static bool process_remote_commit(StringInfo s, ALI_PG_DECODE_MESSAGE *msg); 32 | static bool process_remote_insert(StringInfo s, ALI_PG_DECODE_MESSAGE *msg); 33 | static bool process_remote_update(StringInfo s, ALI_PG_DECODE_MESSAGE *msg); 34 | static bool process_remote_delete(StringInfo s, ALI_PG_DECODE_MESSAGE *msg); 35 | static bool read_tuple_parts(StringInfo s, Decode_TupleData *tup); 36 | static bool process_read_colunm_info(StringInfo s, ALI_PG_DECODE_MESSAGE *msg); 37 | 38 | 39 | 40 | /* 41 | * Read a remote action type and process the action record. 42 | * 43 | * May set got_SIGTERM to stop processing before next record. 44 | */ 45 | bool 46 | bdr_process_remote_action(StringInfo s, ALI_PG_DECODE_MESSAGE *msg) 47 | { 48 | char action = pq_getmsgbyte(s); 49 | bool rc = false; 50 | 51 | switch (action) 52 | { 53 | /* BEGIN */ 54 | case 'B': 55 | rc = process_remote_begin(s,msg); 56 | break; 57 | /* COMMIT */ 58 | case 'C': 59 | rc = process_remote_commit(s,msg); 60 | break; 61 | /* INSERT */ 62 | case 'I': 63 | rc = process_remote_insert(s,msg); 64 | break; 65 | /* UPDATE */ 66 | case 'U': 67 | rc = process_remote_update(s,msg); 68 | break; 69 | /* DELETE */ 70 | case 'D': 71 | rc = process_remote_delete(s,msg); 72 | break; 73 | default: 74 | { 75 | fprintf(stderr, "unknown action of type %c", action); 76 | return false; 77 | } 78 | } 79 | 80 | return true; 81 | } 82 | 83 | static bool 84 | process_remote_begin(StringInfo s, ALI_PG_DECODE_MESSAGE *msg) 85 | { 86 | XLogRecPtr origlsn; 87 | TimestampTz committime; 88 | TransactionId remote_xid; 89 | int flags = 0; 90 | 91 | flags = pq_getmsgint(s, 4); 92 | 93 | origlsn = pq_getmsgint64(s); 94 | Assert(origlsn != InvalidXLogRecPtr); 95 | committime = pq_getmsgint64(s); 96 | remote_xid = pq_getmsgint(s, 4); 97 | 98 | msg->type = MSGKIND_BEGIN; 99 | msg->lsn = origlsn; 100 | msg->tm = committime; 101 | msg->xid = remote_xid; 102 | 103 | return true; 104 | } 105 | 106 | /* 107 | * Process a commit message from the output plugin, advance replication 108 | * identifiers, commit the local transaction, and determine whether replay 109 | * should continue. 110 | * 111 | * Returns true if apply should continue with the next record, false if replay 112 | * should stop after this record. 113 | */ 114 | static bool 115 | process_remote_commit(StringInfo s, ALI_PG_DECODE_MESSAGE *msg) 116 | { 117 | XLogRecPtr commit_lsn; 118 | TimestampTz committime; 119 | TimestampTz end_lsn; 120 | int flags; 121 | 122 | flags = pq_getmsgint(s, 4); 123 | 124 | if (flags != 0) 125 | { 126 | fprintf(stderr, "Commit flags are currently unused, but flags was set to %i", flags); 127 | return false; 128 | } 129 | 130 | /* order of access to fields after flags is important */ 131 | commit_lsn = pq_getmsgint64(s); 132 | end_lsn = pq_getmsgint64(s); 133 | committime = pq_getmsgint64(s); 134 | 135 | msg->type = MSGKIND_COMMIT; 136 | msg->lsn = commit_lsn; 137 | msg->tm = committime; 138 | msg->end_lsn = end_lsn; 139 | 140 | return true; 141 | } 142 | 143 | static bool 144 | process_remote_insert(StringInfo s, ALI_PG_DECODE_MESSAGE *msg) 145 | { 146 | char action; 147 | int relnamelen; 148 | int nspnamelen; 149 | char *schemaname; 150 | char *relname; 151 | 152 | msg->type = MSGKIND_INSERT; 153 | 154 | nspnamelen = pq_getmsgint(s, 2); 155 | schemaname = (char *) pq_getmsgbytes(s, nspnamelen); 156 | 157 | relnamelen = pq_getmsgint(s, 2); 158 | relname = (char *) pq_getmsgbytes(s, relnamelen); 159 | 160 | msg->relname = relname; 161 | msg->schemaname = schemaname; 162 | 163 | action = pq_getmsgbyte(s); 164 | if (action != 'N' && action != 'C') 165 | { 166 | fprintf(stderr, "expected new tuple but got %d", 167 | action); 168 | return false; 169 | } 170 | 171 | if (action == 'C') 172 | { 173 | process_read_colunm_info(s, msg); 174 | action = pq_getmsgbyte(s); 175 | } 176 | 177 | if (action != 'N') 178 | { 179 | fprintf(stderr, "expected new tuple but got %d", 180 | action); 181 | return false; 182 | } 183 | 184 | return read_tuple_parts(s, &msg->newtuple); 185 | } 186 | 187 | static bool 188 | process_remote_update(StringInfo s, ALI_PG_DECODE_MESSAGE *msg) 189 | { 190 | char action; 191 | bool pkey_sent; 192 | int relnamelen; 193 | int nspnamelen; 194 | char *schemaname; 195 | char *relname; 196 | 197 | msg->type = MSGKIND_UPDATE; 198 | 199 | nspnamelen = pq_getmsgint(s, 2); 200 | schemaname = (char *) pq_getmsgbytes(s, nspnamelen); 201 | 202 | relnamelen = pq_getmsgint(s, 2); 203 | relname = (char *) pq_getmsgbytes(s, relnamelen); 204 | 205 | msg->relname = relname; 206 | msg->schemaname = schemaname; 207 | 208 | action = pq_getmsgbyte(s); 209 | 210 | /* old key present, identifying key changed */ 211 | if (action != 'K' && action != 'N' && action != 'C') 212 | { 213 | fprintf(stderr, "expected action 'N' or 'K', got %c", 214 | action); 215 | return false; 216 | } 217 | 218 | if (action == 'C') 219 | { 220 | process_read_colunm_info(s, msg); 221 | action = pq_getmsgbyte(s); 222 | } 223 | 224 | if (action != 'K' && action != 'N') 225 | { 226 | fprintf(stderr, "expected action 'N' or 'K', got %c", 227 | action); 228 | return false; 229 | } 230 | 231 | if (action == 'K') 232 | { 233 | pkey_sent = true; 234 | msg->has_key_or_old = true; 235 | read_tuple_parts(s, &msg->oldtuple); 236 | action = pq_getmsgbyte(s); 237 | } 238 | else 239 | pkey_sent = false; 240 | 241 | /* check for new tuple */ 242 | if (action != 'N') 243 | { 244 | fprintf(stderr, "expected action 'N', got %c", 245 | action); 246 | return false; 247 | } 248 | 249 | /* read new tuple */ 250 | return read_tuple_parts(s, &msg->newtuple); 251 | } 252 | 253 | static bool 254 | process_read_colunm_info(StringInfo s, ALI_PG_DECODE_MESSAGE *msg) 255 | { 256 | int natt; 257 | int i; 258 | char action; 259 | 260 | natt = pq_getmsgint(s, 2); 261 | 262 | msg->natt = natt; 263 | for (i = 0; i < natt; i++) 264 | { 265 | char *tmp; 266 | int len; 267 | 268 | len = pq_getmsgint(s, 2); 269 | if(len == 0) 270 | { 271 | continue; 272 | } 273 | 274 | tmp = (char *) pq_getmsgbytes(s, len); 275 | 276 | msg->attname[i] = tmp; 277 | 278 | { 279 | len = pq_getmsgint(s, 2); 280 | tmp = (char *) pq_getmsgbytes(s, len); 281 | msg->atttype[i] = tmp; 282 | } 283 | } 284 | 285 | action = pq_getmsgbyte(s); 286 | 287 | if (action != 'M' && action != 'P') 288 | { 289 | fprintf(stderr, "expected new tuple but got %d", 290 | action); 291 | return false; 292 | } 293 | 294 | if (action == 'P') 295 | { 296 | return true; 297 | } 298 | 299 | natt = pq_getmsgint(s, 2); 300 | 301 | msg->k_natt = natt; 302 | for (i = 0; i < natt; i++) 303 | { 304 | char *tmp; 305 | int len; 306 | 307 | len = pq_getmsgint(s, 2); 308 | tmp = (char *) pq_getmsgbytes(s, len); 309 | 310 | msg->k_attname[i] = tmp; 311 | } 312 | 313 | return true; 314 | } 315 | 316 | 317 | static bool 318 | process_remote_delete(StringInfo s, ALI_PG_DECODE_MESSAGE *msg) 319 | { 320 | char action; 321 | int relnamelen; 322 | int nspnamelen; 323 | char *schemaname; 324 | char *relname; 325 | 326 | msg->type = MSGKIND_DELETE; 327 | 328 | nspnamelen = pq_getmsgint(s, 2); 329 | schemaname = (char *) pq_getmsgbytes(s, nspnamelen); 330 | 331 | relnamelen = pq_getmsgint(s, 2); 332 | relname = (char *) pq_getmsgbytes(s, relnamelen); 333 | 334 | msg->relname = relname; 335 | msg->schemaname = schemaname; 336 | 337 | action = pq_getmsgbyte(s); 338 | 339 | if (action != 'K' && action != 'E' && action != 'C') 340 | { 341 | fprintf(stderr, "expected action K or E got %c", action); 342 | return false; 343 | } 344 | 345 | if (action == 'C') 346 | { 347 | process_read_colunm_info(s, msg); 348 | action = pq_getmsgbyte(s); 349 | } 350 | 351 | if (action != 'K' && action != 'E') 352 | { 353 | fprintf(stderr, "expected action K or E got %c", action); 354 | return false; 355 | } 356 | 357 | if (action == 'E') 358 | { 359 | fprintf(stderr, "got delete without pkey\n"); 360 | return true; 361 | } 362 | 363 | msg->has_key_or_old = true; 364 | return read_tuple_parts(s, &msg->oldtuple); 365 | 366 | } 367 | 368 | static bool 369 | read_tuple_parts(StringInfo s, Decode_TupleData *tup) 370 | { 371 | int i; 372 | int rnatts; 373 | char action; 374 | 375 | action = pq_getmsgbyte(s); 376 | 377 | if (action != 'T') 378 | { 379 | fprintf(stderr, "expected TUPLE, got %c", action); 380 | return false; 381 | } 382 | 383 | memset(tup->isnull, 1, sizeof(tup->isnull)); 384 | memset(tup->changed, 1, sizeof(tup->changed)); 385 | 386 | rnatts = pq_getmsgint(s, 4); 387 | 388 | tup->natt = rnatts; 389 | 390 | /* FIXME: unaligned data accesses */ 391 | for (i = 0; i < rnatts; i++) 392 | { 393 | char kind = pq_getmsgbyte(s); 394 | //const char *data; 395 | int len; 396 | 397 | switch (kind) 398 | { 399 | case 'n': 400 | tup->svalues[i] = NULL; 401 | break; 402 | case 'u': 403 | tup->isnull[i] = true; 404 | tup->changed[i] = false; 405 | tup->svalues[i] = NULL; 406 | 407 | break; 408 | /* 409 | case 'b': 410 | tup->isnull[i] = false; 411 | len = pq_getmsgint(s, 4); 412 | 413 | data = pq_getmsgbytes(s, len); 414 | 415 | tup->svalues[i] = palloc0(len + 1); 416 | memcpy(tup->svalues[i], data, len); 417 | 418 | break; 419 | case 's': 420 | { 421 | StringInfoData buf; 422 | 423 | tup->isnull[i] = false; 424 | len = pq_getmsgint(s, 4); 425 | 426 | initStringInfo(&buf); 427 | buf.data = (char *) pq_getmsgbytes(s, len); 428 | buf.len = len; 429 | 430 | tup->svalues[i] = palloc0(len + 1); 431 | memcpy(tup->svalues[i], buf.data, len); 432 | 433 | break; 434 | } 435 | */ 436 | case 't': 437 | { 438 | tup->isnull[i] = false; 439 | len = pq_getmsgint(s, 4); 440 | 441 | tup->svalues[i] = (char *) pq_getmsgbytes(s, len); 442 | //tup->svalues[i] = palloc0(len + 1); 443 | //memcpy(tup->svalues[i], data, len); 444 | } 445 | break; 446 | default: 447 | { 448 | fprintf(stderr, "unknown column type '%c'", kind); 449 | return false; 450 | } 451 | } 452 | 453 | } 454 | 455 | return true; 456 | } 457 | 458 | -------------------------------------------------------------------------------- /dbsync/pg_logicaldecode.h: -------------------------------------------------------------------------------- 1 | 2 | 3 | #ifndef PG_LOGICALDECODE_H 4 | #define PG_LOGICALDECODE_H 5 | 6 | #include "postgres_fe.h" 7 | 8 | #include "lib/stringinfo.h" 9 | #include "lib/stringinfo.h" 10 | #include "common/fe_memutils.h" 11 | 12 | #include "libpq-fe.h" 13 | 14 | #include "access/transam.h" 15 | #include "libpq/pqformat.h" 16 | #include "pqexpbuffer.h" 17 | #include "misc.h" 18 | 19 | #ifdef __cplusplus 20 | extern "C" 21 | { 22 | #endif 23 | 24 | /* 25 | * Don't include libpq here, msvc infrastructure requires linking to libpq 26 | * otherwise. 27 | */ 28 | struct pg_conn; 29 | 30 | #ifdef HAVE_INT64_TIMESTAMP 31 | typedef int64 timestamp; 32 | typedef int64 TimestampTz; 33 | #else 34 | typedef double timestamp; 35 | typedef double TimestampTz; 36 | #endif 37 | 38 | #define MaxTupleAttributeNumber 1664 /* 8 * 208 */ 39 | 40 | #define MSGKIND_BEGIN 'B' 41 | #define MSGKIND_COMMIT 'C' 42 | #define MSGKIND_INSERT 'I' 43 | #define MSGKIND_UPDATE 'U' 44 | #define MSGKIND_DELETE 'D' 45 | #define MSGKIND_DDL 'L' 46 | #define MSGKIND_UNKNOWN 'K' 47 | 48 | typedef struct Decode_TupleData 49 | { 50 | int natt; 51 | bool isnull[MaxTupleAttributeNumber]; 52 | bool changed[MaxTupleAttributeNumber]; 53 | char *svalues[MaxTupleAttributeNumber]; 54 | } Decode_TupleData; 55 | 56 | typedef struct ALI_PG_DECODE_MESSAGE 57 | { 58 | char type; 59 | 60 | XLogRecPtr lsn; 61 | TimestampTz tm; 62 | 63 | TransactionId xid; /* begin */ 64 | TimestampTz end_lsn; /* commit */ 65 | 66 | char *relname; 67 | char *schemaname; 68 | 69 | int natt; 70 | char *attname[MaxTupleAttributeNumber]; 71 | char *atttype[MaxTupleAttributeNumber]; 72 | 73 | int k_natt; 74 | char *k_attname[MaxTupleAttributeNumber]; 75 | 76 | bool has_key_or_old; 77 | Decode_TupleData newtuple; 78 | Decode_TupleData oldtuple; 79 | 80 | } ALI_PG_DECODE_MESSAGE; 81 | 82 | typedef struct Decoder_handler 83 | { 84 | bool do_create_slot; 85 | bool do_start_slot; 86 | bool do_drop_slot; 87 | 88 | PGconn *conn; 89 | char *connection_string; 90 | 91 | char *progname; 92 | char *replication_slot; 93 | 94 | int outfd; 95 | char *outfile; 96 | 97 | XLogRecPtr startpos; 98 | 99 | XLogRecPtr recvpos; 100 | XLogRecPtr flushpos; 101 | XLogRecPtr last_recvpos; 102 | 103 | ALI_PG_DECODE_MESSAGE msg; 104 | 105 | int verbose; 106 | 107 | char *copybuf; 108 | int standby_message_timeout; 109 | int64 last_status; 110 | 111 | StringInfo buffer; 112 | } Decoder_handler; 113 | 114 | 115 | /* Error level codes */ 116 | #define DEBUG5 10 /* Debugging messages, in categories of 117 | * decreasing detail. */ 118 | #define DEBUG4 11 119 | #define DEBUG3 12 120 | #define DEBUG2 13 121 | #define DEBUG1 14 /* used by GUC debug_* variables */ 122 | #define LOG 15 /* Server operational messages; sent only to 123 | * server log by default. */ 124 | #define COMMERROR 16 /* Client communication problems; same as LOG 125 | * for server reporting, but never sent to 126 | * client. */ 127 | #define INFO 17 /* Messages specifically requested by user (eg 128 | * VACUUM VERBOSE output); always sent to 129 | * client regardless of client_min_messages, 130 | * but by default not sent to server log. */ 131 | #define NOTICE 18 /* Helpful messages to users about query 132 | * operation; sent to client and server log by 133 | * default. */ 134 | #define WARNING 19 /* Warnings. NOTICE is for expected messages 135 | * like implicit sequence creation by SERIAL. 136 | * WARNING is for unexpected messages. */ 137 | #define ERROR 20 /* user error - abort transaction; return to 138 | * known state */ 139 | 140 | 141 | #ifdef HAVE_FUNCNAME__FUNC 142 | #define PG_FUNCNAME_MACRO __func__ 143 | #else 144 | #ifdef HAVE_FUNCNAME__FUNCTION 145 | #define PG_FUNCNAME_MACRO __FUNCTION__ 146 | #else 147 | #define PG_FUNCNAME_MACRO NULL 148 | #endif 149 | #endif 150 | 151 | extern bool bdr_process_remote_action(StringInfo s, ALI_PG_DECODE_MESSAGE *msg); 152 | 153 | /* 154 | #define elog \ 155 | elog_start(__FILE__, __LINE__, PG_FUNCNAME_MACRO), \ 156 | elog_finish 157 | 158 | extern void elog_start(const char *filename, int lineno, const char *funcname); 159 | extern void 160 | elog_finish(int elevel, const char *fmt,...) 161 | __attribute__((format(PG_PRINTF_ATTRIBUTE, 2, 3))); 162 | */ 163 | 164 | 165 | extern char *timestamptz_to_str(TimestampTz dt); 166 | extern int64 timestamptz_to_time_t(TimestampTz t); 167 | extern void out_put_tuple(ALI_PG_DECODE_MESSAGE *msg, PQExpBuffer buffer, Decode_TupleData *tuple); 168 | extern int out_put_tuple_to_sql(Decoder_handler *hander, ALI_PG_DECODE_MESSAGE *msg, PQExpBuffer buffer); 169 | extern void out_put_key_att(ALI_PG_DECODE_MESSAGE *msg, PQExpBuffer buffer); 170 | extern void out_put_decode_message(Decoder_handler *hander, ALI_PG_DECODE_MESSAGE *msg, int outfd); 171 | extern bool sendFeedback(Decoder_handler *hander, int64 now, bool force, bool replyRequested); 172 | extern int initialize_connection(Decoder_handler *hander); 173 | extern void disconnect(Decoder_handler *hander); 174 | extern int check_handler_parameters(Decoder_handler *hander); 175 | extern char *create_replication_slot(Decoder_handler *hander,XLogRecPtr *lsn, char *replication_slot); 176 | extern int drop_replication_slot(Decoder_handler *hander); 177 | extern int init_logfile(Decoder_handler *hander); 178 | extern int init_streaming(Decoder_handler *hander); 179 | extern ALI_PG_DECODE_MESSAGE *exec_logical_decoder(Decoder_handler *hander, volatile bool *time_to_stop); 180 | extern void pg_sleep(long microsec); 181 | extern Decoder_handler *init_hander(void); 182 | 183 | #ifdef __cplusplus 184 | } 185 | #endif 186 | 187 | 188 | 189 | #endif /* PG_LOGICALDECODE_H */ 190 | 191 | 192 | -------------------------------------------------------------------------------- /dbsync/pgsync.c: -------------------------------------------------------------------------------- 1 | /* 2 | * pg_sync.c 3 | * 4 | */ 5 | 6 | 7 | #include "postgres_fe.h" 8 | #include "lib/stringinfo.h" 9 | #include "common/fe_memutils.h" 10 | 11 | #include "libpq-fe.h" 12 | 13 | #include "access/transam.h" 14 | #include "libpq/pqformat.h" 15 | #include "pg_logicaldecode.h" 16 | #include "pqexpbuffer.h" 17 | #include "pgsync.h" 18 | #include "libpq/pqsignal.h" 19 | 20 | #include 21 | 22 | #ifndef WIN32 23 | #include 24 | #endif 25 | 26 | 27 | 28 | static void *copy_table_data(void *arg); 29 | static char *get_synchronized_snapshot(PGconn *conn); 30 | static bool is_slot_exists(PGconn *conn, char *slotname); 31 | static void *logical_decoding_receive_thread(void *arg); 32 | static void get_task_status(PGconn *conn, char **full_start, char **full_end, char **decoder_start, char **apply_id); 33 | static void update_task_status(PGconn *conn, bool full_start, bool full_end, bool decoder_start, int64 apply_id); 34 | static void *logical_decoding_apply_thread(void *arg); 35 | static int64 get_apply_status(PGconn *conn); 36 | static void sigint_handler(int signum); 37 | 38 | static volatile bool time_to_abort = false; 39 | 40 | 41 | #define ERROR_DUPLICATE_KEY 23505 42 | 43 | #define SQL_TYPE_BEGIN 0 44 | #define SQL_TYPE_COMMIT 1 45 | #define SQL_TYPE_FIRST_STATMENT 2 46 | #define SQL_TYPE_OTHER_STATMENT 3 47 | 48 | #define RECONNECT_SLEEP_TIME 5 49 | 50 | #define ALL_DB_TABLE_SQL "select n.nspname, c.relname from pg_class c, pg_namespace n where n.oid = c.relnamespace and c.relkind = 'r' and n.nspname not in ('pg_catalog','tiger','tiger_data','topology','postgis','information_schema','gp_toolkit','pg_aoseg','pg_toast') order by c.relpages desc;" 51 | #define GET_NAPSHOT "SELECT pg_export_snapshot()" 52 | 53 | #define TASK_ID "1" 54 | 55 | #ifndef WIN32 56 | static void 57 | sigint_handler(int signum) 58 | { 59 | time_to_abort = true; 60 | } 61 | #else 62 | static int64 atoll(const char *nptr) 63 | { 64 | return atol(nptr); 65 | } 66 | #endif 67 | 68 | 69 | /* 70 | * COPY single table over wire. 71 | */ 72 | static void * 73 | copy_table_data(void *arg) 74 | { 75 | ThreadArg *args = (ThreadArg *)arg; 76 | Thread_hd *hd = args->hd; 77 | PGresult *res1; 78 | PGresult *res2; 79 | int bytes; 80 | char *copybuf; 81 | StringInfoData query; 82 | char *nspname; 83 | char *relname; 84 | Task_hd *curr = NULL; 85 | TimevalStruct before, 86 | after; 87 | double elapsed_msec = 0; 88 | 89 | PGconn *origin_conn = args->from; 90 | PGconn *target_conn = args->to; 91 | 92 | origin_conn = pglogical_connect(hd->src, EXTENSION_NAME "_copy"); 93 | if (origin_conn == NULL) 94 | { 95 | fprintf(stderr, "init src conn failed: %s", PQerrorMessage(origin_conn)); 96 | return NULL; 97 | } 98 | 99 | target_conn = pglogical_connect(hd->desc, EXTENSION_NAME "_copy"); 100 | if (target_conn == NULL) 101 | { 102 | fprintf(stderr, "init desc conn failed: %s", PQerrorMessage(target_conn)); 103 | return NULL; 104 | } 105 | 106 | initStringInfo(&query); 107 | while(1) 108 | { 109 | int nlist = 0; 110 | 111 | GETTIMEOFDAY(&before); 112 | pthread_mutex_lock(&hd->t_lock); 113 | nlist = hd->ntask; 114 | if (nlist == 1) 115 | { 116 | curr = hd->l_task; 117 | hd->l_task = NULL; 118 | hd->ntask = 0; 119 | } 120 | else if (nlist > 1) 121 | { 122 | Task_hd *tmp = hd->l_task->next; 123 | curr = hd->l_task; 124 | hd->l_task = tmp; 125 | hd->ntask--; 126 | } 127 | else 128 | { 129 | curr = NULL; 130 | } 131 | 132 | pthread_mutex_unlock(&hd->t_lock); 133 | 134 | if(curr == NULL) 135 | { 136 | break; 137 | } 138 | 139 | start_copy_origin_tx(origin_conn, hd->snapshot, hd->src_version, hd->desc_is_greenplum); 140 | start_copy_target_tx(target_conn, hd->desc_version, hd->desc_is_greenplum); 141 | 142 | nspname = curr->schemaname; 143 | relname = curr->relname; 144 | 145 | /* Build COPY TO query. */ 146 | appendStringInfo(&query, "COPY %s.%s TO stdout", 147 | PQescapeIdentifier(origin_conn, nspname, 148 | strlen(nspname)), 149 | PQescapeIdentifier(origin_conn, relname, 150 | strlen(relname))); 151 | 152 | /* Execute COPY TO. */ 153 | res1 = PQexec(origin_conn, query.data); 154 | if (PQresultStatus(res1) != PGRES_COPY_OUT) 155 | { 156 | fprintf(stderr,"table copy failed Query '%s': %s", 157 | query.data, PQerrorMessage(origin_conn)); 158 | goto exit; 159 | } 160 | 161 | /* Build COPY FROM query. */ 162 | resetStringInfo(&query); 163 | appendStringInfo(&query, "COPY %s.%s FROM stdin", 164 | PQescapeIdentifier(target_conn, nspname, 165 | strlen(nspname)), 166 | PQescapeIdentifier(target_conn, relname, 167 | strlen(relname))); 168 | 169 | /* Execute COPY FROM. */ 170 | res2 = PQexec(target_conn, query.data); 171 | if (PQresultStatus(res2) != PGRES_COPY_IN) 172 | { 173 | fprintf(stderr,"table copy failed Query '%s': %s", 174 | query.data, PQerrorMessage(target_conn)); 175 | goto exit; 176 | } 177 | 178 | while ((bytes = PQgetCopyData(origin_conn, ©buf, false)) > 0) 179 | { 180 | if (PQputCopyData(target_conn, copybuf, bytes) != 1) 181 | { 182 | fprintf(stderr,"writing to target table failed destination connection reported: %s", 183 | PQerrorMessage(target_conn)); 184 | goto exit; 185 | } 186 | args->count++; 187 | curr->count++; 188 | PQfreemem(copybuf); 189 | } 190 | 191 | if (bytes != -1) 192 | { 193 | fprintf(stderr,"reading from origin table failed source connection returned %d: %s", 194 | bytes, PQerrorMessage(origin_conn)); 195 | goto exit; 196 | } 197 | 198 | /* Send local finish */ 199 | if (PQputCopyEnd(target_conn, NULL) != 1) 200 | { 201 | fprintf(stderr,"sending copy-completion to destination connection failed destination connection reported: %s", 202 | PQerrorMessage(target_conn)); 203 | goto exit; 204 | } 205 | 206 | PQclear(res2); 207 | res2 = PQgetResult(target_conn); 208 | if (PQresultStatus(res2) != PGRES_COMMAND_OK) 209 | { 210 | fprintf(stderr, "COPY failed for table \"%s\": %s", 211 | relname, PQerrorMessage(target_conn)); 212 | goto exit; 213 | } 214 | 215 | finish_copy_origin_tx(origin_conn); 216 | finish_copy_target_tx(target_conn); 217 | curr->complete = true; 218 | PQclear(res1); 219 | PQclear(res2); 220 | resetStringInfo(&query); 221 | 222 | GETTIMEOFDAY(&after); 223 | DIFF_MSEC(&after, &before, elapsed_msec); 224 | fprintf(stderr,"thread %d migrate task %d table %s.%s %ld rows complete, time cost %.3f ms\n", 225 | args->id, curr->id, nspname, relname, curr->count, elapsed_msec); 226 | } 227 | 228 | args->all_ok = true; 229 | 230 | exit: 231 | 232 | PQfinish(origin_conn); 233 | PQfinish(target_conn); 234 | ThreadExit(0); 235 | return NULL; 236 | } 237 | 238 | 239 | int 240 | db_sync_main(char *src, char *desc, char *local, int nthread) 241 | { 242 | int i = 0; 243 | Thread_hd th_hd; 244 | Thread *thread = NULL; 245 | PGresult *res = NULL; 246 | PGconn *origin_conn_repl; 247 | PGconn *desc_conn; 248 | PGconn *local_conn; 249 | char *snapshot = NULL; 250 | XLogRecPtr lsn = 0; 251 | long s_count = 0; 252 | long t_count = 0; 253 | bool have_err = false; 254 | TimevalStruct before, 255 | after; 256 | double elapsed_msec = 0; 257 | Decoder_handler *hander = NULL; 258 | struct Thread *decoder = NULL; 259 | bool replication_sync = false; 260 | bool need_full_sync = false; 261 | char *full_start = NULL; 262 | char *full_end = NULL; 263 | char *decoder_start = NULL; 264 | char *apply_id = NULL; 265 | int ntask = 0; 266 | 267 | #ifndef WIN32 268 | signal(SIGINT, sigint_handler); 269 | #endif 270 | 271 | GETTIMEOFDAY(&before); 272 | 273 | memset(&th_hd, 0, sizeof(Thread_hd)); 274 | th_hd.nth = nthread; 275 | th_hd.src = src; 276 | th_hd.desc = desc; 277 | th_hd.local = local; 278 | 279 | origin_conn_repl = pglogical_connect(src, EXTENSION_NAME "_main"); 280 | if (origin_conn_repl == NULL) 281 | { 282 | fprintf(stderr, "conn to src faild: %s", PQerrorMessage(origin_conn_repl)); 283 | return 1; 284 | } 285 | th_hd.src_version = PQserverVersion(origin_conn_repl); 286 | th_hd.src_is_greenplum = is_greenplum(origin_conn_repl); 287 | 288 | desc_conn = pglogical_connect(desc, EXTENSION_NAME "_main"); 289 | if (desc_conn == NULL) 290 | { 291 | fprintf(stderr, "init desc conn failed: %s", PQerrorMessage(desc_conn)); 292 | return 1; 293 | } 294 | th_hd.desc_version = PQserverVersion(desc_conn); 295 | th_hd.desc_is_greenplum = is_greenplum(desc_conn); 296 | PQfinish(desc_conn); 297 | 298 | local_conn = pglogical_connect(local, EXTENSION_NAME "_main"); 299 | if (local_conn == NULL) 300 | { 301 | fprintf(stderr, "init local conn failed: %s", PQerrorMessage(local_conn)); 302 | need_full_sync = true; 303 | } 304 | else 305 | { 306 | ExecuteSqlStatement(local_conn, "CREATE TABLE IF NOT EXISTS sync_sqls(id bigserial, sql text)"); 307 | ExecuteSqlStatement(local_conn, "CREATE TABLE IF NOT EXISTS db_sync_status(id bigserial primary key, full_s_start timestamp DEFAULT NULL, full_s_end timestamp DEFAULT NULL, decoder_start timestamp DEFAULT NULL, apply_id bigint DEFAULT NULL)"); 308 | ExecuteSqlStatement(local_conn, "insert into db_sync_status (id) values (" TASK_ID ");"); 309 | get_task_status(local_conn, &full_start, &full_end, &decoder_start, &apply_id); 310 | 311 | if (full_start && full_end == NULL) 312 | { 313 | fprintf(stderr, "full sync start %s, but not finish.truncate all data and restart dbsync\n", full_start); 314 | return 1; 315 | } 316 | else if(full_start == NULL && full_end == NULL) 317 | { 318 | need_full_sync = true; 319 | fprintf(stderr, "new dbsync task"); 320 | } 321 | else if(full_start && full_end) 322 | { 323 | fprintf(stderr, "full sync start %s, end %s restart decoder sync\n", full_start, full_end); 324 | need_full_sync = false; 325 | } 326 | 327 | if (decoder_start) 328 | { 329 | fprintf(stderr, "decoder sync start %s\n", decoder_start); 330 | } 331 | 332 | if (apply_id) 333 | { 334 | fprintf(stderr, "decoder apply id %s\n", apply_id); 335 | } 336 | } 337 | 338 | if (th_hd.src_is_greenplum == false && th_hd.src_version >= 90400) 339 | { 340 | replication_sync = true; 341 | if (!is_slot_exists(origin_conn_repl, EXTENSION_NAME "_slot")) 342 | { 343 | int rc = 0; 344 | 345 | hander = init_hander(); 346 | hander->connection_string = src; 347 | init_logfile(hander); 348 | rc = initialize_connection(hander); 349 | if(rc != 0) 350 | { 351 | fprintf(stderr, "create replication conn failed\n"); 352 | return 1; 353 | } 354 | hander->do_create_slot = true; 355 | snapshot = create_replication_slot(hander, &lsn, EXTENSION_NAME "_slot"); 356 | if (snapshot == NULL) 357 | { 358 | fprintf(stderr, "create replication slot failed\n"); 359 | return 1; 360 | } 361 | 362 | th_hd.slot_name = hander->replication_slot; 363 | } 364 | else 365 | { 366 | fprintf(stderr, "decoder slot %s exist\n", EXTENSION_NAME "_slot"); 367 | th_hd.slot_name = EXTENSION_NAME "_slot"; 368 | } 369 | } 370 | 371 | if (need_full_sync) 372 | { 373 | const char *setup_query = NULL; 374 | PQExpBuffer query; 375 | 376 | if (th_hd.src_is_greenplum == false) 377 | { 378 | setup_query = "BEGIN TRANSACTION ISOLATION LEVEL REPEATABLE READ, READ ONLY;\n"; 379 | } 380 | else 381 | { 382 | setup_query = "BEGIN"; 383 | } 384 | 385 | query = createPQExpBuffer(); 386 | appendPQExpBuffer(query, "%s", setup_query); 387 | 388 | if (snapshot) 389 | { 390 | appendPQExpBuffer(query, "SET TRANSACTION SNAPSHOT '%s';\n", snapshot); 391 | } 392 | 393 | res = PQexec(origin_conn_repl, query->data); 394 | if (PQresultStatus(res) != PGRES_COMMAND_OK) 395 | { 396 | fprintf(stderr, "init open a tran failed: %s", PQresultErrorMessage(res)); 397 | return 1; 398 | } 399 | resetPQExpBuffer(query); 400 | 401 | if (snapshot == NULL) 402 | { 403 | if (th_hd.src_version >= 90200) 404 | { 405 | snapshot = get_synchronized_snapshot(origin_conn_repl); 406 | th_hd.snapshot = snapshot; 407 | } 408 | } 409 | 410 | appendPQExpBuffer(query, ALL_DB_TABLE_SQL); 411 | res = PQexec(origin_conn_repl, query->data); 412 | if (PQresultStatus(res) != PGRES_TUPLES_OK) 413 | { 414 | fprintf(stderr, "init sql run failed: %s", PQresultErrorMessage(res)); 415 | return 1; 416 | } 417 | 418 | ntask = PQntuples(res); 419 | th_hd.ntask = ntask; 420 | if (th_hd.ntask >= 1) 421 | { 422 | th_hd.task = (Task_hd *)palloc0(sizeof(Task_hd) * th_hd.ntask); 423 | } 424 | 425 | for (i = 0; i < th_hd.ntask; i++) 426 | { 427 | th_hd.task[i].id = i; 428 | th_hd.task[i].schemaname = pstrdup(PQgetvalue(res, i, 0)); 429 | th_hd.task[i].relname = pstrdup(PQgetvalue(res, i, 1)); 430 | th_hd.task[i].count = 0; 431 | th_hd.task[i].complete = false; 432 | if (i != th_hd.ntask - 1) 433 | { 434 | th_hd.task[i].next = &th_hd.task[i+1]; 435 | } 436 | } 437 | 438 | th_hd.l_task = &(th_hd.task[0]); 439 | PQclear(res); 440 | destroyPQExpBuffer(query); 441 | 442 | th_hd.th = (ThreadArg *)malloc(sizeof(ThreadArg) * th_hd.nth); 443 | for (i = 0; i < th_hd.nth; i++) 444 | { 445 | th_hd.th[i].id = i; 446 | th_hd.th[i].count = 0; 447 | th_hd.th[i].all_ok = false; 448 | 449 | th_hd.th[i].hd = &th_hd; 450 | } 451 | pthread_mutex_init(&th_hd.t_lock, NULL); 452 | 453 | fprintf(stderr, "starting full sync"); 454 | if (snapshot) 455 | { 456 | fprintf(stderr, " with snapshot %s", snapshot); 457 | } 458 | fprintf(stderr, "\n"); 459 | 460 | thread = (Thread *)palloc0(sizeof(Thread) * th_hd.nth); 461 | for (i = 0; i < th_hd.nth; i++) 462 | { 463 | ThreadCreate(&thread[i], copy_table_data, &th_hd.th[i]); 464 | } 465 | 466 | update_task_status(local_conn, true, false, false, -1); 467 | } 468 | 469 | if (replication_sync) 470 | { 471 | decoder = (Thread *)palloc0(sizeof(Thread) * 2); 472 | fprintf(stderr, "starting logical decoding sync thread\n"); 473 | ThreadCreate(&decoder[0], logical_decoding_receive_thread, &th_hd); 474 | update_task_status(local_conn, false, false, true, -1); 475 | } 476 | 477 | if (need_full_sync) 478 | { 479 | WaitThreadEnd(th_hd.nth, thread); 480 | update_task_status(local_conn, false, true, false, -1); 481 | 482 | GETTIMEOFDAY(&after); 483 | DIFF_MSEC(&after, &before, elapsed_msec); 484 | 485 | for (i = 0; i < th_hd.nth; i++) 486 | { 487 | if(th_hd.th[i].all_ok) 488 | { 489 | s_count += th_hd.th[i].count; 490 | } 491 | else 492 | { 493 | have_err = true; 494 | } 495 | } 496 | 497 | for (i = 0; i < ntask; i++) 498 | { 499 | t_count += th_hd.task[i].count; 500 | } 501 | 502 | fprintf(stderr, "job migrate row %ld task row %ld \n", s_count, t_count); 503 | fprintf(stderr, "full sync time cost %.3f ms\n", elapsed_msec); 504 | if (have_err) 505 | { 506 | fprintf(stderr, "migration process with errors\n"); 507 | } 508 | } 509 | 510 | if (replication_sync) 511 | { 512 | ThreadCreate(&decoder[1], logical_decoding_apply_thread, &th_hd); 513 | fprintf(stderr, "starting decoder apply thread\n"); 514 | WaitThreadEnd(2, decoder); 515 | } 516 | 517 | PQfinish(origin_conn_repl); 518 | PQfinish(local_conn); 519 | 520 | return 0; 521 | } 522 | 523 | 524 | static char * 525 | get_synchronized_snapshot(PGconn *conn) 526 | { 527 | char *query = "SELECT pg_export_snapshot()"; 528 | char *result; 529 | PGresult *res; 530 | 531 | res = PQexec(conn, query); 532 | if (PQresultStatus(res) != PGRES_TUPLES_OK) 533 | { 534 | fprintf(stderr, "init sql run failed: %s", PQresultErrorMessage(res)); 535 | return NULL; 536 | } 537 | result = pstrdup(PQgetvalue(res, 0, 0)); 538 | PQclear(res); 539 | 540 | return result; 541 | } 542 | 543 | static bool 544 | is_slot_exists(PGconn *conn, char *slotname) 545 | { 546 | PGresult *res; 547 | int ntups; 548 | bool exist = false; 549 | PQExpBuffer query; 550 | 551 | query = createPQExpBuffer(); 552 | appendPQExpBuffer(query, "select slot_name from pg_replication_slots where slot_name = '%s';", 553 | slotname); 554 | 555 | res = PQexec(conn, query->data); 556 | if (PQresultStatus(res) != PGRES_TUPLES_OK) 557 | { 558 | PQclear(res); 559 | destroyPQExpBuffer(query); 560 | return false; 561 | } 562 | 563 | /* Expecting a single result only */ 564 | ntups = PQntuples(res); 565 | if (ntups == 1) 566 | { 567 | exist = true; 568 | } 569 | 570 | PQclear(res); 571 | destroyPQExpBuffer(query); 572 | 573 | return exist; 574 | } 575 | 576 | static void 577 | get_task_status(PGconn *conn, char **full_start, char **full_end, char **decoder_start, char **apply_id) 578 | { 579 | PGresult *res; 580 | char *query = "SELECT full_s_start , full_s_end, decoder_start, apply_id FROM db_sync_status where id =" TASK_ID; 581 | 582 | res = PQexec(conn, query); 583 | if (PQresultStatus(res) != PGRES_TUPLES_OK) 584 | { 585 | PQclear(res); 586 | return; 587 | } 588 | 589 | if (PQntuples(res) != 1) 590 | { 591 | PQclear(res); 592 | return; 593 | } 594 | 595 | if (!PQgetisnull(res, 0, 0)) 596 | { 597 | *full_start = pstrdup(PQgetvalue(res, 0, 0)); 598 | } 599 | 600 | if (!PQgetisnull(res, 0, 1)) 601 | { 602 | *full_end = pstrdup(PQgetvalue(res, 0, 1)); 603 | } 604 | 605 | if (!PQgetisnull(res, 0, 2)) 606 | { 607 | *decoder_start = pstrdup(PQgetvalue(res, 0, 2)); 608 | } 609 | 610 | if (!PQgetisnull(res, 0, 3)) 611 | { 612 | *apply_id = pstrdup(PQgetvalue(res, 0, 3)); 613 | } 614 | 615 | PQclear(res); 616 | 617 | return; 618 | } 619 | 620 | static void 621 | update_task_status(PGconn *conn, bool full_start, bool full_end, bool decoder_start, int64 apply_id) 622 | { 623 | PQExpBuffer query; 624 | 625 | query = createPQExpBuffer(); 626 | 627 | if (full_start) 628 | { 629 | appendPQExpBuffer(query, "UPDATE db_sync_status SET full_s_start = now() WHERE id = %s", 630 | TASK_ID); 631 | ExecuteSqlStatement(conn, query->data); 632 | } 633 | 634 | if (full_end) 635 | { 636 | appendPQExpBuffer(query, "UPDATE db_sync_status SET full_s_end = now() WHERE id = %s", 637 | TASK_ID); 638 | ExecuteSqlStatement(conn, query->data); 639 | } 640 | 641 | if (decoder_start) 642 | { 643 | appendPQExpBuffer(query, "UPDATE db_sync_status SET decoder_start = now() WHERE id = %s", 644 | TASK_ID); 645 | ExecuteSqlStatement(conn, query->data); 646 | } 647 | 648 | if (apply_id >= 0) 649 | { 650 | appendPQExpBuffer(query, "UPDATE db_sync_status SET apply_id = " INT64_FORMAT " WHERE id = %s", 651 | apply_id, TASK_ID); 652 | ExecuteSqlStatement(conn, query->data); 653 | } 654 | 655 | destroyPQExpBuffer(query); 656 | 657 | return; 658 | } 659 | 660 | static void * 661 | logical_decoding_receive_thread(void *arg) 662 | { 663 | Thread_hd *hd = (Thread_hd *)arg; 664 | Decoder_handler *hander; 665 | int rc = 0; 666 | bool init = false; 667 | PGconn *local_conn; 668 | PQExpBuffer buffer; 669 | char *stmtname = "insert_sqls"; 670 | Oid type[1]; 671 | const char *paramValues[1]; 672 | PGresult *res = NULL; 673 | 674 | type[0] = 25; 675 | buffer = createPQExpBuffer(); 676 | 677 | local_conn = pglogical_connect(hd->local, EXTENSION_NAME "_decoding"); 678 | if (local_conn == NULL) 679 | { 680 | fprintf(stderr, "init src conn failed: %s", PQerrorMessage(local_conn)); 681 | goto exit; 682 | } 683 | setup_connection(local_conn, 90400, false); 684 | 685 | res = PQprepare(local_conn, stmtname, "INSERT INTO sync_sqls (sql) VALUES($1)", 1, type); 686 | if (PQresultStatus(res) != PGRES_COMMAND_OK) 687 | { 688 | fprintf(stderr, "create PQprepare failed: %s", PQerrorMessage(local_conn)); 689 | PQfinish(local_conn); 690 | goto exit; 691 | } 692 | PQclear(res); 693 | 694 | hander = init_hander(); 695 | hander->connection_string = hd->src; 696 | init_logfile(hander); 697 | rc = check_handler_parameters(hander); 698 | if(rc != 0) 699 | { 700 | exit(1); 701 | } 702 | 703 | rc = initialize_connection(hander); 704 | if(rc != 0) 705 | { 706 | exit(1); 707 | } 708 | 709 | hander->replication_slot = hd->slot_name; 710 | init_streaming(hander); 711 | init = true; 712 | 713 | while (true) 714 | { 715 | ALI_PG_DECODE_MESSAGE *msg = NULL; 716 | 717 | if (time_to_abort) 718 | { 719 | if (hander->copybuf != NULL) 720 | { 721 | PQfreemem(hander->copybuf); 722 | hander->copybuf = NULL; 723 | } 724 | if (hander->conn) 725 | { 726 | PQfinish(hander->conn); 727 | hander->conn = NULL; 728 | } 729 | if (local_conn) 730 | { 731 | PQdescribePrepared(local_conn, stmtname); 732 | PQfinish(local_conn); 733 | } 734 | break; 735 | } 736 | 737 | if (!init) 738 | { 739 | initialize_connection(hander); 740 | init_streaming(hander); 741 | init = true; 742 | } 743 | 744 | msg = exec_logical_decoder(hander, &time_to_abort); 745 | if (msg != NULL) 746 | { 747 | out_put_tuple_to_sql(hander, msg, buffer); 748 | if(msg->type == MSGKIND_BEGIN) 749 | { 750 | res = PQexec(local_conn, "BEGIN"); 751 | if (PQresultStatus(res) != PGRES_COMMAND_OK) 752 | { 753 | fprintf(stderr, "decoding receive thread begin a local trans failed: %s", PQerrorMessage(local_conn)); 754 | goto exit; 755 | } 756 | PQclear(res); 757 | } 758 | 759 | paramValues[0] = buffer->data; 760 | res = PQexecPrepared(local_conn, stmtname, 1, paramValues, NULL, NULL, 1); 761 | if (PQresultStatus(res) != PGRES_COMMAND_OK) 762 | { 763 | fprintf(stderr, "exec prepare INSERT INTO sync_sqls failed: %s", PQerrorMessage(local_conn)); 764 | time_to_abort = true; 765 | goto exit; 766 | } 767 | PQclear(res); 768 | 769 | hander->flushpos = hander->recvpos; 770 | if(msg->type == MSGKIND_COMMIT) 771 | { 772 | res = PQexec(local_conn, "END"); 773 | if (PQresultStatus(res) != PGRES_COMMAND_OK) 774 | { 775 | fprintf(stderr, "decoding receive thread commit a local trans failed: %s", PQerrorMessage(local_conn)); 776 | goto exit; 777 | } 778 | PQclear(res); 779 | } 780 | 781 | resetPQExpBuffer(buffer); 782 | } 783 | else 784 | { 785 | fprintf(stderr, "decoding receive no record, sleep and reconnect"); 786 | pg_sleep(RECONNECT_SLEEP_TIME * 1000000); 787 | init = false; 788 | } 789 | } 790 | 791 | 792 | exit: 793 | 794 | destroyPQExpBuffer(buffer); 795 | 796 | ThreadExit(0); 797 | return NULL; 798 | } 799 | 800 | static void * 801 | logical_decoding_apply_thread(void *arg) 802 | { 803 | Thread_hd *hd = (Thread_hd *)arg; 804 | PGconn *local_conn = NULL; 805 | PGconn *local_conn_u = NULL; 806 | PGconn *apply_conn = NULL; 807 | Oid type[1]; 808 | PGresult *resreader = NULL; 809 | PGresult *applyres = NULL; 810 | int pgversion; 811 | bool is_gp = false; 812 | int64 apply_id = 0; 813 | 814 | type[0] = 25; 815 | 816 | local_conn = pglogical_connect(hd->local, EXTENSION_NAME "apply_reader"); 817 | if (local_conn == NULL) 818 | { 819 | fprintf(stderr, "decoding applyer init src conn failed: %s", PQerrorMessage(local_conn)); 820 | goto exit; 821 | } 822 | setup_connection(local_conn, 90400, false); 823 | apply_id = get_apply_status(local_conn); 824 | if (apply_id == -1) 825 | { 826 | goto exit; 827 | } 828 | 829 | local_conn_u = pglogical_connect(hd->local, EXTENSION_NAME "apply_update_status"); 830 | if (local_conn_u == NULL) 831 | { 832 | fprintf(stderr, "decoding applyer init src conn failed: %s", PQerrorMessage(local_conn_u)); 833 | goto exit; 834 | } 835 | setup_connection(local_conn_u, 90400, false); 836 | 837 | apply_conn = pglogical_connect(hd->desc, EXTENSION_NAME "_decoding_apply"); 838 | if (apply_conn == NULL) 839 | { 840 | fprintf(stderr, "decoding_apply init desc conn failed: %s", PQerrorMessage(apply_conn)); 841 | goto exit; 842 | } 843 | pgversion = PQserverVersion(apply_conn); 844 | is_gp = is_greenplum(apply_conn); 845 | setup_connection(apply_conn, pgversion, is_gp); 846 | 847 | while (!time_to_abort) 848 | { 849 | const char *paramValues[1]; 850 | char *ssql; 851 | char tmp[16]; 852 | int n_commit = 0; 853 | int sqltype = SQL_TYPE_BEGIN; 854 | 855 | sprintf(tmp, INT64_FORMAT, apply_id); 856 | paramValues[0] = tmp; 857 | 858 | resreader = PQexec(local_conn, "BEGIN"); 859 | if (PQresultStatus(resreader) != PGRES_COMMAND_OK) 860 | { 861 | fprintf(stderr, "BEGIN command failed: %s\n", PQerrorMessage(local_conn)); 862 | PQclear(resreader); 863 | goto exit; 864 | } 865 | PQclear(resreader); 866 | 867 | resreader = PQexecParams(local_conn, 868 | "DECLARE ali_decoder_cursor CURSOR FOR select id, sql from sync_sqls where id > $1 order by id", 869 | 1, 870 | NULL, 871 | paramValues, 872 | NULL, 873 | NULL, 874 | 1); 875 | 876 | if (PQresultStatus(resreader) != PGRES_COMMAND_OK) 877 | { 878 | fprintf(stderr, "DECLARE CURSOR command failed: %s\n", PQerrorMessage(local_conn)); 879 | PQclear(resreader); 880 | goto exit; 881 | } 882 | PQclear(resreader); 883 | 884 | while(!time_to_abort) 885 | { 886 | resreader = PQexec(local_conn, "FETCH FROM ali_decoder_cursor"); 887 | if (PQresultStatus(resreader) != PGRES_TUPLES_OK) 888 | { 889 | fprintf(stderr, "FETCH ALL command didn't return tuples properly: %s\n", PQerrorMessage(local_conn)); 890 | PQclear(resreader); 891 | } 892 | 893 | if (PQntuples(resreader) == 0) 894 | { 895 | PQclear(resreader); 896 | resreader = PQexec(local_conn, "CLOSE ali_decoder_cursor"); 897 | PQclear(resreader); 898 | resreader = PQexec(local_conn, "END"); 899 | PQclear(resreader); 900 | 901 | if (n_commit != 0) 902 | { 903 | n_commit = 0; 904 | update_task_status(local_conn_u, false, false, false, apply_id); 905 | } 906 | 907 | pg_sleep(1000000); 908 | break; 909 | } 910 | 911 | ssql = PQgetvalue(resreader, 0, 1); 912 | if(strcmp(ssql,"begin;") == 0) 913 | { 914 | sqltype = SQL_TYPE_BEGIN; 915 | } 916 | else if (strcmp(ssql,"commit;") == 0) 917 | { 918 | sqltype = SQL_TYPE_COMMIT; 919 | } 920 | else if(sqltype == SQL_TYPE_BEGIN) 921 | { 922 | sqltype = SQL_TYPE_FIRST_STATMENT; 923 | } 924 | else 925 | { 926 | sqltype = SQL_TYPE_OTHER_STATMENT; 927 | } 928 | 929 | applyres = PQexec(apply_conn, ssql); 930 | if (PQresultStatus(applyres) != PGRES_COMMAND_OK) 931 | { 932 | char *sqlstate = PQresultErrorField(applyres, PG_DIAG_SQLSTATE); 933 | int errcode = 0; 934 | fprintf(stderr, "exec apply id %s, sql %s failed: %s\n", PQgetvalue(resreader, 0, 0), ssql, PQerrorMessage(apply_conn)); 935 | errcode = atoi(sqlstate); 936 | if (errcode == ERROR_DUPLICATE_KEY && sqltype == SQL_TYPE_FIRST_STATMENT) 937 | { 938 | PQclear(applyres); 939 | applyres = PQexec(apply_conn, "END"); 940 | if (PQresultStatus(applyres) != PGRES_COMMAND_OK) 941 | { 942 | goto exit; 943 | } 944 | PQclear(applyres); 945 | applyres = PQexec(apply_conn, "BEGIN"); 946 | if (PQresultStatus(applyres) != PGRES_COMMAND_OK) 947 | { 948 | goto exit; 949 | } 950 | sqltype = SQL_TYPE_BEGIN; 951 | } 952 | else 953 | { 954 | PQclear(resreader); 955 | PQclear(applyres); 956 | goto exit; 957 | } 958 | } 959 | 960 | if (sqltype == SQL_TYPE_COMMIT) 961 | { 962 | n_commit++; 963 | apply_id = atoll(PQgetvalue(resreader, 0, 0)); 964 | if(n_commit == 5) 965 | { 966 | n_commit = 0; 967 | update_task_status(local_conn_u, false, false, false, apply_id); 968 | } 969 | } 970 | PQclear(resreader); 971 | PQclear(applyres); 972 | } 973 | } 974 | 975 | exit: 976 | 977 | if (local_conn) 978 | { 979 | PQfinish(local_conn); 980 | } 981 | 982 | if (local_conn_u) 983 | { 984 | PQfinish(local_conn_u); 985 | } 986 | 987 | if (apply_conn) 988 | { 989 | PQfinish(apply_conn); 990 | } 991 | 992 | time_to_abort = true; 993 | 994 | ThreadExit(0); 995 | return NULL; 996 | } 997 | 998 | static int64 999 | get_apply_status(PGconn *conn) 1000 | { 1001 | PGresult *res; 1002 | char *query = "SELECT apply_id FROM db_sync_status where id =" TASK_ID; 1003 | int64 rc = 0; 1004 | 1005 | res = PQexec(conn, query); 1006 | if (PQresultStatus(res) != PGRES_TUPLES_OK) 1007 | { 1008 | PQclear(res); 1009 | return -1; 1010 | } 1011 | 1012 | if (PQntuples(res) != 1) 1013 | { 1014 | PQclear(res); 1015 | return -1; 1016 | } 1017 | 1018 | if (!PQgetisnull(res, 0, 0)) 1019 | { 1020 | char *tmp = PQgetvalue(res, 0, 0); 1021 | rc = atoll(tmp); 1022 | } 1023 | 1024 | PQclear(res); 1025 | 1026 | return rc; 1027 | } 1028 | 1029 | -------------------------------------------------------------------------------- /dbsync/pgsync.h: -------------------------------------------------------------------------------- 1 | 2 | 3 | #ifndef PG_SYNC_H 4 | #define PG_SYNC_H 5 | 6 | #include "postgres_fe.h" 7 | 8 | #include "lib/stringinfo.h" 9 | #include "lib/stringinfo.h" 10 | #include "common/fe_memutils.h" 11 | 12 | #include "libpq/pqformat.h" 13 | #include "pqexpbuffer.h" 14 | 15 | #include "misc.h" 16 | 17 | #ifdef __cplusplus 18 | extern "C" 19 | { 20 | #endif 21 | 22 | #ifndef WIN32 23 | #include 24 | 25 | typedef struct timeval TimevalStruct; 26 | 27 | #define GETTIMEOFDAY(T) gettimeofday(T, NULL) 28 | #define DIFF_MSEC(T, U, res) \ 29 | do { \ 30 | res = ((((int) ((T)->tv_sec - (U)->tv_sec)) * 1000000.0 + \ 31 | ((int) ((T)->tv_usec - (U)->tv_usec))) / 1000.0); \ 32 | } while(0) 33 | 34 | #else 35 | 36 | #include 37 | #include 38 | #include 39 | 40 | typedef LARGE_INTEGER TimevalStruct; 41 | #define GETTIMEOFDAY(T) QueryPerformanceCounter(T) 42 | #define DIFF_MSEC(T, U, res) \ 43 | do { \ 44 | LARGE_INTEGER frq; \ 45 | \ 46 | QueryPerformanceFrequency(&frq); \ 47 | res = (double)(((T)->QuadPart - (U)->QuadPart)/(double)frq.QuadPart); \ 48 | res *= 1000; \ 49 | } while(0) 50 | 51 | #endif 52 | 53 | #ifdef WIN32 54 | static int64 atoll(const char *nptr); 55 | #endif 56 | 57 | 58 | #include "mysql.h" 59 | 60 | typedef struct ThreadArg 61 | { 62 | int id; 63 | long count; 64 | bool all_ok; 65 | PGconn *from; 66 | PGconn *to; 67 | 68 | struct Thread_hd *hd; 69 | }ThreadArg; 70 | 71 | typedef struct mysql_conn_info 72 | { 73 | char *host; 74 | int port; 75 | char *user; 76 | char *passwd; 77 | char *encoding; 78 | char *db; 79 | char *encodingdir; 80 | char **tabnames; 81 | char **queries; 82 | }mysql_conn_info; 83 | 84 | typedef struct Thread_hd 85 | { 86 | int nth; 87 | struct ThreadArg *th; 88 | 89 | const char *snapshot; 90 | char *src; 91 | int src_version; 92 | bool src_is_greenplum; 93 | 94 | char *slot_name; 95 | 96 | mysql_conn_info *mysql_src; 97 | 98 | char *desc; 99 | int desc_version; 100 | bool desc_is_greenplum; 101 | char *local; 102 | 103 | int ntask; 104 | struct Task_hd *task; 105 | struct Task_hd *l_task; 106 | pthread_mutex_t t_lock; 107 | 108 | int ntask_com; 109 | struct Task_hd *task_com; 110 | pthread_mutex_t t_lock_com; 111 | 112 | uint32 ignore_error_count; 113 | }Thread_hd; 114 | 115 | typedef struct Task_hd 116 | { 117 | int id; 118 | char *schemaname; /* the schema name, or NULL */ 119 | char *relname; /* the relation/sequence name */ 120 | char *query; 121 | long count; 122 | bool complete; 123 | 124 | struct Task_hd *next; 125 | }Task_hd; 126 | 127 | 128 | #define EXTENSION_NAME "rds_logical_sync" 129 | 130 | extern int db_sync_main(char *src, char *desc, char *local, int nthread); 131 | 132 | 133 | extern int mysql2pgsql_sync_main(char *desc, int nthread, mysql_conn_info *hd, char* target_schema, uint32 ignore_error_count); 134 | 135 | 136 | #ifdef __cplusplus 137 | } 138 | #endif 139 | 140 | #endif 141 | 142 | -------------------------------------------------------------------------------- /dbsync/pqformat.c: -------------------------------------------------------------------------------- 1 | /*------------------------------------------------------------------------- 2 | * 3 | * pqformat.c 4 | * Routines for formatting and parsing frontend/backend messages 5 | * 6 | * Outgoing messages are built up in a StringInfo buffer (which is expansible) 7 | * and then sent in a single call to pq_putmessage. This module provides data 8 | * formatting/conversion routines that are needed to produce valid messages. 9 | * Note in particular the distinction between "raw data" and "text"; raw data 10 | * is message protocol characters and binary values that are not subject to 11 | * character set conversion, while text is converted by character encoding 12 | * rules. 13 | * 14 | * Incoming messages are similarly read into a StringInfo buffer, via 15 | * pq_getmessage, and then parsed and converted from that using the routines 16 | * in this module. 17 | * 18 | * These same routines support reading and writing of external binary formats 19 | * (typsend/typreceive routines). The conversion routines for individual 20 | * data types are exactly the same, only initialization and completion 21 | * are different. 22 | * 23 | * 24 | * Portions Copyright (c) 1996-2014, PostgreSQL Global Development Group 25 | * Portions Copyright (c) 1994, Regents of the University of California 26 | * 27 | * src/backend/libpq/pqformat.c 28 | * 29 | *------------------------------------------------------------------------- 30 | */ 31 | /* 32 | * INTERFACE ROUTINES 33 | * Message assembly and output: 34 | * pq_beginmessage - initialize StringInfo buffer 35 | * pq_sendbyte - append a raw byte to a StringInfo buffer 36 | * pq_sendint - append a binary integer to a StringInfo buffer 37 | * pq_sendint64 - append a binary 8-byte int to a StringInfo buffer 38 | * pq_sendfloat4 - append a float4 to a StringInfo buffer 39 | * pq_sendfloat8 - append a float8 to a StringInfo buffer 40 | * pq_sendbytes - append raw data to a StringInfo buffer 41 | * pq_sendcountedtext - append a counted text string (with character set conversion) 42 | * pq_sendtext - append a text string (with conversion) 43 | * pq_sendstring - append a null-terminated text string (with conversion) 44 | * pq_send_ascii_string - append a null-terminated text string (without conversion) 45 | * pq_endmessage - send the completed message to the frontend 46 | * Note: it is also possible to append data to the StringInfo buffer using 47 | * the regular StringInfo routines, but this is discouraged since required 48 | * character set conversion may not occur. 49 | * 50 | * typsend support (construct a bytea value containing external binary data): 51 | * pq_begintypsend - initialize StringInfo buffer 52 | * pq_endtypsend - return the completed string as a "bytea*" 53 | * 54 | * Special-case message output: 55 | * pq_puttextmessage - generate a character set-converted message in one step 56 | * pq_putemptymessage - convenience routine for message with empty body 57 | * 58 | * Message parsing after input: 59 | * pq_getmsgbyte - get a raw byte from a message buffer 60 | * pq_getmsgint - get a binary integer from a message buffer 61 | * pq_getmsgint64 - get a binary 8-byte int from a message buffer 62 | * pq_getmsgfloat4 - get a float4 from a message buffer 63 | * pq_getmsgfloat8 - get a float8 from a message buffer 64 | * pq_getmsgbytes - get raw data from a message buffer 65 | * pq_copymsgbytes - copy raw data from a message buffer 66 | * pq_getmsgtext - get a counted text string (with conversion) 67 | * pq_getmsgstring - get a null-terminated text string (with conversion) 68 | * pq_getmsgend - verify message fully consumed 69 | */ 70 | 71 | #include "postgres_fe.h" 72 | 73 | #include "lib/stringinfo.h" 74 | #include "pg_logicaldecode.h" 75 | 76 | #include "utils.h" 77 | 78 | //#include 79 | //#include 80 | //#include 81 | 82 | //#include "libpq/libpq.h" 83 | //#include "libpq/pqformat.h" 84 | //#include "mb/pg_wchar.h" 85 | 86 | /* -------------------------------- 87 | * pq_beginmessage - initialize for sending a message 88 | * -------------------------------- 89 | */ 90 | void 91 | pq_beginmessage(StringInfo buf, char msgtype) 92 | { 93 | initStringInfo(buf); 94 | 95 | /* 96 | * We stash the message type into the buffer's cursor field, expecting 97 | * that the pq_sendXXX routines won't touch it. We could alternatively 98 | * make it the first byte of the buffer contents, but this seems easier. 99 | */ 100 | buf->cursor = msgtype; 101 | } 102 | 103 | /* -------------------------------- 104 | * pq_sendbyte - append a raw byte to a StringInfo buffer 105 | * -------------------------------- 106 | */ 107 | void 108 | pq_sendbyte(StringInfo buf, int byt) 109 | { 110 | appendStringInfoCharMacro(buf, byt); 111 | } 112 | 113 | /* -------------------------------- 114 | * pq_sendbytes - append raw data to a StringInfo buffer 115 | * -------------------------------- 116 | */ 117 | void 118 | pq_sendbytes(StringInfo buf, const char *data, int datalen) 119 | { 120 | appendBinaryStringInfo(buf, data, datalen); 121 | } 122 | 123 | /* -------------------------------- 124 | * pq_sendcountedtext - append a counted text string (with character set conversion) 125 | * 126 | * The data sent to the frontend by this routine is a 4-byte count field 127 | * followed by the string. The count includes itself or not, as per the 128 | * countincludesself flag (pre-3.0 protocol requires it to include itself). 129 | * The passed text string need not be null-terminated, and the data sent 130 | * to the frontend isn't either. 131 | * -------------------------------- 132 | */ 133 | /* 134 | void 135 | pq_sendcountedtext(StringInfo buf, const char *str, int slen, 136 | bool countincludesself) 137 | { 138 | int extra = countincludesself ? 4 : 0; 139 | char *p; 140 | 141 | p = pg_server_to_client(str, slen); 142 | if (p != str) 143 | { 144 | slen = strlen(p); 145 | pq_sendint(buf, slen + extra, 4); 146 | appendBinaryStringInfo(buf, p, slen); 147 | pfree(p); 148 | } 149 | else 150 | { 151 | pq_sendint(buf, slen + extra, 4); 152 | appendBinaryStringInfo(buf, str, slen); 153 | } 154 | } 155 | */ 156 | /* -------------------------------- 157 | * pq_sendtext - append a text string (with conversion) 158 | * 159 | * The passed text string need not be null-terminated, and the data sent 160 | * to the frontend isn't either. Note that this is not actually useful 161 | * for direct frontend transmissions, since there'd be no way for the 162 | * frontend to determine the string length. But it is useful for binary 163 | * format conversions. 164 | * -------------------------------- 165 | */ 166 | /* 167 | void 168 | pq_sendtext(StringInfo buf, const char *str, int slen) 169 | { 170 | char *p; 171 | 172 | p = pg_server_to_client(str, slen); 173 | if (p != str) 174 | { 175 | slen = strlen(p); 176 | appendBinaryStringInfo(buf, p, slen); 177 | pfree(p); 178 | } 179 | else 180 | appendBinaryStringInfo(buf, str, slen); 181 | } 182 | */ 183 | 184 | /* -------------------------------- 185 | * pq_sendstring - append a null-terminated text string (with conversion) 186 | * 187 | * NB: passed text string must be null-terminated, and so is the data 188 | * sent to the frontend. 189 | * -------------------------------- 190 | */ 191 | /* 192 | void 193 | pq_sendstring(StringInfo buf, const char *str) 194 | { 195 | int slen = strlen(str); 196 | char *p; 197 | 198 | p = pg_server_to_client(str, slen); 199 | if (p != str) 200 | { 201 | slen = strlen(p); 202 | appendBinaryStringInfo(buf, p, slen + 1); 203 | pfree(p); 204 | } 205 | else 206 | appendBinaryStringInfo(buf, str, slen + 1); 207 | } 208 | */ 209 | 210 | /* -------------------------------- 211 | * pq_send_ascii_string - append a null-terminated text string (without conversion) 212 | * 213 | * This function intentionally bypasses encoding conversion, instead just 214 | * silently replacing any non-7-bit-ASCII characters with question marks. 215 | * It is used only when we are having trouble sending an error message to 216 | * the client with normal localization and encoding conversion. The caller 217 | * should already have taken measures to ensure the string is just ASCII; 218 | * the extra work here is just to make certain we don't send a badly encoded 219 | * string to the client (which might or might not be robust about that). 220 | * 221 | * NB: passed text string must be null-terminated, and so is the data 222 | * sent to the frontend. 223 | * -------------------------------- 224 | */ 225 | void 226 | pq_send_ascii_string(StringInfo buf, const char *str) 227 | { 228 | while (*str) 229 | { 230 | char ch = *str++; 231 | 232 | if (IS_HIGHBIT_SET(ch)) 233 | ch = '?'; 234 | appendStringInfoCharMacro(buf, ch); 235 | } 236 | appendStringInfoChar(buf, '\0'); 237 | } 238 | 239 | /* -------------------------------- 240 | * pq_sendint - append a binary integer to a StringInfo buffer 241 | * -------------------------------- 242 | */ 243 | void 244 | pq_sendint(StringInfo buf, int i, int b) 245 | { 246 | unsigned char n8; 247 | uint16 n16; 248 | uint32 n32; 249 | 250 | switch (b) 251 | { 252 | case 1: 253 | n8 = (unsigned char) i; 254 | appendBinaryStringInfo(buf, (char *) &n8, 1); 255 | break; 256 | case 2: 257 | n16 = htons((uint16) i); 258 | appendBinaryStringInfo(buf, (char *) &n16, 2); 259 | break; 260 | case 4: 261 | n32 = htonl((uint32) i); 262 | appendBinaryStringInfo(buf, (char *) &n32, 4); 263 | break; 264 | default: 265 | fprintf(stderr, "unsupported integer size %d", b); 266 | break; 267 | } 268 | } 269 | 270 | /* -------------------------------- 271 | * pq_sendint64 - append a binary 8-byte int to a StringInfo buffer 272 | * 273 | * It is tempting to merge this with pq_sendint, but we'd have to make the 274 | * argument int64 for all data widths --- that could be a big performance 275 | * hit on machines where int64 isn't efficient. 276 | * -------------------------------- 277 | */ 278 | void 279 | pq_sendint64(StringInfo buf, int64 i) 280 | { 281 | uint32 n32; 282 | 283 | /* High order half first, since we're doing MSB-first */ 284 | n32 = (uint32) (i >> 32); 285 | n32 = htonl(n32); 286 | appendBinaryStringInfo(buf, (char *) &n32, 4); 287 | 288 | /* Now the low order half */ 289 | n32 = (uint32) i; 290 | n32 = htonl(n32); 291 | appendBinaryStringInfo(buf, (char *) &n32, 4); 292 | } 293 | 294 | /* -------------------------------- 295 | * pq_sendfloat4 - append a float4 to a StringInfo buffer 296 | * 297 | * The point of this routine is to localize knowledge of the external binary 298 | * representation of float4, which is a component of several datatypes. 299 | * 300 | * We currently assume that float4 should be byte-swapped in the same way 301 | * as int4. This rule is not perfect but it gives us portability across 302 | * most IEEE-float-using architectures. 303 | * -------------------------------- 304 | */ 305 | void 306 | pq_sendfloat4(StringInfo buf, float4 f) 307 | { 308 | union 309 | { 310 | float4 f; 311 | uint32 i; 312 | } swap; 313 | 314 | swap.f = f; 315 | swap.i = htonl(swap.i); 316 | 317 | appendBinaryStringInfo(buf, (char *) &swap.i, 4); 318 | } 319 | 320 | /* -------------------------------- 321 | * pq_sendfloat8 - append a float8 to a StringInfo buffer 322 | * 323 | * The point of this routine is to localize knowledge of the external binary 324 | * representation of float8, which is a component of several datatypes. 325 | * 326 | * We currently assume that float8 should be byte-swapped in the same way 327 | * as int8. This rule is not perfect but it gives us portability across 328 | * most IEEE-float-using architectures. 329 | * -------------------------------- 330 | */ 331 | void 332 | pq_sendfloat8(StringInfo buf, float8 f) 333 | { 334 | union 335 | { 336 | float8 f; 337 | int64 i; 338 | } swap; 339 | 340 | swap.f = f; 341 | pq_sendint64(buf, swap.i); 342 | } 343 | 344 | /* -------------------------------- 345 | * pq_endmessage - send the completed message to the frontend 346 | * 347 | * The data buffer is pfree()d, but if the StringInfo was allocated with 348 | * makeStringInfo then the caller must still pfree it. 349 | * -------------------------------- 350 | */ 351 | /* 352 | void 353 | pq_endmessage(StringInfo buf) 354 | { 355 | 356 | (void) pq_putmessage(buf->cursor, buf->data, buf->len); 357 | 358 | pfree(buf->data); 359 | buf->data = NULL; 360 | } 361 | */ 362 | 363 | /* -------------------------------- 364 | * pq_begintypsend - initialize for constructing a bytea result 365 | * -------------------------------- 366 | */ 367 | void 368 | pq_begintypsend(StringInfo buf) 369 | { 370 | initStringInfo(buf); 371 | /* Reserve four bytes for the bytea length word */ 372 | appendStringInfoCharMacro(buf, '\0'); 373 | appendStringInfoCharMacro(buf, '\0'); 374 | appendStringInfoCharMacro(buf, '\0'); 375 | appendStringInfoCharMacro(buf, '\0'); 376 | } 377 | 378 | /* -------------------------------- 379 | * pq_endtypsend - finish constructing a bytea result 380 | * 381 | * The data buffer is returned as the palloc'd bytea value. (We expect 382 | * that it will be suitably aligned for this because it has been palloc'd.) 383 | * We assume the StringInfoData is just a local variable in the caller and 384 | * need not be pfree'd. 385 | * -------------------------------- 386 | */ 387 | /* 388 | bytea * 389 | pq_endtypsend(StringInfo buf) 390 | { 391 | bytea *result = (bytea *) buf->data; 392 | 393 | Assert(buf->len >= VARHDRSZ); 394 | SET_VARSIZE(result, buf->len); 395 | 396 | return result; 397 | } 398 | */ 399 | 400 | /* -------------------------------- 401 | * pq_puttextmessage - generate a character set-converted message in one step 402 | * 403 | * This is the same as the pqcomm.c routine pq_putmessage, except that 404 | * the message body is a null-terminated string to which encoding 405 | * conversion applies. 406 | * -------------------------------- 407 | */ 408 | /* 409 | void 410 | pq_puttextmessage(char msgtype, const char *str) 411 | { 412 | int slen = strlen(str); 413 | char *p; 414 | 415 | p = pg_server_to_client(str, slen); 416 | if (p != str) 417 | { 418 | (void) pq_putmessage(msgtype, p, strlen(p) + 1); 419 | pfree(p); 420 | return; 421 | } 422 | (void) pq_putmessage(msgtype, str, slen + 1); 423 | } 424 | */ 425 | 426 | /* -------------------------------- 427 | * pq_putemptymessage - convenience routine for message with empty body 428 | * -------------------------------- 429 | */ 430 | /* 431 | void 432 | pq_putemptymessage(char msgtype) 433 | { 434 | (void) pq_putmessage(msgtype, NULL, 0); 435 | } 436 | */ 437 | 438 | /* -------------------------------- 439 | * pq_getmsgbyte - get a raw byte from a message buffer 440 | * -------------------------------- 441 | */ 442 | int 443 | pq_getmsgbyte(StringInfo msg) 444 | { 445 | if (msg->cursor >= msg->len) 446 | { 447 | fprintf(stderr,"no data left in message"); 448 | return -1; 449 | } 450 | return (unsigned char) msg->data[msg->cursor++]; 451 | } 452 | 453 | /* -------------------------------- 454 | * pq_getmsgint - get a binary integer from a message buffer 455 | * 456 | * Values are treated as unsigned. 457 | * -------------------------------- 458 | */ 459 | unsigned int 460 | pq_getmsgint(StringInfo msg, int b) 461 | { 462 | unsigned int result; 463 | unsigned char n8; 464 | uint16 n16; 465 | uint32 n32; 466 | 467 | switch (b) 468 | { 469 | case 1: 470 | pq_copymsgbytes(msg, (char *) &n8, 1); 471 | result = n8; 472 | break; 473 | case 2: 474 | pq_copymsgbytes(msg, (char *) &n16, 2); 475 | result = ntohs(n16); 476 | break; 477 | case 4: 478 | pq_copymsgbytes(msg, (char *) &n32, 4); 479 | result = ntohl(n32); 480 | break; 481 | default: 482 | fprintf(stderr, "unsupported integer size %d", b); 483 | result = 0; /* keep compiler quiet */ 484 | break; 485 | } 486 | return result; 487 | } 488 | 489 | /* -------------------------------- 490 | * pq_getmsgint64 - get a binary 8-byte int from a message buffer 491 | * 492 | * It is tempting to merge this with pq_getmsgint, but we'd have to make the 493 | * result int64 for all data widths --- that could be a big performance 494 | * hit on machines where int64 isn't efficient. 495 | * -------------------------------- 496 | */ 497 | int64 498 | pq_getmsgint64(StringInfo msg) 499 | { 500 | int64 result; 501 | uint32 h32; 502 | uint32 l32; 503 | 504 | pq_copymsgbytes(msg, (char *) &h32, 4); 505 | pq_copymsgbytes(msg, (char *) &l32, 4); 506 | h32 = ntohl(h32); 507 | l32 = ntohl(l32); 508 | 509 | result = h32; 510 | result <<= 32; 511 | result |= l32; 512 | 513 | return result; 514 | } 515 | 516 | /* -------------------------------- 517 | * pq_getmsgfloat4 - get a float4 from a message buffer 518 | * 519 | * See notes for pq_sendfloat4. 520 | * -------------------------------- 521 | */ 522 | float4 523 | pq_getmsgfloat4(StringInfo msg) 524 | { 525 | union 526 | { 527 | float4 f; 528 | uint32 i; 529 | } swap; 530 | 531 | swap.i = pq_getmsgint(msg, 4); 532 | return swap.f; 533 | } 534 | 535 | /* -------------------------------- 536 | * pq_getmsgfloat8 - get a float8 from a message buffer 537 | * 538 | * See notes for pq_sendfloat8. 539 | * -------------------------------- 540 | */ 541 | float8 542 | pq_getmsgfloat8(StringInfo msg) 543 | { 544 | union 545 | { 546 | float8 f; 547 | int64 i; 548 | } swap; 549 | 550 | swap.i = pq_getmsgint64(msg); 551 | return swap.f; 552 | } 553 | 554 | /* -------------------------------- 555 | * pq_getmsgbytes - get raw data from a message buffer 556 | * 557 | * Returns a pointer directly into the message buffer; note this 558 | * may not have any particular alignment. 559 | * -------------------------------- 560 | */ 561 | const char * 562 | pq_getmsgbytes(StringInfo msg, int datalen) 563 | { 564 | const char *result; 565 | 566 | if (datalen < 0 || datalen > (msg->len - msg->cursor)) 567 | { 568 | fprintf(stderr,"insufficient data left in message"); 569 | return NULL; 570 | } 571 | result = &msg->data[msg->cursor]; 572 | msg->cursor += datalen; 573 | return result; 574 | } 575 | 576 | /* -------------------------------- 577 | * pq_copymsgbytes - copy raw data from a message buffer 578 | * 579 | * Same as above, except data is copied to caller's buffer. 580 | * -------------------------------- 581 | */ 582 | void 583 | pq_copymsgbytes(StringInfo msg, char *buf, int datalen) 584 | { 585 | if (datalen < 0 || datalen > (msg->len - msg->cursor)) 586 | { 587 | fprintf(stderr, "insufficient data left in message"); 588 | return; 589 | } 590 | memcpy(buf, &msg->data[msg->cursor], datalen); 591 | msg->cursor += datalen; 592 | } 593 | 594 | /* -------------------------------- 595 | * pq_getmsgtext - get a counted text string (with conversion) 596 | * 597 | * Always returns a pointer to a freshly palloc'd result. 598 | * The result has a trailing null, *and* we return its strlen in *nbytes. 599 | * -------------------------------- 600 | */ 601 | /* 602 | char * 603 | pq_getmsgtext(StringInfo msg, int rawbytes, int *nbytes) 604 | { 605 | char *str; 606 | char *p; 607 | 608 | if (rawbytes < 0 || rawbytes > (msg->len - msg->cursor)) 609 | ereport(ERROR, 610 | (errcode(ERRCODE_PROTOCOL_VIOLATION), 611 | errmsg("insufficient data left in message"))); 612 | str = &msg->data[msg->cursor]; 613 | msg->cursor += rawbytes; 614 | 615 | p = pg_client_to_server(str, rawbytes); 616 | if (p != str) 617 | *nbytes = strlen(p); 618 | else 619 | { 620 | p = (char *) palloc(rawbytes + 1); 621 | memcpy(p, str, rawbytes); 622 | p[rawbytes] = '\0'; 623 | *nbytes = rawbytes; 624 | } 625 | return p; 626 | } 627 | */ 628 | 629 | /* -------------------------------- 630 | * pq_getmsgstring - get a null-terminated text string (with conversion) 631 | * 632 | * May return a pointer directly into the message buffer, or a pointer 633 | * to a palloc'd conversion result. 634 | * -------------------------------- 635 | */ 636 | /* 637 | const char * 638 | pq_getmsgstring(StringInfo msg) 639 | { 640 | char *str; 641 | int slen; 642 | 643 | str = &msg->data[msg->cursor]; 644 | 645 | slen = strlen(str); 646 | if (msg->cursor + slen >= msg->len) 647 | ereport(ERROR, 648 | (errcode(ERRCODE_PROTOCOL_VIOLATION), 649 | errmsg("invalid string in message"))); 650 | msg->cursor += slen + 1; 651 | 652 | return pg_client_to_server(str, slen); 653 | } 654 | */ 655 | 656 | /* -------------------------------- 657 | * pq_getmsgend - verify message fully consumed 658 | * -------------------------------- 659 | */ 660 | void 661 | pq_getmsgend(StringInfo msg) 662 | { 663 | if (msg->cursor != msg->len) 664 | { 665 | fprintf(stderr, "invalid message format"); 666 | } 667 | } 668 | 669 | 670 | -------------------------------------------------------------------------------- /dbsync/readcfg.cpp: -------------------------------------------------------------------------------- 1 | 2 | #include "postgres_fe.h" 3 | #include 4 | #include 5 | #include 6 | #include 7 | #include 8 | #include 9 | #include 10 | 11 | #include 12 | #include 13 | 14 | #include 15 | #include 16 | 17 | #include "readcfg.h" 18 | #include 19 | 20 | using namespace std; 21 | 22 | using std::string; 23 | 24 | Config::Config(const string &filename) : _conf(NULL) { 25 | if (filename != "") this->_conf = ini_load(filename.c_str()); 26 | if (this->_conf == NULL) { 27 | //fprintf(stderr,"Failed to load config file\n"); 28 | cout<<"Failed to load config file\n"<_conf) ini_free(this->_conf); 34 | } 35 | 36 | string Config::Get(const string &sec, const string &key, 37 | const string &defaultvalue) { 38 | string ret = defaultvalue; 39 | if ((key == "") || (sec == "")) return ret; 40 | 41 | if (this->_conf) { 42 | const char *tmp = ini_get(this->_conf, sec.c_str(), key.c_str()); 43 | if (tmp) ret = tmp; 44 | } 45 | return ret; 46 | } 47 | 48 | bool Config::Scan(const string &sec, const string &key, const char *scanfmt, 49 | void *dst) { 50 | if ((key == "") || (sec == "")) return false; 51 | 52 | if (this->_conf) { 53 | return ini_sget(this->_conf, sec.c_str(), key.c_str(), scanfmt, dst); 54 | } 55 | return false; 56 | } 57 | 58 | bool to_bool(std::string str) { 59 | std::transform(str.begin(), str.end(), str.begin(), ::tolower); 60 | if ((str == "yes") || (str == "true") || (str == "y") || (str == "t") || 61 | (str == "1")) { 62 | return true; 63 | } else { 64 | return false; 65 | } 66 | } 67 | 68 | void find_replace(string &str, const string &find, const string &replace) { 69 | if (find.empty()) return; 70 | 71 | size_t pos = 0; 72 | 73 | while ((pos = str.find(find, pos)) != string::npos) { 74 | str.replace(pos, find.length(), replace); 75 | pos += replace.length(); 76 | } 77 | } 78 | 79 | void * 80 | init_config(char *cfgpath) 81 | { 82 | Config* s3cfg = NULL; 83 | 84 | s3cfg = new Config(cfgpath); 85 | if (!s3cfg || !s3cfg->Handle()) 86 | { 87 | if (s3cfg) 88 | { 89 | delete s3cfg; 90 | s3cfg = NULL; 91 | } 92 | return NULL; 93 | } 94 | 95 | return s3cfg; 96 | } 97 | 98 | int 99 | get_config(void *cfg, char *sec, char* key, char **value) 100 | { 101 | Config* s3cfg = (Config *)cfg; 102 | string ssec; 103 | string skey; 104 | string rc; 105 | 106 | if (sec == NULL || key == NULL) 107 | return 1; 108 | 109 | ssec = string(sec); 110 | skey = string(key); 111 | rc = s3cfg->Get(ssec, skey, ""); 112 | if (rc == "") 113 | return 1; 114 | 115 | *value = strdup(rc.c_str()); 116 | 117 | return 0; 118 | } 119 | 120 | -------------------------------------------------------------------------------- /dbsync/readcfg.h: -------------------------------------------------------------------------------- 1 | #ifndef _READCFG_H_ 2 | #define _READCFG_H_ 3 | 4 | #include "postgres_fe.h" 5 | #include "c.h" 6 | 7 | #include 8 | #include 9 | 10 | #include 11 | #include 12 | 13 | #include "ini.h" 14 | 15 | using std::string; 16 | 17 | #define uint64_t uint64 18 | 19 | class Config { 20 | public: 21 | Config(const string& filename); 22 | ~Config(); 23 | string Get(const string& sec, const string& key, 24 | const string& defaultvalue); 25 | bool Scan(const string& sec, const string& key, const char* scanfmt, 26 | void* dst); 27 | void* Handle() { return (void*)this->_conf; }; 28 | 29 | private: 30 | ini_t* _conf; 31 | }; 32 | 33 | bool to_bool(std::string str); 34 | 35 | void find_replace(string& str, const string& find, const string& replace); 36 | 37 | #endif // _UTILFUNCTIONS_ 38 | -------------------------------------------------------------------------------- /dbsync/stringinfo.c: -------------------------------------------------------------------------------- 1 | 2 | #include "postgres_fe.h" 3 | #include "lib/stringinfo.h" 4 | #include "misc.h" 5 | 6 | 7 | /* 8 | * makeStringInfo 9 | * 10 | * Create an empty 'StringInfoData' & return a pointer to it. 11 | */ 12 | StringInfo 13 | makeStringInfo(void) 14 | { 15 | StringInfo res; 16 | 17 | res = (StringInfo) palloc(sizeof(StringInfoData)); 18 | 19 | initStringInfo(res); 20 | 21 | return res; 22 | } 23 | 24 | /* 25 | * initStringInfo 26 | * 27 | * Initialize a StringInfoData struct (with previously undefined contents) 28 | * to describe an empty string. 29 | */ 30 | void 31 | initStringInfo(StringInfo str) 32 | { 33 | int size = 1024; /* initial default buffer size */ 34 | 35 | str->data = (char *) palloc(size); 36 | str->maxlen = size; 37 | resetStringInfo(str); 38 | } 39 | 40 | /* 41 | * resetStringInfo 42 | * 43 | * Reset the StringInfo: the data buffer remains valid, but its 44 | * previous content, if any, is cleared. 45 | */ 46 | void 47 | resetStringInfo(StringInfo str) 48 | { 49 | str->data[0] = '\0'; 50 | str->len = 0; 51 | str->cursor = 0; 52 | } 53 | 54 | /* 55 | * appendStringInfo 56 | * 57 | * Format text data under the control of fmt (an sprintf-style format string) 58 | * and append it to whatever is already in str. More space is allocated 59 | * to str if necessary. This is sort of like a combination of sprintf and 60 | * strcat. 61 | */ 62 | void 63 | appendStringInfo(StringInfo str, const char *fmt,...) 64 | { 65 | for (;;) 66 | { 67 | va_list args; 68 | int needed; 69 | 70 | /* Try to format the data. */ 71 | va_start(args, fmt); 72 | needed = appendStringInfoVA(str, fmt, args); 73 | va_end(args); 74 | 75 | if (needed == 0) 76 | break; /* success */ 77 | 78 | /* Increase the buffer size and try again. */ 79 | enlargeStringInfo(str, needed); 80 | } 81 | } 82 | 83 | /* 84 | * appendStringInfoVA 85 | * 86 | * Attempt to format text data under the control of fmt (an sprintf-style 87 | * format string) and append it to whatever is already in str. If successful 88 | * return zero; if not (because there's not enough space), return an estimate 89 | * of the space needed, without modifying str. Typically the caller should 90 | * pass the return value to enlargeStringInfo() before trying again; see 91 | * appendStringInfo for standard usage pattern. 92 | * 93 | * XXX This API is ugly, but there seems no alternative given the C spec's 94 | * restrictions on what can portably be done with va_list arguments: you have 95 | * to redo va_start before you can rescan the argument list, and we can't do 96 | * that from here. 97 | */ 98 | int 99 | appendStringInfoVA(StringInfo str, const char *fmt, va_list args) 100 | { 101 | int avail; 102 | size_t nprinted; 103 | 104 | Assert(str != NULL); 105 | 106 | /* 107 | * If there's hardly any space, don't bother trying, just fail to make the 108 | * caller enlarge the buffer first. We have to guess at how much to 109 | * enlarge, since we're skipping the formatting work. 110 | */ 111 | avail = str->maxlen - str->len; 112 | if (avail < 16) 113 | return 32; 114 | 115 | nprinted = pvsnprintf(str->data + str->len, (size_t) avail, fmt, args); 116 | 117 | if (nprinted < (size_t) avail) 118 | { 119 | /* Success. Note nprinted does not include trailing null. */ 120 | str->len += (int) nprinted; 121 | return 0; 122 | } 123 | 124 | /* Restore the trailing null so that str is unmodified. */ 125 | str->data[str->len] = '\0'; 126 | 127 | /* 128 | * Return pvsnprintf's estimate of the space needed. (Although this is 129 | * given as a size_t, we know it will fit in int because it's not more 130 | * than MaxAllocSize.) 131 | */ 132 | return (int) nprinted; 133 | } 134 | 135 | /* 136 | * appendStringInfoString 137 | * 138 | * Append a null-terminated string to str. 139 | * Like appendStringInfo(str, "%s", s) but faster. 140 | */ 141 | void 142 | appendStringInfoString(StringInfo str, const char *s) 143 | { 144 | appendBinaryStringInfo(str, s, strlen(s)); 145 | } 146 | 147 | /* 148 | * appendStringInfoChar 149 | * 150 | * Append a single byte to str. 151 | * Like appendStringInfo(str, "%c", ch) but much faster. 152 | */ 153 | void 154 | appendStringInfoChar(StringInfo str, char ch) 155 | { 156 | /* Make more room if needed */ 157 | if (str->len + 1 >= str->maxlen) 158 | enlargeStringInfo(str, 1); 159 | 160 | /* OK, append the character */ 161 | str->data[str->len] = ch; 162 | str->len++; 163 | str->data[str->len] = '\0'; 164 | } 165 | 166 | /* 167 | * appendStringInfoSpaces 168 | * 169 | * Append the specified number of spaces to a buffer. 170 | */ 171 | void 172 | appendStringInfoSpaces(StringInfo str, int count) 173 | { 174 | if (count > 0) 175 | { 176 | /* Make more room if needed */ 177 | enlargeStringInfo(str, count); 178 | 179 | /* OK, append the spaces */ 180 | while (--count >= 0) 181 | str->data[str->len++] = ' '; 182 | str->data[str->len] = '\0'; 183 | } 184 | } 185 | 186 | /* 187 | * appendBinaryStringInfo 188 | * 189 | * Append arbitrary binary data to a StringInfo, allocating more space 190 | * if necessary. 191 | */ 192 | void 193 | appendBinaryStringInfo(StringInfo str, const char *data, int datalen) 194 | { 195 | Assert(str != NULL); 196 | 197 | /* Make more room if needed */ 198 | enlargeStringInfo(str, datalen); 199 | 200 | /* OK, append the data */ 201 | memcpy(str->data + str->len, data, datalen); 202 | str->len += datalen; 203 | 204 | /* 205 | * Keep a trailing null in place, even though it's probably useless for 206 | * binary data. (Some callers are dealing with text but call this because 207 | * their input isn't null-terminated.) 208 | */ 209 | str->data[str->len] = '\0'; 210 | } 211 | 212 | /* 213 | * enlargeStringInfo 214 | * 215 | * Make sure there is enough space for 'needed' more bytes 216 | * ('needed' does not include the terminating null). 217 | * 218 | * External callers usually need not concern themselves with this, since 219 | * all stringinfo.c routines do it automatically. However, if a caller 220 | * knows that a StringInfo will eventually become X bytes large, it 221 | * can save some palloc overhead by enlarging the buffer before starting 222 | * to store data in it. 223 | * 224 | * NB: because we use repalloc() to enlarge the buffer, the string buffer 225 | * will remain allocated in the same memory context that was current when 226 | * initStringInfo was called, even if another context is now current. 227 | * This is the desired and indeed critical behavior! 228 | */ 229 | void 230 | enlargeStringInfo(StringInfo str, int needed) 231 | { 232 | int newlen; 233 | 234 | /* 235 | * Guard against out-of-range "needed" values. Without this, we can get 236 | * an overflow or infinite loop in the following. 237 | */ 238 | if (needed < 0) /* should not happen */ 239 | { 240 | fprintf(stderr, "invalid string enlargement request size: %d", needed); 241 | return; 242 | } 243 | if (((Size) needed) >= (MaxAllocSize - (Size) str->len)) 244 | { 245 | fprintf(stderr,"Cannot enlarge string buffer containing %d bytes by %d more bytes.", 246 | str->len, needed); 247 | return; 248 | } 249 | 250 | needed += str->len + 1; /* total space required now */ 251 | 252 | /* Because of the above test, we now have needed <= MaxAllocSize */ 253 | 254 | if (needed <= str->maxlen) 255 | return; /* got enough space already */ 256 | 257 | /* 258 | * We don't want to allocate just a little more space with each append; 259 | * for efficiency, double the buffer size each time it overflows. 260 | * Actually, we might need to more than double it if 'needed' is big... 261 | */ 262 | newlen = 2 * str->maxlen; 263 | while (needed > newlen) 264 | newlen = 2 * newlen; 265 | 266 | /* 267 | * Clamp to MaxAllocSize in case we went past it. Note we are assuming 268 | * here that MaxAllocSize <= INT_MAX/2, else the above loop could 269 | * overflow. We will still have newlen >= needed. 270 | */ 271 | if (newlen > (int) MaxAllocSize) 272 | newlen = (int) MaxAllocSize; 273 | 274 | str->data = (char *) repalloc(str->data, newlen); 275 | 276 | str->maxlen = newlen; 277 | } 278 | -------------------------------------------------------------------------------- /dbsync/test/decode_test.sql: -------------------------------------------------------------------------------- 1 | 2 | create schema test_case; 3 | set search_path=test_case; 4 | 5 | -- no index 6 | create table a(a int ,b text, c timestamptz); 7 | 8 | insert into a values(1,'test','1999-01-08 04:05:06'); 9 | insert into a values(2,'test','1999-01-08 04:05:06'); 10 | insert into a values(3,'test','1999-01-08 04:05:06'); 11 | 12 | update a set b = 'test1'; 13 | update a set b = 'test2' where a = 3; 14 | 15 | delete from a where a = 2; 16 | delete from a; 17 | 18 | -- primary key 19 | create table b(a int primary key ,b text, c timestamptz); 20 | 21 | insert into b values(1,'test','1999-01-08 04:05:06'); 22 | insert into b values(2,'test','1999-01-08 04:05:06'); 23 | insert into b values(3,'test','1999-01-08 04:05:06'); 24 | 25 | update b set b = 'test1'; 26 | update b set a = 5 where a = 1; 27 | update b set b = 'test2' where a = 3; 28 | update b set c = '1999-01-08 04:05:06' where a = 5; 29 | update b set a = 6, c = '1999-01-08 04:05:06' where a = 5; 30 | update b set a = 5, b = 't',c = '1999-01-08 04:05:06' where a = 6; 31 | 32 | delete from b where a = 2; 33 | delete from b; 34 | 35 | -- mprimary key 36 | create table c(a int ,b text, c timestamptz, d bigint, primary key(a,d)); 37 | 38 | insert into c values(1,'test','1999-01-08 04:05:06',3); 39 | insert into c values(2,'test','1999-01-08 04:05:06',2); 40 | insert into c values(3,'test','1999-01-08 04:05:06',1); 41 | 42 | update c set b = 'test1'; 43 | update c set a = 5 where a = 1; 44 | update c set b = null where a = 3; 45 | delete from c where a = 2; 46 | 47 | -- REPLICA index 48 | create table d(a int ,b text, c timestamptz, d bigint); 49 | CREATE UNIQUE INDEX idx_d_a_d ON d(a,d); 50 | alter table d ALTER COLUMN a set not null; 51 | alter table d ALTER COLUMN d set not null; 52 | alter table d REPLICA IDENTITY USING INDEX idx_d_a_d; 53 | 54 | insert into d values(1,'test','1999-01-08 04:05:06',3); 55 | insert into d values(2,'test','1999-01-08 04:05:06',2); 56 | insert into d values(3,'test','1999-01-08 04:05:06',1); 57 | 58 | update d set b = 'test1'; 59 | update d set a = 5 where a = 1; 60 | update d set b = 'test2' where a = 3; 61 | update d set a = 5, b = 't',c = '1999-01-08 04:05:06' where a = 3; 62 | delete from d; 63 | 64 | -- full data 65 | create table e(a int ,b text, c timestamptz, d bigint); 66 | alter table e REPLICA IDENTITY FULL; 67 | 68 | insert into e values(1,'test','1999-01-08 04:05:06',3); 69 | insert into e values(2,'test','1999-01-08 04:05:06',2); 70 | insert into e values(3,'test','1999-01-08 04:05:06',1); 71 | 72 | update e set b = 'test1'; 73 | update e set a = 5 where a = 1; 74 | update e set b = 'test2' where a = 3; 75 | update e set a = 5, b = 't',c = '1999-01-08 04:05:06' where a = 3; 76 | 77 | delete from e; 78 | 79 | 80 | -- full data and primary key 81 | create table f(a int primary key,b text, c timestamptz, d bigint); 82 | alter table f REPLICA IDENTITY FULL; 83 | 84 | insert into f values(1,'test','1999-01-08 04:05:06',3); 85 | insert into f values(2,'test','1999-01-08 04:05:06',2); 86 | insert into f values(3,'test','1999-01-08 04:05:06',1); 87 | 88 | update f set b = 'test1'; 89 | update f set a = 5 where a = 1; 90 | 91 | alter table f REPLICA IDENTITY DEFAULT; 92 | update f set a = 7 where a = 2; 93 | 94 | update f set b = 'test2' where a = 3; 95 | update f set a = 6, b = 't',c = '1999-01-08 04:05:06' where a = 3; 96 | 97 | delete from f; 98 | 99 | -- data type 100 | create table test_data_type_1(a smallint,b integer,c bigint,d decimal,e numeric); 101 | insert into test_data_type_1 values(-32768, 2147483647, 9223372036854775807, 111.111, 111.111); 102 | 103 | create table test_data_type_2(a real,b double precision,c smallserial,d serial,e bigserial); 104 | insert into test_data_type_2 values(111.111, 111.111, 32767, 2147483647, 9223372036854775807); 105 | 106 | create table test_data_type_3(a money,b character varying(20),c character(20),d text,e char(20)); 107 | insert into test_data_type_3 values('12.34', '12.34', '12.34', '12.34', '12.34'); 108 | 109 | create table test_data_type_4(a bytea,b bytea,c bytea,d bytea,e bytea); 110 | insert into test_data_type_4 values('\\xDEADBEEF', '\\000', '0', '\\134', '\\176'); 111 | 112 | create table test_data_type_5(a timestamp without time zone ,b timestamp with time zone,c timestamp,d time,e time with time zone); 113 | insert into test_data_type_5 values('1999-01-08 04:05:06', '1999-01-08 04:05:06 +8:00', '1999-01-08 04:05:06 -8:00', '1999-01-08 04:05:06 -8:00', '1999-01-08 04:05:06 -8:00'); 114 | 115 | CREATE TYPE mood AS ENUM ('sad', 'ok', 'happy'); 116 | create table test_data_type_6(a boolean,b mood,c point,d line,e lseg); 117 | insert into test_data_type_6 values(TRUE, 'happy', '(1,1)', '{1,2,1}', '[(1,2),(2,1)]'); 118 | 119 | create table test_data_type_7(a path,b path,c polygon,d circle,e circle); 120 | insert into test_data_type_7 values('[(1,3),(2,2)]', '((1,3),(2,2))', '((1,3),(2,2))', '<(2,2),2>', '((2,3),1)'); 121 | 122 | create table test_data_type_8(a cidr,b cidr,c inet,d macaddr,e macaddr); 123 | insert into test_data_type_8 values('192.168.100.128/25', '2001:4f8:3:ba:2e0:81ff:fe22:d1f1/128', '1.2.3.4', '08-00-2b-01-02-03', '08:00:2b:01:02:03'); 124 | 125 | CREATE TABLE test_9 (a BIT(3), b BIT VARYING(5)); 126 | INSERT INTO test_9 VALUES (B'101', B'00'); 127 | INSERT INTO test_9 VALUES (B'10'::bit(3), B'101'); 128 | 129 | CREATE TABLE test_10 (a tsvector, b tsvector,c tsquery, d tsquery); 130 | INSERT INTO test_10 VALUES ('a fat cat sat on a mat and ate a fat rat', 'a:1 fat:2 cat:3 sat:4 on:5 a:6 mat:7 and:8 ate:9 a:10 fat:11 rat:12','fat & rat','Fat:ab & Cats'); 131 | 132 | create extension "uuid-ossp"; 133 | CREATE TABLE test_11 (a uuid, b uuid,c uuid, d uuid); 134 | INSERT INTO test_11 VALUES ('25285134-7314-11e5-8e45-d89d672b3560', 'c12a3d5f-53bb-4223-9fca-0af78b4d269f', 'cf16fe52-3365-3a1f-8572-288d8d2aaa46', '252852d8-7314-11e5-8e45-2f1f0837ccca'); 135 | 136 | CREATE TABLE test_12 (a xml, b xml,c xml); 137 | INSERT INTO test_12 VALUES (xml 'bar', XMLPARSE (DOCUMENT 'Manual...'), XMLPARSE (CONTENT 'abcbarfoo')); 138 | 139 | CREATE TABLE test_13 (a xml, b xml,c xml); 140 | INSERT INTO test_13 VALUES ('{"reading": 1.230e-5}', 141 | '[1, 2, "foo", null]', 142 | '{"bar": "baz", "balance": 7.77, "active":false}'); 143 | 144 | CREATE TABLE sal_emp_14 ( 145 | name text, 146 | pay_by_quarter integer[], 147 | schedule text[][] 148 | ); 149 | 150 | INSERT INTO sal_emp_14 151 | VALUES ('Bill', 152 | '{10000, 10000, 10000, 10000}', 153 | '{{"meeting", "lunch"}, {"training", "presentation"}}'); 154 | 155 | INSERT INTO sal_emp_14 156 | VALUES ('Carol', 157 | '{20000, 25000, 25000, 25000}', 158 | '{{"breakfast", "consulting"}, {"meeting", "lunch"}}'); 159 | 160 | INSERT INTO sal_emp_14 161 | VALUES ('Carol', 162 | ARRAY[20000, 25000, 25000, 25000], 163 | ARRAY[['breakfast', 'consulting'], ['meeting', 'lunch']]); 164 | 165 | CREATE TYPE complex AS ( 166 | r double precision, 167 | i double precision 168 | ); 169 | 170 | CREATE TYPE inventory_item AS ( 171 | name text, 172 | supplier_id integer, 173 | price numeric 174 | ); 175 | 176 | CREATE TABLE on_hand_15 ( 177 | item inventory_item, 178 | count integer 179 | ); 180 | 181 | INSERT INTO on_hand_15 VALUES (ROW('fuzzy dice', 42, 1.99), 1000); 182 | 183 | CREATE TABLE reservation_16 (room int, during tsrange); 184 | INSERT INTO reservation_16 VALUES 185 | (1108, '[2010-01-01 14:30, 2010-01-01 15:30)'); 186 | 187 | CREATE TYPE floatrange AS RANGE ( 188 | subtype = float8, 189 | subtype_diff = float8mi 190 | ); 191 | 192 | create table t_range_16(a floatrange); 193 | insert into t_range_16 values('[1.234, 5.678]'); 194 | 195 | create extension hstore; 196 | create table hstore_test_17(item_id serial, data hstore); 197 | INSERT INTO hstore_test_17 (data) VALUES ('"key1"=>"value1", "key2"=>"value2", "key3"=>"value3"'); 198 | UPDATE hstore_test_17 SET data = delete(data, 'key2'); 199 | UPDATE hstore_test_17 SET data = data || '"key4"=>"some value"'::hstore; 200 | 201 | CREATE EXTENSION postgis; 202 | CREATE EXTENSION postgis_topology; 203 | CREATE EXTENSION fuzzystrmatch; 204 | CREATE EXTENSION postgis_tiger_geocoder; 205 | 206 | create table test_18 (myID int4, pt geometry, myName varchar ); 207 | insert into test_18 values (1, 'POINT(0 0)', 'beijing' ); 208 | insert into test_18 values (2, 'MULTIPOINT(1 1, 3 4, -1 3)', 'shanghai' ); 209 | insert into test_18 values (3, 'LINESTRING(1 1, 2 2, 3 4)', 'tianjin' ); 210 | insert into test_18 values (3, 'POLYGON((0 0, 0 1, 1 1, 1 0, 0 0))', 'tianjin' ); 211 | insert into test_18 values (3, 'MULTIPOLYGON(((0 0,4 0,4 4,0 4,0 0),(1 1,2 1,2 2,1 2,1 1)), ((-1 -1,-1 -2,-2 -2,-2 -1,-1 -1)))', 'tianjin' ); 212 | insert into test_18 values (3, 'MULTILINESTRING((1 1, 2 2, 3 4),(2 2, 3 3, 4 5))', 'tianjin' ); 213 | 214 | insert into test_18 values (3, '01060000000200000001030000000200000005000000000000000000000000000000000000000000000000001040000000000000000000000000000010400000000000001040000000000000000000000000000010400000000000000000000000000000000005000000000000000000F03F000000000000F03F0000000000000040000000000000F03F00000000000000400000000000000040000000000000F03F0000000000000040000000000000F03F000000000000F03F01030000000100000005000000000000000000F0BF000000000000F0BF000000000000F0BF00000000000000C000000000000000C000000000000000C000000000000000C0000000000000F0BF000000000000F0BF000000000000F0BF', 'tianjin' ); 215 | 216 | -- m sql in a tran 217 | create table msql(a int primary key,b text, c timestamptz); 218 | 219 | begin; 220 | insert into msql values(1,'test','1999-01-08 04:05:06'); 221 | insert into msql values(2,'test','1999-01-08 04:05:06'); 222 | insert into msql values(3,'test','1999-01-08 04:05:06'); 223 | update msql set b = 'test' where a = 1; 224 | delete from msql where a = 3; 225 | commit; 226 | 227 | -- alter table 228 | create table msql_1(a int primary key,b text, c timestamptz); 229 | 230 | insert into msql_1 values(1,'test','1999-01-08 04:05:06'); 231 | alter table msql_1 add COLUMN d int; 232 | insert into msql_1 values(2,'test','1999-01-08 04:05:06',1); 233 | alter table msql_1 drop COLUMN b; 234 | insert into msql_1 values(3,'1999-01-08 04:05:06',2); 235 | 236 | update msql_1 set c = '1999-01-08 04:05:07'; 237 | delete from msql_1; 238 | 239 | -- alter table in a tran 240 | create table msql_2(a int primary key,b text, c timestamptz); 241 | begin; 242 | insert into msql_2 values(1,'test','1999-01-08 04:05:06'); 243 | alter table msql_2 add COLUMN d int; 244 | insert into msql_2 values(2,'test','1999-01-08 04:05:06',1); 245 | alter table msql_2 drop COLUMN b; 246 | insert into msql_2 values(3,'1999-01-08 04:05:06',2); 247 | update msql_2 set c = '1999-01-08 04:05:07'; 248 | commit; 249 | 250 | 251 | -- alter table drop pk 252 | create table msql_3(a int primary key,b text, c timestamptz); 253 | begin; 254 | insert into msql_3 values(1,'test','1999-01-08 04:05:06'); 255 | insert into msql_3 values(5,'test','1999-01-08 04:05:06'); 256 | alter table msql_3 add COLUMN d int; 257 | insert into msql_3 values(2,'test','1999-01-08 04:05:06',1); 258 | alter table msql_3 drop COLUMN a; 259 | insert into msql_3 values('test','1999-01-08 04:05:06',2); 260 | delete from msql_3; 261 | commit; 262 | 263 | -- SERIAL 264 | CREATE TABLE seq_test 265 | ( 266 | id SERIAL primary key , 267 | name text 268 | ) ; 269 | 270 | insert into seq_test (name) values('test'); 271 | 272 | -- toast 273 | -- create table t_kenyon(id int,vname varchar(48),remark text); 274 | -- select oid,relname,reltoastrelid from pg_class where relname = 't_kenyon'; 275 | -- insert into t_kenyon select generate_series(1,2000),repeat('kenyon here'||'^_^',2),repeat('^_^ Kenyon is not God',500); 276 | -- insert into t_kenyon select generate_series(1,2),repeat('kenyon here'||'^_^',2),repeat('^_^ Kenyon is not God,Remark here!!',2000); 277 | -- insert into t_kenyon select generate_series(3,4),repeat('kenyon here'||'^_^',2),repeat('^_^ Kenyon is not God,Remark here!!',4000); 278 | -- insert into t_kenyon select generate_series(5,6),repeat('kenyon here'||'^_^',2),repeat('^_^ Kenyon is not God,Remark here!!',5500); 279 | -- insert into t_kenyon select generate_series(1,2),repeat('kenyon here'||'^_^',2),repeat('^_^ Kenyon is not God,Remark here!!',10000); 280 | -- insert into t_kenyon select generate_series(7,8),repeat('kenyon here'||'^_^',2),repeat('^_^ Kenyon is not God,Remark here!!',20000); 281 | 282 | -- utf8 283 | create table chinese_text(t text); 284 | insert into chinese_text values('微软 Surface Pro 4 中国开放预售 价格公布'); 285 | insert into chinese_text values('\'); 286 | insert into chinese_text values('\\'); 287 | insert into chinese_text values('\\\'); 288 | insert into chinese_text values('///'); 289 | insert into chinese_text values('//'); 290 | insert into chinese_text values('/'); 291 | insert into chinese_text values(''''); 292 | insert into chinese_text values('"''"'); 293 | 294 | -- bug extra_float_digits default 3 295 | create table tf(c1 float4, c2 float8 ,c3 numeric); 296 | insert into tf values (1.5555555555555555555555,1.5555555555555555555555,1.5555555555555555555555); 297 | 298 | drop schema test_case cascade; 299 | -------------------------------------------------------------------------------- /dbsync/utils.h: -------------------------------------------------------------------------------- 1 | 2 | 3 | #ifndef PG_UTILS_H 4 | #define PG_UTILS_H 5 | 6 | #include "postgres_fe.h" 7 | 8 | #include "lib/stringinfo.h" 9 | #include "lib/stringinfo.h" 10 | #include "common/fe_memutils.h" 11 | 12 | //#include "libpq-fe.h" 13 | 14 | #include "access/transam.h" 15 | #include "libpq/pqformat.h" 16 | #include "pqexpbuffer.h" 17 | 18 | 19 | 20 | extern void pq_copymsgbytes(StringInfo msg, char *buf, int datalen); 21 | extern void fe_sendint64(int64 i, char *buf); 22 | extern PGconn *GetConnection(char *connection_string); 23 | extern int64 feGetCurrentTimestamp(void); 24 | extern bool feTimestampDifferenceExceeds(int64 start_time, 25 | int64 stop_time, 26 | int msec); 27 | extern void feTimestampDifference(int64 start_time, int64 stop_time, 28 | long *secs, int *microsecs); 29 | extern int64 fe_recvint64(char *buf); 30 | extern int getopt(int nargc, char *const * nargv, const char *ostr); 31 | 32 | 33 | #endif 34 | 35 | 36 | -------------------------------------------------------------------------------- /doc/design.md: -------------------------------------------------------------------------------- 1 | # PostgreSQL 增量同步方案详细设计 2 | 3 | ## 一:方案目的 4 | 5 | 通过PG实例的增量数据同步方案,解决PG在数据迁移中需要进行全量同步,需要长时间停服务的问题. 6 | 通过该方案,打通PG和其他数据产品间的数据通道,做到PG和其他数据产品和异构数据库间的实时的同步. 7 | 8 | ## 二:增量同步方案的技术背景 9 | 10 | 该方案基于PostgreSQL 9.4版本的逻辑流复制技术. 11 | PG9.4 可以做到,利用逻辑流复制,把表的增量数据,以自定义格式的形式组织起来被客户端订阅. 12 | 自定义格式的增量数据是逻辑流复制的关键,通过PG内核给出了开放的接口(5个回调函数),可以把增量数据以任意的形式输出. 13 | PG的客户端根据需要按照服务端规定的格式解析,就能得到完整的增量数据. 14 | 15 | ## 三:订阅增量数据的订阅规则 16 | 17 | PG增量同步方案只能获取对应DB中表的变化信息. 18 | 一张表能够被逻辑流复制订阅,需要满足下列三个条件之一 19 | 1)这张表的流复制方式为全复制,即 REPLICA = FULL 20 | 可以通过DDL语句 alter table t REPLICA FULL 定制对应的表 21 | 这个选项使得对应表中每一行的变化数据被完整的记录到WAL中.所以带来了较多的IO负担. 22 | 2)这张表具有主键约束 23 | 有主键的表的变化信息会被记录到WAL中,相对于REPLICA FULL模式,old row没有变化的列将不会被记录到WAL中,对IO的影响相对FULL更小. 24 | 3)为该表指定一个非空的唯一索引,作为REPLICA INDEX. 25 | 可以通过DDL来定制相应的表 26 | CREATE UNIQUE INDEX idx ON d(a); 27 | alter table t ALTER COLUMN a set not null; 28 | alter table t REPLICA IDENTITY USING INDEX idx; 29 | 该模式记录到WAL中的数据和主键模式相当. 30 | 可以在pg_class的 relreplident 列中看到对应的表处于什么模式. 31 | PG的流复制的规则类似于MYSQL binlog模式的row模式,对于用户相对于更加灵活一些. 32 | 可以阅读相关文档,了解细节信息. 33 | 1 http://www.postgresql.org/docs/9.4/static/sql-altertable.html#SQL-CREATETABLE-REPLICA-IDENTITY 34 | 2 http://www.postgresql.org/docs/9.4/static/logicaldecoding.html 35 | 36 | ## 四:架构解析 37 | 38 | PG的增量方案架构上分为两个大的部分 39 | 1 服务器端 40 | 在被订阅的服务器端,嵌入一个用于 decode 的插件--ali_decoding 41 | 该插件实现了流复制的自定义数据流格式,用户使用pg_create_logical_replication_slot函数创建 logical replication 时指定插件 ali_decoding 和逻辑 slot 名. 42 | 在客户端使用 START_REPLICATION 命令开启对应的流复制传输后,服务器端会有一个backend 进程加载该插件(ali_decoding)开始向用户传输增量数据. 43 | 44 | 2 客户端 45 | START_REPLICATION 命令需要指定对应的逻辑 slot 名和位点. 46 | 所谓的位点是PG WAL日志的日志名和日志文件的偏移的组合. 47 | PG的位点概念对于物理流复制和逻辑流复制是统一的. 48 | 1) PG对应逻辑 slot 的位点信息保存在数据库端,具有断点续传特性. 49 | 2) 除非需要跳过某部分数据,则使用 0/0 去请求拉取对应 slot 的逻辑日志. 50 | 3) PG的位点不对应时间点. 51 | 52 | 对于客户端,在建立连接后,读到的增量信息以消息的形式出现,可以分为三类信息 53 | 1) 事务开始,begin 54 | 2) 事务结束,commit 55 | 3) 事务中单个表的变化信息,分为 56 | a) insert 57 | b) update 58 | c) delete 59 | 根据表 REPLICA 的状态不同,各类DML收到的信息略有变化. 60 | 61 | ## 五:编译和使用 62 | 63 | ### 1 编译 64 | 65 | 1 编译机上下载安装 PG94 或更高版本的二进制,或用源码安装. 66 | 2 使用软链接或别的方式,把对应版本的 pg_config 链接到公共目录 67 | 例: ln -s /u01/pgsql_20150924/bin/pg_config /usr/bin/ 或 68 | export PATH=/u01/pgsql_20150924/bin 69 | 3 下载服务器端和客户端代码,make;make install; 70 | 71 | ### 2 使用 72 | 73 | 1 使用SQL或 demo 中的API创建 logical slot 74 | 例: SELECT * FROM pg_create_logical_replication_slot('regression_slot', 'ali_decoding'); 75 | 76 | 2 可以用相应的 SQL 语句查看创建好的 slot 77 | 例: SELECT * FROM pg_replication_slots; 78 | 79 | 3 在客户端目录下,编辑 demo.cpp 填入需要拉取增量的服务器的连接参数,并重新编译成新的demo. 80 | 执行demo. 81 | 对应的增量信息会输出到客户端. 82 | 可以参考 out_put_decode_message 解析和读取增量消息中的数据. 83 | 84 | ## 六:限制 85 | 1 以 ctid 为条件的更新语句,在表处于 REPLICA FULL 时,update 语句无法完整还原. -------------------------------------------------------------------------------- /doc/mysql2gp.md: -------------------------------------------------------------------------------- 1 | # mysql2gp 使用和部署说明 2 | 3 | ## 一 mysql2gp 介绍 4 | mysql2gp 实现了从 MySQL 中迁移增量数据到 PostgreSQL 或 Greenplum 5 | 6 | 其中增量数据来自于 MySQL 的 binlog, 结合全量数据迁移工具 mysql2pgsql 可以把 MySQL 中的数据完整的迁移到 PostgreSQL Greenplum 中，且保持准实时同步。 7 | 8 | ### 1.1 支持特性和限制 9 | 1 支持拉取 MySQL 5.1 5.5 5.6 5.7 版本的 binlog，需要 binlog 相关参数 10 | 11 | binlog_format = ROW 12 | binlog_row_image = FULL 13 | 14 | 2 支持同步指定表的各类数据变化到目标DB中，包括对应行的 insert update delete。 15 | 16 | 3 数据同步的表需要有单列主键。 17 | 18 | 4 支持对主键进行修改。 19 | 20 | 5 暂时不支持异构数据库的 DDL 同步。 21 | 22 | 6 支持指定表镜像方式同步到 PostgreSQL 或 Greenplum（配置文件方式）。 23 | 24 | 7 支持指定模式的表同步。 25 | 26 | ### 1.2 mysql2gp 实现架构 27 | 简单的说，mysql2gp 的实现方式是： 28 | 29 | 1 在客户端主机（也可以部署在其他主机）上启动一个临时 PG 数据库，用于临时存放从 MySQL 拉去到的 binlog 数据。 30 | 31 | 2 binlog_miner 从源 MySQL 的一个 binlog 文件开始，拉取和解析 binlog 并存放到临时 PG 中。 32 | 33 | 3 binlog_loader 从临时 PG 中读取增量数据，并做适当的处理，最终批量写入到目标 PostgreSQL 或 Greenplum 中去。 34 | 35 | ### 1.3 mysql2gp 模块介绍 36 | 37 | mysql2gp 分为5个部分 38 | 1 binlog_miner 用于拉取目标库中的 binlog, 并保存到临时 DB 中。 39 | 40 | 2 binlog_loader 用于读取临时 DB 中的 binlog 数据并加载到目标库中。 41 | 42 | 3 my.cfg 配置文件，设置需要同步数据的源和目标数据库的链接信息和相关参数。 43 | 44 | 4 loader_table_list.txt 配置文件，设置需要同步的表名列表，用回车符隔开。 45 | 46 | 5 临时 DB，用户保存增量数据的临时数据库。建议和 binlog_miner binlog_loader 部署在同一个主机。 47 | 48 | ## 二 mysql2gp 部署 49 | 建议临时 DB 和客户端二进制部署在同主机 50 | 51 | 部署步骤： 52 | 53 | ### 2.1 部署临时 PG DB 54 | 在目标主机部署一个临时 PG DB 用户存放临时数据，主机需要为临时数据预留足够的保存增量数据的空间。部署完成后获得一个连接临时 PG DB 的连接串，如 “dbname=test port=5432 user=test password=pgsql” 55 | 56 | ### 2.2 配置文件 57 | #### 2.2.1 MySQL 相关 58 | 59 | my.cnf 60 | 61 | ``` 62 | [src.mysql] 63 | host = "192.168.1.1" 64 | port = "3301" 65 | user = "test" 66 | password = "123456" 67 | db = "test" 68 | encodingdir = "share" 69 | encoding = "utf8" 70 | binlogfile = "mysql-bin.000001" 71 | ``` 72 | 73 | 注意： 74 | 75 | 1 MySQL 的连接信息需要有 select 权限和拉取 binlog 的权限。 76 | 77 | 2 binlogfile 为读取 binlog 的启始文件，必须设置。该配置和全量数据同步工具配合使用。 78 | 通常在开始同步全量 MySQL 数据时记录当前正在写的 binlog 文件名，并配置到 my.cnf 中。 79 | 80 | #### 2.2.2 临时数据库 81 | 82 | my.cnf 83 | 84 | ``` 85 | [local.pgsql] 86 | connect_string = "dbname=test port=5432 user=test password=pgsql" 87 | 88 | ``` 89 | 90 | 注意： 91 | 92 | 1 连接本地数据库可以不指定 host 信息，这样的链接模式效率较高。 93 | 94 | 95 | #### 2.2.3 目的数据库 96 | 97 | my.cnf 98 | 99 | ``` 100 | [desc.pgsql] 101 | connect_string = "host=192.167.1.2 dbname=postgres port=5432 user=test password=pgsql" 102 | target_schema = "test" 103 | 104 | ``` 105 | 106 | 注意: 107 | 108 | 1 target_schema 用于指定目标表存在的 schema，也可以不指定，不指时默认 schema 为 public。 109 | 110 | 111 | #### 2.2.4 设置需要同步的表 112 | 113 | 1 my.cnf 114 | 115 | ``` 116 | [binlogloader] 117 | loader_table_list = "loader_table_list.txt" 118 | 119 | ``` 120 | 121 | 2 loader_table_list.txt 122 | 123 | ``` 124 | a 125 | b 126 | ``` 127 | 128 | ### 2.3 启动同步进程 129 | 130 | #### 2.3.1 启动 binlog 拉取进程 131 | 132 | 推荐命令行： 133 | 134 | nohup ./binlog_miner 1>minner.log 2>&1 & 135 | 136 | #### 2.3.2 启动 binlog 写入进程 137 | 138 | 推荐命令行： 139 | 140 | nohup ./binlog_loader 1>loader.log 2>&1 & 141 | -------------------------------------------------------------------------------- /doc/mysql2pgsql_ch.md: -------------------------------------------------------------------------------- 1 | ## mysql2pgsql 2 | 3 | 工具 mysql2pgsql 支持不落地的把 MYSQL 中的表迁移到 HybridDB/Greenplum Database/PostgreSQL/PPAS。此工具的原理是，同时连接源端 mysql 数据库和目的端数据库，从 mysql 库中通过查询得到要导出的数据，然后通过 COPY 命令导入到目的端。此工具支持多线程导入（每个工作线程负责导入一部分数据库表）。 4 | 5 | ## 参数配置 6 | 7 | 修改配置文件 my.cfg、配置源和目的库连接信息。 8 | 9 | - 源库 mysql 的连接信息如下： 10 | 11 | **注意：**源库 mysql 的连接信息中，用户需要有对所有用户表的读权限。 12 | 13 | ``` 14 | [src.mysql] 15 | host = "192.168.1.1" 16 | port = "3306" 17 | user = "test" 18 | password = "test" 19 | db = "test" 20 | encodingdir = "share" 21 | encoding = "utf8" 22 | ``` 23 | 24 | - 目的库 pgsql （包括 PostgreSQL、PPAS 和 HybridDB for PostgreSQL ）的连接信息如下： 25 | 26 | **注意：**目的库 pgsql 的连接信息，用户需要对目标表有写的权限。 27 | 28 | ``` 29 | [desc.pgsql] 30 | connect_string = "host=192.168.1.1 dbname=test port=5888 user=test password=pgsql" 31 | ``` 32 | 33 | ## mysql2pgsql 用法 34 | 35 | mysql2pgsql 的用法如下所示： 36 | 37 | ``` 38 | ./mysql2pgsql -l -d -n -j -s 39 | 40 | ``` 41 | 42 | 参数说明： 43 | 44 | - -l：可选参数，指定一个文本文件，文件中含有需要同步的表；如果不指定此参数，则同步配置文件中指定数据库下的所有表。``````为一个文件名，里面含有需要同步的表集合以及表上查询的条件，其内容格式示例如下： 45 | 46 | ``` 47 | table1 : select * from table_big where column1 < '2016-08-05' 48 | table2 : 49 | table3 50 | table4 : select column1, column2 from tableX where column1 != 10 51 | table5 : select * from table_big where column1 >= '2016-08-05' 52 | ``` 53 | 54 | - -d：可选参数，表示只生成目的表的建表 DDL 语句，不实际进行数据同步。 55 | 56 | - -n：可选参数，需要与-d一起使用，指定在 DDL 语句中不包含表分区定义。 57 | 58 | - -j：可选参数，指定使用多少线程进行数据同步；如果不指定此参数，会使用 5 个线程并发。 59 | 60 | - -s：可选参数，指定目标表的schema，一次命令只能指定一个schema。如果不指定此参数，则数据会导入到public下的表。 61 | 62 | ### 典型用法 63 | 64 | #### 全库迁移 65 | 66 | 全库迁移的操作步骤如下所示： 67 | 68 | 1\. 通过如下命令，获取目的端对应表的 DDL。 69 | 70 | ``` 71 | ./mysql2pgsql -d 72 | ``` 73 | 74 | 2\. 根据这些 DDL，再加入 distribution key 等信息，在目的端创建表。 75 | 76 | 3\. 执行如下命令，同步所有表： 77 | 78 | ``` 79 | ./mysql2pgsql 80 | ``` 81 | 82 | 此命令会把配置文件中所指定数据库中的所有 mysql 表数据迁移到目的端。过程中使用 5 个线程（即缺省线程数为 5），读取和导入所有涉及的表数据。 83 | 84 | #### 部分表迁移 85 | 86 | 1\. 编辑一个新文件 tab_list.txt，放入如下内容： 87 | 88 | ``` 89 | t1 90 | t2 : select * from t2 where c1 > 138888 91 | ``` 92 | 93 | 2\. 执行如下命令，同步指定的 t1 和 t2 表（注意 t2 表只迁移符合 c1 > 138888 条件的数据）： 94 | 95 | ``` 96 | ./mysql2pgsql -l tab_list.txt 97 | ``` 98 | 99 | ## mysql2pgsql 二进制安装包下载 100 | 101 | 下载地址：单击[这里](https://github.com/aliyun/rds_dbsync/releases)。 102 | 103 | ## mysql2pgsql 源码编译说明 104 | 105 | 查看源码编译说明，单击[这里](https://github.com/aliyun/rds_dbsync/blob/master/README.md)。 106 | -------------------------------------------------------------------------------- /doc/mysql2pgsql_en.md: -------------------------------------------------------------------------------- 1 | ## mysql2pgsql Import data from MySQL 2 | 3 | The mysql2pgsql tool supports migrating tables in MySQL to HybridDB for PostgreSQL, Greenplum Database, PostgreSQL, or PPAS without storing the data separately. This tool connects to the source MySQL database and the target database at the same time, querries and retrieves the data to be exported in the MySQL database, and then imports the data to the target database by using the COPY command. It supports multithread import (every worker thread is in charge of importing a part of database tables). 4 | 5 | 6 | 7 | ## Parameters configuration 8 | 9 | Modify the “my.cfg” configuration file, and configure the source and target database connection information. 10 | 11 | - The connection information of the source MySQL database is as follows: 12 | 13 | **Note:** You need to have the read permission on all user tables in the source MySQL database connection information. 14 | 15 | ``` 16 | [src.mysql] 17 | host = "192.168.1.1" 18 | port = "3306" 19 | user = "test" 20 | password = "test" 21 | db = "test" 22 | encodingdir = "share" 23 | encoding = "utf8" 24 | ``` 25 | 26 | - The connection information of the target PostgreSQL database (including PostgreSQL, PPAS and HybridDB for PostgreSQL) is as follows: 27 | 28 | **Note:** You need to have the write permission on the target table in the target PostgreSQL database. 29 | 30 | 31 | ``` 32 | [desc.pgsql] 33 | connect_string = "host=192.168.1.1 dbname=test port=5888 user=test password=pgsql" 34 | ``` 35 | 36 | ## mysql2pgsql Usage discription 37 | 38 | The usage of mysql2pgsql is described as follows: 39 | 40 | 41 | ``` 42 | ./mysql2pgsql -l -d -n -j -s 43 | 44 | ``` 45 | 46 | Parameter descriptions: 47 | 48 | - -l: Optional parameter, used to specify a text file that contains tables to be synchronized. If this parameter is not specified, all the tables in the database specified in the configuration file are synchronized. is a file name. The file contains tables set to be synchronized and query conditions on the tables. An example of the content format is shown as follows: 49 | 50 | 51 | ``` 52 | table1 : select * from table_big where column1 < '2016-08-05' 53 | table2 : 54 | table3 55 | table4 : select column1, column2 from tableX where column1 != 10 56 | table5 : select * from table_big where column1 >= '2016-08-05' 57 | ``` 58 | 59 | - -d: Optional parameter, indicating to only generate the tabulation DDL statement of the target table without performing actual data synchronization. 60 | 61 | - -j: Optional parameter, specifying the number of threads used for data synchronization. If this parameter is not specified, five threads are used concurrently. 62 | 63 | 64 | ### Typical usage 65 | 66 | #### Full-database migration 67 | 68 | The procedure is as follows: 69 | 70 | 1\. Run the following command to get the DDL statements of the corresponding table on the target end: 71 | 72 | 73 | ``` 74 | ./mysql2pgsql -d 75 | ``` 76 | 77 | 2\. Create a table on the target based on these DDL statements with the distribution key information added. 78 | 79 | 3\. Run the following command to synchronize all tables: 80 | 81 | 82 | ``` 83 | ./mysql2pgsql 84 | ``` 85 | 86 | This command migrates the data from all MySQL tables in the database specified in the configuration file to the target. Five threads are used during the process (the default thread number is five) to read and import the data from all tables involved. 87 | 88 | #### Partial table migration 89 | 90 | The procedure is as follows: 91 | 92 | 1\. Create a new file (tab_list.txt) and insert the following content: 93 | 94 | 95 | ``` 96 | t1 97 | t2 : select * from t2 where c1 > 138888 98 | ``` 99 | 100 | 2\. Run the following command to synchronize the specified t1 and t2 tables: 101 | 102 | 103 | ``` 104 | ./mysql2pgsql -l tab_list.txt 105 | ``` 106 | 107 | **Note:** For the t2 table, only the data that meets the c1 > 138888 condition is migrated. 108 | 109 | ## Download and instructions 110 | 111 | [Download the binary installer of mysql2pgsql](https://github.com/aliyun/rds_dbsync/releases) 112 | 113 | ## rds_dbsync project 114 | 115 | [View the mysql2pgsql source code compilation instructions](https://github.com/aliyun/rds_dbsync/blob/master/README.md) 116 | -------------------------------------------------------------------------------- /doc/pgsql2pgsql_ch.md: -------------------------------------------------------------------------------- 1 | # pgsql2pgsql 2 | 工具 pgsql2pgsql 支持不落地的把 GreenPlum/PostgreSQL/PPAS 中的表迁移到 GreenPlum/PostgreSQL/PPAS 3 | 4 | # pgsql2pgsql 支持的功能 5 | 6 | 1 PostgreSQL/PPAS/GreenPlum 全量数据迁移到 PostgreSQL/PPAS/GreenPlum 7 | 8 | 2 PostgreSQL/PPAS(版本大于9.4) 全量+增量迁移到 PostgreSQL/PPAS 9 | 10 | # 参数配置 11 | 修改配置文件 my.cfg，配置源和目的库连接信息 12 | 13 | 1. 源库 pgsql 连接信息 14 | [src.pgsql] 15 | connect_string = "host=192.168.1.1 dbname=test port=5888 user=test password=pgsql" 16 | 17 | 2. 本地临时DB pgsql 连接信息 18 | [local.pgsql] 19 | connect_string = "host=192.168.1.1 dbname=test port=5888 user=test2 password=pgsql" 20 | 21 | 3. 目的库 pgsql 连接信息 22 | [desc.pgsql] 23 | connect_string = "host=192.168.1.1 dbname=test port=5888 user=test3 password=pgsql" 24 | 25 | 26 | #注意 27 | 1. 如果要做增量数据同步，连接源库需要有创建 replication slot 的权限 28 | 2. 源库 pgsql 的连接信息中，用户最好是对应 DB 的 owner 29 | 3. 目的库 pgsql 的连接信息，用户需要对目标表有写权限 30 | 4. PostgreSQL 9.4 以及以上的版本因为支持逻辑流复制，所以支持作为数据源的增量迁移。打开下列内核参数才能让内核支持逻辑流复制功能。 31 | a. wal_level = logical 32 | b. max_wal_senders = 6 33 | c. max_replication_slots = 6 34 | 35 | # mysql2pgsql用法 36 | 37 | 38 | 1 全库迁移 39 | 40 | ./pgsql2pgsql 41 | 迁移程序会默认把对应 pgsql 库中所有的用户表数据将迁移到 pgsql 42 | 43 | 2 状态信息查询 44 | 连接本地临时DB，可以查看到单次迁移过程中的状态信息。他们放在表 db_sync_status 中，包括全量迁移的开始和结束时间，增量迁移的开始时间，增量同步的数据情况。 45 | 46 | -------------------------------------------------------------------------------- /doc/pgsql2pgsql_en.md: -------------------------------------------------------------------------------- 1 | # pgsql2pgsql Import data from PostgreSQL 2 | 3 | The pgsql2pgsql tool supports migrating tables in HybridDB for PostgreSQL, Greenplum Database, PostgreSQL, or PPAS to HybridDB for PostgreSQL, Greenplum Database, PostgreSQL, or PPAS without storing the data separately. 4 | 5 | # Features 6 | 7 | pgsql2pgsql supports the following features: 8 | 9 | * 1 Full-database migration from PostgreSQL, PPAS, Greenplum Database, or HybridDB for PostgreSQL to PostgreSQL, PPAS, Greenplum Database, or HybridDB for PostgreSQL. 10 | 11 | * 2 Full-database migration and incremental data migration from PostgreSQL or PPAS (9.4 or later versions) to PostgreSQL, or PPAS. 12 | 13 | # Parameters configuration 14 | Modify the “my.cfg” configuration file, and configure the source and target database connection information. 15 | 16 | * The connection information of the source PostgreSQL database is shown as follows: 17 | 18 | **Note:** The user is preferably the corresponding database owner in the source PostgreSQL database connection information. 19 | 20 | ``` 21 | [src.pgsql] 22 | connect_string = "host=192.168.1.1 dbname=test port=5888 user=test password=pgsql" 23 | ``` 24 | 25 | * The connection information of the local temporary PostgreSQL database is shown as follows: 26 | 27 | ``` 28 | [local.pgsql] 29 | connect_string = "host=192.168.1.1 dbname=test port=5888 user=test2 password=pgsql" 30 | ``` 31 | 32 | * The connection information of the target PostgreSQL database is shown as follows: 33 | 34 | **Note:** You need to have the write permission on the target table of the target PostgreSQL database. 35 | 36 | 37 | ``` 38 | [desc.pgsql] 39 | connect_string = "host=192.168.1.1 dbname=test port=5888 user=test3 password=pgsql" 40 | ``` 41 | 42 | #Note: 43 | 44 | * If you want to perform incremental data synchronization, the connected source database must have the permission to create replication slots. 45 | 46 | * PostgreSQL 9.4 and later versions support logic flow replication, so it supports the incremental migration if PostgreSQL serves as the data source. The kernel only supports logic flow replication after you enable the following kernel parameters. 47 | 48 | 49 | ``` 50 | wal_level = logical 51 | max_wal_senders = 6 52 | max_replication_slots = 6 53 | ``` 54 | 55 | 56 | # Use pgsql2pgsql 57 | 58 | ## Full-database migration 59 | 60 | Run the following command to perform a full-database migration: 61 | 62 | ``` 63 | ./pgsql2pgsql 64 | ``` 65 | 66 | 67 | By default, the migration program migrates the table data of all the users in the corresponding PostgreSQL database to PostgreSQL. 68 | 69 | ## Status information query 70 | 71 | Connect to the local temporary database, and you can view the status information in a single migration process. The information is stored in the db_sync_status table, including the start and end time of the full-database migration, the start time of the incremental data migration, and the data situation of incremental synchronization. 72 | 73 | 74 | # Download and instructions 75 | 76 | ## binary download link 77 | 78 | [Download the binary installer of pgsql2pgsql](https://github.com/aliyun/rds_dbsync/releases) 79 | 80 | ## rds_dbsync project 81 | 82 | [View the mysql2pgsql source code compilation instructions](https://github.com/aliyun/rds_dbsync/blob/master/README.md) 83 | 84 | 85 | --------------------------------------------------------------------------------