├── .github ├── CODEOWNERS ├── ISSUE_TEMPLATE │ ├── QUESTION.md │ ├── config.yml │ ├── FEATURE_REQUEST.md │ └── BUG_REPORT.md ├── dependabot.yml ├── workflows │ ├── pythonpublish.yml │ └── ci.yml └── PULL_REQUEST_TEMPLATE.md ├── docker-compose.yml ├── .noserc ├── .gitignore ├── sample_logging.conf ├── tests ├── integration │ ├── resources │ │ ├── invalid-json.json │ │ ├── invalid-message-order.json │ │ ├── messages-with-nested-schema.json │ │ ├── messages-with-non-db-friendly-columns.json │ │ ├── messages-with-invalid-records.json │ │ ├── messages-with-reserved-name-as-table-name.json │ │ ├── messages-with-unicode-characters.json │ │ ├── messages-with-space-in-table-name.json │ │ ├── messages-with-multiple-streams-modified-column.json │ │ ├── messages-with-multiple-streams.json │ │ ├── messages-pg-logical-streams-no-records.json │ │ └── messages-with-long-texts.json │ └── utils.py └── unit │ ├── test_target_postgres.py │ ├── test_db_sync.py │ └── resources │ └── logical-streams.json ├── Makefile ├── setup.py ├── CHANGELOG.md ├── README.md ├── LICENSE ├── target_postgres ├── __init__.py └── db_sync.py └── .pylintrc /.github/CODEOWNERS: -------------------------------------------------------------------------------- 1 | * @transferwise/analytics-platform 2 | -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/QUESTION.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Question 3 | about: Ask anything about this project 4 | title: '' 5 | labels: help wanted 6 | assignees: '' 7 | 8 | --- 9 | 10 | **Your question** -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/config.yml: -------------------------------------------------------------------------------- 1 | blank_issues_enabled: false 2 | contact_links: 3 | - name: PipelineWise Community Slack channel 4 | url: https://singer-io.slack.com/messages/pipelinewise 5 | about: Open discussion about PipelineWise -------------------------------------------------------------------------------- /docker-compose.yml: -------------------------------------------------------------------------------- 1 | version: "3" 2 | 3 | services: 4 | db: 5 | image: postgres:12-alpine 6 | environment: 7 | POSTGRES_DB: "target_db" 8 | POSTGRES_USER: "my_user" 9 | POSTGRES_PASSWORD: "secret" 10 | ports: 11 | - 5432:5432 12 | -------------------------------------------------------------------------------- /.noserc: -------------------------------------------------------------------------------- 1 | [nosetests] 2 | # enable coverage 3 | with-coverage=1 4 | 5 | # cover only the main package tap_postgres 6 | cover-package=tap_postgres 7 | 8 | # set coverage minimum to 45, it's the current value 9 | cover-min-percentage=45 10 | 11 | # product html coverage report 12 | cover-html=1 13 | 14 | # folder where to produce the html coverage report 15 | cover-html-dir=coverage_report -------------------------------------------------------------------------------- /.github/dependabot.yml: -------------------------------------------------------------------------------- 1 | # This is an automatically generated base configuration 2 | # For further configuration options and tuning: 3 | # https://docs.github.com/en/free-pro-team@latest/github/administering-a-repository/configuration-options-for-dependency-updates 4 | 5 | version: 2 6 | updates: 7 | - package-ecosystem: "pip" 8 | directory: "/" 9 | schedule: 10 | interval: "weekly" 11 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # IDE 2 | .vscode 3 | .idea/* 4 | 5 | 6 | # Python 7 | __pycache__/ 8 | *.py[cod] 9 | *$py.class 10 | .virtualenvs 11 | virtualenvs/ 12 | *.egg-info/ 13 | *__pycache__/ 14 | *~ 15 | dist/ 16 | 17 | # Singer JSON files 18 | properties.json 19 | config.json 20 | state.json 21 | 22 | *.db 23 | .DS_Store 24 | venv 25 | env 26 | blog_old.md 27 | node_modules 28 | *.pyc 29 | tmp 30 | 31 | # Docs 32 | docs/_build/ 33 | docs/_templates/ 34 | -------------------------------------------------------------------------------- /sample_logging.conf: -------------------------------------------------------------------------------- 1 | [loggers] 2 | keys=root 3 | 4 | [handlers] 5 | keys=stderr 6 | 7 | [formatters] 8 | keys=child 9 | 10 | [logger_root] 11 | level=INFO 12 | handlers=stderr 13 | formatter=child 14 | propagate=0 15 | 16 | [handler_stderr] 17 | level=INFO 18 | class=StreamHandler 19 | formatter=child 20 | args=(sys.stderr,) 21 | 22 | [formatter_child] 23 | class=logging.Formatter 24 | format=time=%(asctime)s name=%(name)s level=%(levelname)s message=%(message)s 25 | datefmt=%Y-%m-%d %H:%M:%S 26 | -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/FEATURE_REQUEST.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Feature request 3 | about: Suggest an idea for this project 4 | title: '' 5 | labels: enhancement 6 | assignees: '' 7 | 8 | --- 9 | 10 | **Is your feature request related to a problem? Please describe.** 11 | A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] 12 | 13 | **Describe the solution you'd like** 14 | A clear and concise description of what you want to happen. 15 | 16 | **Describe alternatives you've considered** 17 | A clear and concise description of any alternative solutions or features you've considered. 18 | 19 | **Additional context** 20 | Add any other context or screenshots about the feature request here. -------------------------------------------------------------------------------- /tests/integration/resources/invalid-json.json: -------------------------------------------------------------------------------- 1 | {"type": "STATE", "value": {"currently_syncing": "tap_mysql_test-test_table_one"}} 2 | {"type": "SCHEMA", "stream": "tap_mysql_test-test_table_one", "schema": {"properties": {"c_pk": {"inclusion": "automatic", "minimum": -2147483648, "maximum": 2147483647, "type": ["null", "integer"]}, "c_varchar": {"inclusion": "available", "maxLength": 16, "type": ["null", "string"]}, "c_int": {"inclusion": "available", "minimum": -2147483648, "maximum": 2147483647, "type": ["null", "integer"]}}, "type": "object"}, "key_properties": ["c_pk"]} 3 | THIS IS A TEST INPUT FROM A TAP WITH A LINE WITH INVALID JSON 4 | {"type": "ACTIVATE_VERSION", "stream": "tap_mysql_test-test_table_one", "version": 1} 5 | -------------------------------------------------------------------------------- /.github/workflows/pythonpublish.yml: -------------------------------------------------------------------------------- 1 | name: Upload Python Package 2 | 3 | on: 4 | release: 5 | types: [created] 6 | 7 | jobs: 8 | deploy: 9 | runs-on: ubuntu-latest 10 | steps: 11 | - uses: actions/checkout@v2 12 | - name: Set up Python 13 | uses: actions/setup-python@v1 14 | with: 15 | python-version: '3.x' 16 | - name: Install dependencies 17 | run: | 18 | python -m pip install --upgrade pip 19 | pip install setuptools wheel twine 20 | - name: Build and publish 21 | env: 22 | TWINE_USERNAME: ${{ secrets.PYPI_USERNAME }} 23 | TWINE_PASSWORD: ${{ secrets.PYPI_PASSWORD }} 24 | run: | 25 | python setup.py sdist bdist_wheel 26 | twine upload dist/* 27 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | venv: 2 | python3 -m venv venv ;\ 3 | . ./venv/bin/activate ;\ 4 | pip install --upgrade pip setuptools wheel ;\ 5 | pip install -e .[test] 6 | 7 | pylint: 8 | . ./venv/bin/activate ;\ 9 | pylint --rcfile .pylintrc target_postgres/ 10 | 11 | unit_test: 12 | . ./venv/bin/activate ;\ 13 | pytest --cov=target_postgres --cov-fail-under=44 tests/unit -v 14 | 15 | env: 16 | export TARGET_POSTGRES_PORT=5432 17 | export TARGET_POSTGRES_DBNAME=target_db 18 | export TARGET_POSTGRES_USER=my_user 19 | export TARGET_POSTGRES_PASSWORD=secret 20 | export TARGET_POSTGRES_HOST=localhost 21 | export TARGET_POSTGRES_SCHEMA=public 22 | 23 | integration_test: env 24 | . ./venv/bin/activate ;\ 25 | pytest tests/integration --cov=target_postgres --cov-fail-under=87 -v 26 | -------------------------------------------------------------------------------- /tests/integration/resources/invalid-message-order.json: -------------------------------------------------------------------------------- 1 | {"type": "SCHEMA", "stream": "tap_mysql_test-test_table_one", "schema": {"properties": {"c_pk": {"inclusion": "automatic", "minimum": -2147483648, "maximum": 2147483647, "type": ["null", "integer"]}, "c_varchar": {"inclusion": "available", "maxLength": 16, "type": ["null", "string"]}, "c_int": {"inclusion": "available", "minimum": -2147483648, "maximum": 2147483647, "type": ["null", "integer"]}}, "type": "object"}, "key_properties": ["c_pk"]} 2 | {"type": "RECORD", "stream": "tap_mysql_test-test_table_one", "record": {"c_pk": 1, "c_varchar": "1", "c_int": 1}, "version": 1, "time_extracted": "2019-01-31T15:51:47.465408Z"} 3 | {"type": "RECORD", "stream": "tap_mysql_test-test_table_two", "record": {"c_pk": 2, "c_varchar": "2", "c_int": 2, "c_date": "2019-02-10 02:00:00"}, "version": 3, "time_extracted": "2019-01-31T15:51:48.861962Z"} -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/BUG_REPORT.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Bug report 3 | about: Create a bug report to help us improve 4 | title: '' 5 | labels: bug 6 | assignees: '' 7 | 8 | --- 9 | 10 | --- 11 | name: Bug report 12 | about: Create a report to help us improve 13 | title: '' 14 | labels: '' 15 | assignees: '' 16 | 17 | --- 18 | 19 | **Describe the bug** 20 | A clear and concise description of what the bug is. 21 | 22 | **To Reproduce** 23 | Steps to reproduce the behavior: 24 | 1. Prepare the data as '...' 25 | 2. Run the command '....' 26 | 4. See error 27 | 28 | **Expected behavior** 29 | A clear and concise description of what you expected to happen. 30 | 31 | **Screenshots** 32 | If applicable, add screenshots to help explain your problem. 33 | 34 | **Your environment** 35 | - Version of target: [e.g. 2.0.0] 36 | - Version of python [e.g. 3.8] 37 | 38 | **Additional context** 39 | Add any other context about the problem here. -------------------------------------------------------------------------------- /.github/workflows/ci.yml: -------------------------------------------------------------------------------- 1 | name: CI 2 | 3 | on: 4 | pull_request: 5 | push: 6 | branches: 7 | - master 8 | 9 | jobs: 10 | lint_and_test: 11 | name: Linting and Testing 12 | runs-on: ubuntu-latest 13 | strategy: 14 | matrix: 15 | python-version: [ 3.7, 3.8, 3.9, '3.10' ] 16 | 17 | steps: 18 | - name: Checkout repository 19 | uses: actions/checkout@v2 20 | 21 | - name: Start PG test container 22 | run: docker-compose up -d --build db 23 | 24 | - name: Set up Python ${{ matrix.python-version }} 25 | uses: actions/setup-python@v2 26 | with: 27 | python-version: ${{ matrix.python-version }} 28 | 29 | - name: Setup virtual environment 30 | run: make venv 31 | 32 | - name: Pylinting 33 | run: make pylint 34 | 35 | - name: Unit Tests 36 | run: make unit_test 37 | 38 | - name: Integration Tests 39 | env: 40 | LOGGING_CONF_FILE: ./sample_logging.conf 41 | run: make integration_test 42 | 43 | - name: Shutdown PG test container 44 | run: docker-compose down 45 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | from setuptools import setup 4 | 5 | with open('README.md') as f: 6 | long_description = f.read() 7 | 8 | setup(name="pipelinewise-target-postgres", 9 | version="2.1.2", 10 | description="Singer.io target for loading data to PostgreSQL - PipelineWise compatible", 11 | long_description=long_description, 12 | long_description_content_type='text/markdown', 13 | author="TransferWise", 14 | url='https://github.com/transferwise/pipelinewise-target-postgres', 15 | classifiers=[ 16 | 'License :: OSI Approved :: Apache Software License', 17 | 'Programming Language :: Python :: 3 :: Only' 18 | ], 19 | py_modules=["target_postgres"], 20 | install_requires=[ 21 | 'pipelinewise-singer-python==1.*', 22 | 'psycopg2-binary==2.9.5', 23 | 'inflection==0.3.1', 24 | 'joblib==1.2.0' 25 | ], 26 | extras_require={ 27 | "test": [ 28 | 'pytest==6.2.5', 29 | 'pylint==2.6.0', 30 | 'pytest-cov==2.10.1', 31 | ] 32 | }, 33 | entry_points=""" 34 | [console_scripts] 35 | target-postgres=target_postgres:main 36 | """, 37 | packages=["target_postgres"], 38 | package_data={}, 39 | include_package_data=True, 40 | ) 41 | -------------------------------------------------------------------------------- /tests/unit/test_target_postgres.py: -------------------------------------------------------------------------------- 1 | import unittest 2 | import os 3 | import gzip 4 | import tempfile 5 | 6 | from unittest.mock import patch 7 | 8 | import target_postgres 9 | 10 | 11 | def _mock_record_to_csv_line(record): 12 | return record 13 | 14 | 15 | class TestTargetPostgres(unittest.TestCase): 16 | 17 | def setUp(self): 18 | self.config = {} 19 | 20 | @patch('target_postgres.flush_streams') 21 | @patch('target_postgres.DbSync') 22 | def test_persist_lines_with_40_records_and_batch_size_of_20_expect_flushing_once(self, 23 | dbsync_mock, 24 | flush_streams_mock): 25 | self.config['batch_size_rows'] = 20 26 | self.config['flush_all_streams'] = True 27 | 28 | with open(f'{os.path.dirname(__file__)}/resources/logical-streams.json', 'r') as f: 29 | lines = f.readlines() 30 | 31 | instance = dbsync_mock.return_value 32 | instance.create_schema_if_not_exists.return_value = None 33 | instance.sync_table.return_value = None 34 | 35 | flush_streams_mock.return_value = '{"currently_syncing": null}' 36 | 37 | target_postgres.persist_lines(self.config, lines) 38 | 39 | flush_streams_mock.assert_called_once() 40 | -------------------------------------------------------------------------------- /.github/PULL_REQUEST_TEMPLATE.md: -------------------------------------------------------------------------------- 1 | ## Problem 2 | 3 | _Describe the problem your PR is trying to solve_ 4 | 5 | ## Proposed changes 6 | 7 | _Describe the big picture of your changes here to communicate to the maintainers why we should accept this pull request. 8 | If it fixes a bug or resolves a feature request, be sure to link to that issue._ 9 | 10 | 11 | ## Types of changes 12 | 13 | What types of changes does your code introduce to PipelineWise? 14 | _Put an `x` in the boxes that apply_ 15 | 16 | - [ ] Bugfix (non-breaking change which fixes an issue) 17 | - [ ] New feature (non-breaking change which adds functionality) 18 | - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) 19 | - [ ] Documentation Update (if none of the other choices apply) 20 | 21 | 22 | ## Checklist 23 | 24 | - [ ] Description above provides context of the change 25 | - [ ] I have added tests that prove my fix is effective or that my feature works 26 | - [ ] Unit tests for changes (not needed for documentation changes) 27 | - [ ] CI checks pass with my changes 28 | - [ ] Bumping version in `setup.py` is an individual PR and not mixed with feature or bugfix PRs 29 | - [ ] Commit message/PR title starts with `[AP-NNNN]` (if applicable. AP-NNNN = JIRA ID) 30 | - [ ] Branch name starts with `AP-NNN` (if applicable. AP-NNN = JIRA ID) 31 | - [ ] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" 32 | - [ ] Relevant documentation is updated including usage instructions -------------------------------------------------------------------------------- /tests/integration/resources/messages-with-nested-schema.json: -------------------------------------------------------------------------------- 1 | {"type": "STATE", "value": {"currently_syncing": "tap_mysql_test-test_table_nested_schema"}} 2 | {"type": "SCHEMA", "stream": "tap_mysql_test-test_table_nested_schema", "schema": {"properties": {"c_pk": {"inclusion": "automatic", "minimum": -2147483648, "maximum": 2147483647, "type": ["null", "integer"]}, "c_array": {"type": ["null", "object"]}, "c_object": {"type": ["null", "object"]}, "c_object_with_props": {"type": ["null", "object"], "properties": {"key_1": {"type": ["null", "string"]}}}, "c_nested_object": {"type": ["null", "object"], "properties": {"nested_prop_1": {"type": ["null", "string"]}, "nested_prop_2": {"type": ["null", "string"]}, "nested_prop_3": {"type": ["null", "object"], "properties": {"multi_nested_prop_1": {"type": ["null", "string"]}, "multi_nested_prop_2": {"type": ["null", "string"]}}}}}}, "type": "object"}, "key_properties": ["c_pk"]} 3 | {"type": "ACTIVATE_VERSION", "stream": "tap_mysql_test-test_table_nested_schema", "version": 1} 4 | {"type": "RECORD", "stream": "tap_mysql_test-test_table_nested_schema", "record": {"c_pk": 1, "c_array": [1, 2, 3], "c_object": {"key_1": "value_1"}, "c_object_with_props": {"key_1": "value_1"}, "c_nested_object": {"nested_prop_1": "nested_value_1", "nested_prop_2": "nested_value_2", "nested_prop_3": {"multi_nested_prop_1": "multi_value_1", "multi_nested_prop_2": "multi_value_2"}}}, "version": 1, "time_extracted": "2019-01-31T15:51:50.215998Z"} 5 | {"type": "STATE", "value": {"currently_syncing": "tap_mysql_test-test_table_nested_schema", "bookmarks": {"tap_mysql_test-test_table_nested_schema": {"initial_full_table_complete": true}}}} 6 | {"type": "ACTIVATE_VERSION", "stream": "tap_mysql_test-test_table_nested_schema", "version": 1} 7 | {"type": "STATE", "value": {"currently_syncing": null, "bookmarks": {"tap_mysql_test-test_table_nested_schema": {"initial_full_table_complete": true}}}} 8 | -------------------------------------------------------------------------------- /tests/integration/utils.py: -------------------------------------------------------------------------------- 1 | import os 2 | import json 3 | 4 | 5 | def get_db_config(): 6 | config = {} 7 | 8 | # -------------------------------------------------------------------------- 9 | # Default configuration settings for integration tests. 10 | # -------------------------------------------------------------------------- 11 | # The following values needs to be defined in environment variables with 12 | # valid details to a Postgres instace 13 | # -------------------------------------------------------------------------- 14 | # Postgres instance 15 | config['host'] = os.environ.get('TARGET_POSTGRES_HOST') 16 | config['port'] = os.environ.get('TARGET_POSTGRES_PORT') 17 | config['user'] = os.environ.get('TARGET_POSTGRES_USER') 18 | config['password'] = os.environ.get('TARGET_POSTGRES_PASSWORD') 19 | config['dbname'] = os.environ.get('TARGET_POSTGRES_DBNAME') 20 | config['default_target_schema'] = os.environ.get("TARGET_POSTGRES_SCHEMA") 21 | 22 | 23 | # -------------------------------------------------------------------------- 24 | # The following variables needs to be empty. 25 | # The tests cases will set them automatically whenever it's needed 26 | # -------------------------------------------------------------------------- 27 | config['disable_table_cache'] = None 28 | config['schema_mapping'] = None 29 | config['add_metadata_columns'] = None 30 | config['hard_delete'] = None 31 | config['flush_all_streams'] = None 32 | 33 | 34 | return config 35 | 36 | 37 | def get_test_config(): 38 | db_config = get_db_config() 39 | 40 | return db_config 41 | 42 | 43 | def get_test_tap_lines(filename): 44 | lines = [] 45 | with open('{}/resources/{}'.format(os.path.dirname(__file__), filename)) as tap_stdout: 46 | for line in tap_stdout.readlines(): 47 | lines.append(line) 48 | 49 | return lines 50 | 51 | -------------------------------------------------------------------------------- /tests/integration/resources/messages-with-non-db-friendly-columns.json: -------------------------------------------------------------------------------- 1 | {"type": "STATE", "value": {"currently_syncing": "tap_mysql_test-test_table_non_db_friendly_columns"}} 2 | {"type": "SCHEMA", "stream": "tap_mysql_test-test_table_non_db_friendly_columns", "schema": {"properties": {"c_pk": {"inclusion": "automatic", "minimum": -2147483648, "maximum": 2147483647, "type": ["null", "integer"]}, "camelcaseColumn": {"inclusion": "available", "maxLength": 16, "type": ["null", "string"]}, "minus-column": {"inclusion": "available", "maxLength": 16, "type": ["null", "string"]}}, "type": "object"}, "key_properties": ["c_pk"]} 3 | {"type": "ACTIVATE_VERSION", "stream": "tap_mysql_test-test_table_non_db_friendly_columns", "version": 1} 4 | {"type": "RECORD", "stream": "tap_mysql_test-test_table_non_db_friendly_columns", "record": {"c_pk": 1, "camelcaseColumn": "Dummy row 1", "minus-column": "Dummy row 1"}, "version": 1, "time_extracted": "2019-01-31T15:51:50.215998Z"} 5 | {"type": "RECORD", "stream": "tap_mysql_test-test_table_non_db_friendly_columns", "record": {"c_pk": 2, "camelcaseColumn": "Dummy row 2", "minus-column": "Dummy row 2"}, "version": 1, "time_extracted": "2019-01-31T15:51:50.215998Z"} 6 | {"type": "RECORD", "stream": "tap_mysql_test-test_table_non_db_friendly_columns", "record": {"c_pk": 3, "camelcaseColumn": "Dummy row 3", "minus-column": "Dummy row 3"}, "version": 1, "time_extracted": "2019-01-31T15:51:50.215998Z"} 7 | {"type": "RECORD", "stream": "tap_mysql_test-test_table_non_db_friendly_columns", "record": {"c_pk": 4, "camelcaseColumn": "Dummy row 4", "minus-column": "Dummy row 4"}, "version": 1, "time_extracted": "2019-01-31T15:51:50.215998Z"} 8 | {"type": "RECORD", "stream": "tap_mysql_test-test_table_non_db_friendly_columns", "record": {"c_pk": 5, "camelcaseColumn": "Dummy row 5", "minus-column": "Dummy row 5"}, "version": 1, "time_extracted": "2019-01-31T15:51:50.215998Z"} 9 | {"type": "STATE", "value": {"currently_syncing": "tap_mysql_test-test_table_non_db_friendly_columns", "bookmarks": {"tap_mysql_test-test_table_one": {"initial_full_table_complete": true}, "tap_mysql_test-test_table_two": {"initial_full_table_complete": true}}}} 10 | {"type": "ACTIVATE_VERSION", "stream": "tap_mysql_test-test_table_non_db_friendly_columns", "version": 1} 11 | {"type": "STATE", "value": {"currently_syncing": null, "bookmarks": {"tap_mysql_test-test_table_non_db_friendly_columns": {"initial_full_table_complete": true}}}} 12 | -------------------------------------------------------------------------------- /CHANGELOG.md: -------------------------------------------------------------------------------- 1 | 2.1.2 (2023-01-17) 2 | ------------------- 3 | - Add python 3.10 compatibility 4 | - Bump `joblib` from 0.16.0 to 1.2.0 5 | - Bump `psycopg2-binary` from 2.8.5 to 2.9.5 to get macOS arm64 (Apple M1) wheels 6 | 7 | 2.1.1 (2021-05-11) 8 | ------------------- 9 | - Bump `joblib` from 0.13.2 to 0.16.0 to be Python 3.8 compatible 10 | 11 | 12 | 2.1.0 (2020-09-22) 13 | ------------------- 14 | 15 | - Enable SSL mode if `ssl` option is `"true"` in config 16 | 17 | 2.0.1 (2020-06-17) 18 | ------------------- 19 | 20 | - Switch jsonschema to use Draft7Validator 21 | 22 | 2.0.0 (2020-05-02) 23 | ------------------- 24 | 25 | **WARNING**: This release includes non backward compatible changes. 26 | Starting from `pipelinewise-target-postgres-2.0.0` the `integer` JSON Schema column types with minimum and maximum 27 | boundaries are loaded into Postgres `SMALLINT`, `INTEGER` and `BIGINT` values. If you're upgrading from an 28 | earlier version of pipelinewise-target-postgres then it's recommended to re-sync every table otherwise all the existing 29 | `NUMERIC` columns in Postgres will be versioned to the corresponding integer type. 30 | 31 | Further info about versioning columns at https://transferwise.github.io/pipelinewise/user_guide/schema_changes.html?highlight=versioning#versioning-columns 32 | 33 | ### Changes 34 | - Add `flush_all_streams` option 35 | - Add `parallelism` option 36 | - Add `max_parallelism` option 37 | - Add `validate_records` option 38 | - Log inserts, updates and csv size_bytes in a more consumable format 39 | - Fixed an issue when JSON values sometimes not sent correctly 40 | - Support usage of reserved words as table and column names 41 | - Add `temp_dir` optional parameter to config 42 | - Load `integer` JSON Schema types with min and max boundaries to Postgres `SMALLINT`, `INTEGER`, `BIGINT` column types 43 | - Switch to `psychopg-binary` 2.8.5 44 | 45 | 1.1.0 (2019-02-18) 46 | ------------------- 47 | 48 | - Support custom logging configuration by setting `LOGGING_CONF_FILE` env variable to the absolute path of a .conf file 49 | 50 | 1.0.4 (2019-10-01) 51 | ------------------- 52 | 53 | - Grant privileges correctly when table created 54 | 55 | 1.0.3 (2019-09-01) 56 | ------------------- 57 | 58 | - Fixed type mapping 59 | 60 | 1.0.2 (2019-08-16) 61 | ------------------- 62 | 63 | - Add license details 64 | 65 | 1.0.1 (2019-08-12) 66 | ------------------- 67 | 68 | - Sync column versioning feature to other supported targets 69 | - Sync data flattening option to other supported targets 70 | - Sync metadata columns behaviour to other supported targets 71 | - Bump `psycopg2` to 2.8.2 72 | 73 | 1.0.0 (2019-06-03) 74 | ------------------- 75 | 76 | - Initial release 77 | -------------------------------------------------------------------------------- /tests/integration/resources/messages-with-invalid-records.json: -------------------------------------------------------------------------------- 1 | {"type": "STATE", "value": {"currently_syncing": "tap_mysql_test-test_table_invalid_record"}} 2 | {"type": "SCHEMA", "stream": "tap_mysql_test-test_table_invalid_record", "schema": {"properties": {"c_pk": {"inclusion": "automatic", "minimum": -2147483648, "maximum": 2147483647, "type": ["null", "integer"]}, "c_varchar": {"inclusion": "available", "maxLength": 16, "type": ["null", "string"]}, "c_int": {"inclusion": "available", "minimum": -2147483648, "maximum": 2147483647, "type": ["null", "integer"]}}, "type": "object"}, "key_properties": ["c_pk"]} 3 | {"type": "ACTIVATE_VERSION", "stream": "tap_mysql_test-test_table_invalid_record", "version": 1} 4 | {"type": "RECORD", "stream": "tap_mysql_test-test_table_invalid_record", "record": {"c_pk": "Non Numeric PK", "c_varchar": "Hello World", "c_int": 1}, "version": 1, "time_extracted": "2019-01-31T15:51:50.215998Z"} 5 | {"type": "RECORD", "stream": "tap_mysql_test-test_table_invalid_record", "record": {"c_pk": 2, "c_varchar": "Hello Asia", "c_int": 2}, "version": 1, "time_extracted": "2019-01-31T15:51:50.215998Z"} 6 | {"type": "RECORD", "stream": "tap_mysql_test-test_table_invalid_record", "record": {"c_pk": 3, "c_varchar": "Hello Europe", "c_int": 3}, "version": 1, "time_extracted": "2019-01-31T15:51:50.215998Z"} 7 | {"type": "RECORD", "stream": "tap_mysql_test-test_table_invalid_record", "record": {"c_pk": 4, "c_varchar": "Hello Americas", "c_int": 4}, "version": 1, "time_extracted": "2019-01-31T15:51:50.215998Z"} 8 | {"type": "RECORD", "stream": "tap_mysql_test-test_table_invalid_record", "record": {"c_pk": 5, "c_varchar": "Hello Africa", "c_int": 5}, "version": 1, "time_extracted": "2019-01-31T15:51:50.215998Z"} 9 | {"type": "RECORD", "stream": "tap_mysql_test-test_table_invalid_record", "record": {"c_pk": 6, "c_varchar": "Hello Antarctica", "c_int": 6}, "version": 1, "time_extracted": "2019-01-31T15:51:50.215998Z"} 10 | {"type": "RECORD", "stream": "tap_mysql_test-test_table_invalid_record", "record": {"c_pk": 7, "c_varchar": "Hello Oceania", "c_int": 7}, "version": 1, "time_extracted": "2019-01-31T15:51:50.215998Z"} 11 | {"type": "RECORD", "stream": "tap_mysql_test-test_table_invalid_record", "record": {"c_pk": 8, "c_varchar": "Hello Australia", "c_int": 8}, "version": 1, "time_extracted": "2019-01-31T15:51:50.215998Z"} 12 | {"type": "STATE", "value": {"currently_syncing": "tap_mysql_test-test_table_invalid_record", "bookmarks": {"tap_mysql_test-test_table_one": {"initial_full_table_complete": true}, "tap_mysql_test-test_table_two": {"initial_full_table_complete": true}}}} 13 | {"type": "ACTIVATE_VERSION", "stream": "tap_mysql_test-test_table_invalid_record", "version": 1} 14 | {"type": "STATE", "value": {"currently_syncing": null, "bookmarks": {"tap_mysql_test-test_table_invalid_record": {"initial_full_table_complete": true}}}} 15 | -------------------------------------------------------------------------------- /tests/integration/resources/messages-with-reserved-name-as-table-name.json: -------------------------------------------------------------------------------- 1 | {"type": "STATE", "value": {"currently_syncing": "my_db-order"}} 2 | {"type": "SCHEMA", "stream": "my_db-order", "schema": {"properties": {"data": {"inclusion": "available", "format": "binary", "type": ["null", "string"]}, "new": {"inclusion": "automatic", "format": "binary", "type": ["null", "string"]}, "created_at": {"inclusion": "available", "format": "date-time", "type": ["null", "string"]}}, "type": "object"}, "key_properties": ["new"]} 3 | {"type": "ACTIVATE_VERSION", "stream": "my_db-order", "version": 1576670613163} 4 | {"type": "RECORD", "stream": "my_db-order", "record": {"data": "6461746132", "new": "706b32", "created_at": "2019-12-17T16:02:55+00:00"}, "version": 1576670613163, "time_extracted": "2019-12-18T12:03:33.174343Z"} 5 | {"type": "RECORD", "stream": "my_db-order", "record": {"data": "64617461313030", "new": "706b33", "created_at": "2019-12-18T11:46:38+00:00"}, "version": 1576670613163, "time_extracted": "2019-12-18T12:03:33.174343Z"} 6 | {"type": "RECORD", "stream": "my_db-order", "record": {"data": "6461746134", "new": "706b34", "created_at": "2019-12-17T16:32:22+00:00"}, "version": 1576670613163, "time_extracted": "2019-12-18T12:03:33.174343Z"} 7 | {"type": "STATE", "value": {"currently_syncing": "my_db-order", "bookmarks": {"my_db-order": {"version": 1576670613163}}}} 8 | {"type": "ACTIVATE_VERSION", "stream": "my_db-order", "version": 1576670613163} 9 | {"type": "STATE", "value": {"currently_syncing": null, "bookmarks": {"my_db-order": {"version": 1576670613163, "log_file": "mysql-bin.000004", "log_pos": 945}}}} 10 | {"type": "STATE", "value": {"currently_syncing": null, "bookmarks": {"my_db-order": {"version": 1576670613163, "log_file": "mysql-bin.000004", "log_pos": 945}}}} 11 | {"type": "SCHEMA", "stream": "my_db-order", "schema": {"properties": {"data": {"inclusion": "available", "format": "binary", "type": ["null", "string"]}, "created_at": {"inclusion": "available", "format": "date-time", "type": ["null", "string"]}, "new": {"inclusion": "automatic", "format": "binary", "type": ["null", "string"]}}, "type": "object"}, "key_properties": ["new"]} 12 | {"type": "RECORD", "stream": "my_db-order", "record": {"new": "706b35", "data": "6461746135", "created_at": "2019-12-18T13:19:20+00:00"}, "version": 1576670613163, "time_extracted": "2019-12-18T13:24:31.441849Z"} 13 | {"type": "RECORD", "stream": "my_db-order", "record": {"new": "706b35", "data": "64617461313030", "created_at": "2019-12-18T13:19:35+00:00"}, "version": 1576670613163, "time_extracted": "2019-12-18T13:24:31.441849Z"} 14 | {"type": "RECORD", "stream": "my_db-order", "record": {"new": "706b33", "data": "64617461313030", "created_at": "2019-12-18T11:46:38+00:00", "_sdc_deleted_at": "2019-12-18T13:19:44+00:00+00:00"}, "version": 1576670613163, "time_extracted": "2019-12-18T13:24:31.441849Z"} 15 | {"type": "RECORD", "stream": "my_db-order", "record": {"new": "706b35", "data": "64617461313030", "created_at": "2019-12-18T13:19:35+00:00", "_sdc_deleted_at": "2019-12-18T13:19:44+00:00+00:00"}, "version": 1576670613163, "time_extracted": "2019-12-18T13:24:31.441849Z"} 16 | {"type": "STATE", "value": {"currently_syncing": null, "bookmarks": {"my_db-order": {"version": 1576670613163, "log_file": "mysql-bin.000004", "log_pos": 1867}}}} 17 | -------------------------------------------------------------------------------- /tests/integration/resources/messages-with-unicode-characters.json: -------------------------------------------------------------------------------- 1 | {"type": "STATE", "value": {"currently_syncing": "tap_mysql_test-test_table_unicode"}} 2 | {"type": "SCHEMA", "stream": "tap_mysql_test-test_table_unicode", "schema": {"properties": {"c_pk": {"inclusion": "automatic", "minimum": -2147483648, "maximum": 2147483647, "type": ["null", "integer"]}, "c_varchar": {"inclusion": "available", "maxLength": 16, "type": ["null", "string"]}, "c_int": {"inclusion": "available", "minimum": -2147483648, "maximum": 2147483647, "type": ["null", "integer"]}}, "type": "object"}, "key_properties": ["c_pk"]} 3 | {"type": "ACTIVATE_VERSION", "stream": "tap_mysql_test-test_table_unicode", "version": 1} 4 | {"type": "RECORD", "stream": "tap_mysql_test-test_table_unicode", "record": {"c_pk": 1, "c_varchar": "Hello world, \u039a\u03b1\u03bb\u03b7\u03bc\u1f73\u03c1\u03b1 \u03ba\u1f79\u03c3\u03bc\u03b5, \u30b3\u30f3\u30cb\u30c1\u30cf", "c_int": 1}, "version": 1, "time_extracted": "2019-01-31T15:51:50.215998Z"} 5 | {"type": "RECORD", "stream": "tap_mysql_test-test_table_unicode", "record": {"c_pk": 2, "c_varchar": "Chinese: \u548c\u6bdb\u6cfd\u4e1c <<\u91cd\u4e0a\u4e95\u5188\u5c71>>. \u4e25\u6c38\u6b23, \u4e00\u4e5d\u516b\u516b\u5e74.", "c_int": 2}, "version": 1, "time_extracted": "2019-01-31T15:51:50.215998Z"} 6 | {"type": "RECORD", "stream": "tap_mysql_test-test_table_unicode", "record": {"c_pk": 3, "c_varchar": "Russian: \u0417\u0430\u0440\u0435\u0433\u0438\u0441\u0442\u0440\u0438\u0440\u0443\u0439\u0442\u0435\u0441\u044c \u0441\u0435\u0439\u0447\u0430\u0441 \u043d\u0430 \u0414\u0435\u0441\u044f\u0442\u0443\u044e \u041c\u0435\u0436\u0434\u0443\u043d\u0430\u0440\u043e\u0434\u043d\u0443\u044e \u041a\u043e\u043d\u0444\u0435\u0440\u0435\u043d\u0446\u0438\u044e \u043f\u043e", "c_int": 3}, "version": 1, "time_extracted": "2019-01-31T15:51:50.215998Z"} 7 | {"type": "RECORD", "stream": "tap_mysql_test-test_table_unicode", "record": {"c_pk": 4, "c_varchar": "Thai: \u0e41\u0e1c\u0e48\u0e19\u0e14\u0e34\u0e19\u0e2e\u0e31\u0e48\u0e19\u0e40\u0e2a\u0e37\u0e48\u0e2d\u0e21\u0e42\u0e17\u0e23\u0e21\u0e41\u0e2a\u0e19\u0e2a\u0e31\u0e07\u0e40\u0e27\u0e0a", "c_int": 4, "_sdc_deleted_at": "2019-02-10T15:51:50.215998Z"}, "version": 1, "time_extracted": "2019-01-31T15:51:50.215998Z"} 8 | {"type": "RECORD", "stream": "tap_mysql_test-test_table_unicode", "record": {"c_pk": 5, "c_varchar": "Arabic: \u0644\u0642\u062f \u0644\u0639\u0628\u062a \u0623\u0646\u062a \u0648\u0623\u0635\u062f\u0642\u0627\u0624\u0643 \u0644\u0645\u062f\u0629 \u0648\u062d\u0635\u0644\u062a\u0645 \u0639\u0644\u064a \u0645\u0646 \u0625\u062c\u0645\u0627\u0644\u064a \u0627\u0644\u0646\u0642\u0627\u0637", "c_int": 5, "_sdc_deleted_at": "2019-02-10T15:51:50.215998Z"}, "version": 1, "time_extracted": "2019-01-31T15:51:50.215998Z"} 9 | {"type": "RECORD", "stream": "tap_mysql_test-test_table_unicode", "record": {"c_pk": 6, "c_varchar": "Special Characters: [\",'!@£$%^&*()]", "c_int": 6}, "version": 1, "time_extracted": "2019-01-31T15:51:50.215998Z"} 10 | {"type": "STATE", "value": {"currently_syncing": "tap_mysql_test-test_table_unicode", "bookmarks": {"tap_mysql_test-test_table_one": {"initial_full_table_complete": true}, "tap_mysql_test-test_table_two": {"initial_full_table_complete": true}}}} 11 | {"type": "ACTIVATE_VERSION", "stream": "tap_mysql_test-test_table_unicode", "version": 1} 12 | {"type": "STATE", "value": {"currently_syncing": null, "bookmarks": {"tap_mysql_test-test_table_unicode": {"initial_full_table_complete": true}}}} 13 | -------------------------------------------------------------------------------- /tests/integration/resources/messages-with-space-in-table-name.json: -------------------------------------------------------------------------------- 1 | {"type": "STATE", "value": {"currently_syncing": "my_db-table with space and UPPERCase"}} 2 | {"type": "SCHEMA", "stream": "my_db-table with space and UPPERCase", "schema": {"properties": {"data": {"inclusion": "available", "format": "binary", "type": ["null", "string"]}, "new": {"inclusion": "automatic", "format": "binary", "type": ["null", "string"]}, "created_at": {"inclusion": "available", "format": "date-time", "type": ["null", "string"]}}, "type": "object"}, "key_properties": ["new"]} 3 | {"type": "ACTIVATE_VERSION", "stream": "my_db-table with space and UPPERCase", "version": 1576670613163} 4 | {"type": "RECORD", "stream": "my_db-table with space and UPPERCase", "record": {"data": "6461746132", "new": "706b32", "created_at": "2019-12-17T16:02:55+00:00"}, "version": 1576670613163, "time_extracted": "2019-12-18T12:03:33.174343Z"} 5 | {"type": "RECORD", "stream": "my_db-table with space and UPPERCase", "record": {"data": "64617461313030", "new": "706b33", "created_at": "2019-12-18T11:46:38+00:00"}, "version": 1576670613163, "time_extracted": "2019-12-18T12:03:33.174343Z"} 6 | {"type": "RECORD", "stream": "my_db-table with space and UPPERCase", "record": {"data": "6461746134", "new": "706b34", "created_at": "2019-12-17T16:32:22+00:00"}, "version": 1576670613163, "time_extracted": "2019-12-18T12:03:33.174343Z"} 7 | {"type": "STATE", "value": {"currently_syncing": "my_db-table with space and UPPERCase", "bookmarks": {"my_db-table with space and UPPERCase": {"version": 1576670613163}}}} 8 | {"type": "ACTIVATE_VERSION", "stream": "my_db-table with space and UPPERCase", "version": 1576670613163} 9 | {"type": "STATE", "value": {"currently_syncing": null, "bookmarks": {"my_db-table with space and UPPERCase": {"version": 1576670613163, "log_file": "mysql-bin.000004", "log_pos": 945}}}} 10 | {"type": "STATE", "value": {"currently_syncing": null, "bookmarks": {"my_db-table with space and UPPERCase": {"version": 1576670613163, "log_file": "mysql-bin.000004", "log_pos": 945}}}} 11 | {"type": "SCHEMA", "stream": "my_db-table with space and UPPERCase", "schema": {"properties": {"data": {"inclusion": "available", "format": "binary", "type": ["null", "string"]}, "created_at": {"inclusion": "available", "format": "date-time", "type": ["null", "string"]}, "new": {"inclusion": "automatic", "format": "binary", "type": ["null", "string"]}}, "type": "object"}, "key_properties": ["new"]} 12 | {"type": "RECORD", "stream": "my_db-table with space and UPPERCase", "record": {"new": "706b35", "data": "6461746135", "created_at": "2019-12-18T13:19:20+00:00"}, "version": 1576670613163, "time_extracted": "2019-12-18T13:24:31.441849Z"} 13 | {"type": "RECORD", "stream": "my_db-table with space and UPPERCase", "record": {"new": "706b35", "data": "64617461313030", "created_at": "2019-12-18T13:19:35+00:00"}, "version": 1576670613163, "time_extracted": "2019-12-18T13:24:31.441849Z"} 14 | {"type": "RECORD", "stream": "my_db-table with space and UPPERCase", "record": {"new": "706b33", "data": "64617461313030", "created_at": "2019-12-18T11:46:38+00:00", "_sdc_deleted_at": "2019-12-18T13:19:44+00:00+00:00"}, "version": 1576670613163, "time_extracted": "2019-12-18T13:24:31.441849Z"} 15 | {"type": "RECORD", "stream": "my_db-table with space and UPPERCase", "record": {"new": "706b35", "data": "64617461313030", "created_at": "2019-12-18T13:19:35+00:00", "_sdc_deleted_at": "2019-12-18T13:19:44+00:00+00:00"}, "version": 1576670613163, "time_extracted": "2019-12-18T13:24:31.441849Z"} 16 | {"type": "STATE", "value": {"currently_syncing": null, "bookmarks": {"my_db-table with space and UPPERCase": {"version": 1576670613163, "log_file": "mysql-bin.000004", "log_pos": 1867}}}} 17 | -------------------------------------------------------------------------------- /tests/integration/resources/messages-with-multiple-streams-modified-column.json: -------------------------------------------------------------------------------- 1 | {"type": "STATE", "value": {"currently_syncing": "tap_mysql_test-test_table_one"}} 2 | {"type": "SCHEMA", "stream": "tap_mysql_test-test_table_one", "schema": {"properties": {"c_pk": {"inclusion": "automatic", "minimum": -2147483648, "maximum": 2147483647, "type": ["null", "integer"]}, "c_varchar": {"inclusion": "available", "maxLength": 16, "type": ["null", "string"]}, "c_int": {"inclusion": "available", "minimum": -2147483648, "maximum": 2147483647, "type": ["null", "integer"]}}, "type": "object"}, "key_properties": ["c_pk"]} 3 | {"type": "ACTIVATE_VERSION", "stream": "tap_mysql_test-test_table_one", "version": 1} 4 | {"type": "RECORD", "stream": "tap_mysql_test-test_table_one", "record": {"c_pk": 1, "c_varchar": "1", "c_int": 1}, "version": 1, "time_extracted": "2019-01-31T15:51:47.465408Z"} 5 | {"type": "STATE", "value": {"currently_syncing": "tap_mysql_test-test_table_one"}} 6 | {"type": "ACTIVATE_VERSION", "stream": "tap_mysql_test-test_table_one", "version": 1} 7 | {"type": "STATE", "value": {"currently_syncing": "tap_mysql_test-test_table_one", "bookmarks": {"tap_mysql_test-test_table_one": {"initial_full_table_complete": true}}}} 8 | {"type": "STATE", "value": {"currently_syncing": "tap_mysql_test-test_table_two", "bookmarks": {"tap_mysql_test-test_table_two": {"initial_full_table_complete": true}}}} 9 | {"type": "SCHEMA", "stream": "tap_mysql_test-test_table_two", "schema": {"properties": {"c_pk": {"inclusion": "automatic", "minimum": -2147483648, "maximum": 2147483647, "type": ["null", "integer"]}, "c_varchar": {"inclusion": "available", "maxLength": 16, "type": ["null", "string"]}, "c_int": {"inclusion": "available", "minimum": -2147483648, "maximum": 2147483647, "type": ["null", "integer"]}, "c_date": {"inclusion": "available", "type": ["null", "string"]}}, "type": "object"}, "key_properties": ["c_pk"]} 10 | {"type": "ACTIVATE_VERSION", "stream": "tap_mysql_test-test_table_two", "version": 3} 11 | {"type": "RECORD", "stream": "tap_mysql_test-test_table_two", "record": {"c_pk": 2, "c_varchar": "2", "c_int": 2, "c_date": "2019-02-12 02:00:00"}, "version": 3, "time_extracted": "2019-01-31T15:51:48.861962Z"} 12 | {"type": "RECORD", "stream": "tap_mysql_test-test_table_two", "record": {"c_pk": 3, "c_varchar": "2", "c_int": 3, "c_date": "2019-02-15 02:00:00"}, "version": 3, "time_extracted": "2019-01-31T15:51:48.861962Z"} 13 | {"type": "STATE", "value": {"currently_syncing": "tap_mysql_test-test_table_two", "bookmarks": {"tap_mysql_test-test_table_wo": {"initial_full_table_complete": true}}}} 14 | {"type": "ACTIVATE_VERSION", "stream": "tap_mysql_test-test_table_three", "version": 3} 15 | {"type": "STATE", "value": {"currently_syncing": "tap_mysql_test-test_table_two", "bookmarks": {"tap_mysql_test-test_table_one": {"initial_full_table_complete": true}, "tap_mysql_test-test_table_three": {"initial_full_table_complete": true}}}} 16 | {"type": "STATE", "value": {"currently_syncing": "tap_mysql_test-test_table_three", "bookmarks": {"tap_mysql_test-test_table_one": {"initial_full_table_complete": true}, "tap_mysql_test-test_table_three": {"initial_full_table_complete": true}}}} 17 | {"type": "SCHEMA", "stream": "tap_mysql_test-test_table_three", "schema": {"properties": {"c_pk": {"inclusion": "automatic", "minimum": -2147483648, "maximum": 2147483647, "type": ["null", "integer"]}, "c_varchar": {"inclusion": "available", "maxLength": 16, "type": ["null", "string"]}, "c_int": {"inclusion": "available", "minimum": -2147483648, "maximum": 2147483647, "type": ["null", "integer"]}, "c_time_renamed": {"format": "time", "inclusion": "available", "type": ["null", "string"]}}, "type": "object"}, "key_properties": ["c_pk"]} 18 | {"type": "ACTIVATE_VERSION", "stream": "tap_mysql_test-test_table_three", "version": 2} 19 | {"type": "RECORD", "stream": "tap_mysql_test-test_table_three", "record": {"c_pk": 3, "c_varchar": "3", "c_int": 3, "c_time_renamed": "08:15:00"}, "version": 2, "time_extracted": "2019-01-31T15:51:50.215998Z"} 20 | {"type": "RECORD", "stream": "tap_mysql_test-test_table_three", "record": {"c_pk": 4, "c_varchar": "4", "c_int": 4, "c_time_renamed": "23:00:03", "_sdc_deleted_at": "2019-02-10T15:51:50.215998Z"}, "version": 2, "time_extracted": "2019-01-31T15:51:50.215998Z"} 21 | {"type": "STATE", "value": {"currently_syncing": "tap_mysql_test-test_table_three", "bookmarks": {"tap_mysql_test-test_table_one": {"initial_full_table_complete": true}, "tap_mysql_test-test_table_two": {"initial_full_table_complete": true}}}} 22 | {"type": "ACTIVATE_VERSION", "stream": "tap_mysql_test-test_table_three", "version": 2} 23 | {"type": "STATE", "value": {"currently_syncing": "tap_mysql_test-test_table_three", "bookmarks": {"tap_mysql_test-test_table_one": {"initial_full_table_complete": true}, "tap_mysql_test-test_table_two": {"initial_full_table_complete": true}, "tap_mysql_test-test_table_three": {"initial_full_table_complete": true}}}} 24 | {"type": "STATE", "value": {"currently_syncing": null, "bookmarks": {"tap_mysql_test-test_table_one": {"initial_full_table_complete": true}, "tap_mysql_test-test_table_two": {"initial_full_table_complete": true}, "tap_mysql_test-test_table_three": {"initial_full_table_complete": true}}}} 25 | -------------------------------------------------------------------------------- /tests/integration/resources/messages-with-multiple-streams.json: -------------------------------------------------------------------------------- 1 | {"type": "STATE", "value": {"currently_syncing": "tap_mysql_test-test_table_one"}} 2 | {"type": "SCHEMA", "stream": "tap_mysql_test-test_table_one", "schema": {"properties": {"c_pk": {"inclusion": "automatic", "minimum": -2147483648, "maximum": 2147483647, "type": ["null", "integer"]}, "c_varchar": {"inclusion": "available", "maxLength": 16, "type": ["null", "string"]}, "c_int": {"inclusion": "available", "minimum": -2147483648, "maximum": 2147483647, "type": ["null", "integer"]}}, "type": "object"}, "key_properties": ["c_pk"]} 3 | {"type": "ACTIVATE_VERSION", "stream": "tap_mysql_test-test_table_one", "version": 1} 4 | {"type": "RECORD", "stream": "tap_mysql_test-test_table_one", "record": {"c_pk": 1, "c_varchar": "1", "c_int": 1}, "version": 1, "time_extracted": "2019-01-31T15:51:47.465408Z"} 5 | {"type": "STATE", "value": {"currently_syncing": "tap_mysql_test-test_table_one"}} 6 | {"type": "ACTIVATE_VERSION", "stream": "tap_mysql_test-test_table_one", "version": 1} 7 | {"type": "STATE", "value": {"currently_syncing": "tap_mysql_test-test_table_one", "bookmarks": {"tap_mysql_test-test_table_one": {"initial_full_table_complete": true}}}} 8 | {"type": "STATE", "value": {"currently_syncing": "tap_mysql_test-test_table_two", "bookmarks": {"tap_mysql_test-test_table_two": {"initial_full_table_complete": true}}}} 9 | {"type": "SCHEMA", "stream": "tap_mysql_test-test_table_two", "schema": {"properties": {"c_pk": {"inclusion": "automatic", "minimum": -2147483648, "maximum": 2147483647, "type": ["null", "integer"]}, "c_varchar": {"inclusion": "available", "maxLength": 16, "type": ["null", "string"]}, "c_int": {"inclusion": "available", "minimum": -2147483648, "maximum": 2147483647, "type": ["null", "integer"]}, "c_date": {"format": "date-time", "inclusion": "available", "type": ["null", "string"]}}, "type": "object"}, "key_properties": ["c_pk"]} 10 | {"type": "ACTIVATE_VERSION", "stream": "tap_mysql_test-test_table_two", "version": 3} 11 | {"type": "RECORD", "stream": "tap_mysql_test-test_table_two", "record": {"c_pk": 1, "c_varchar": "1", "c_int": 1, "c_date": "2019-02-01 15:12:45", "_sdc_deleted_at": "2019-02-12T01:10:10.000000Z"}, "version": 3, "time_extracted": "2019-01-31T15:51:48.861962Z"} 12 | {"type": "RECORD", "stream": "tap_mysql_test-test_table_two", "record": {"c_pk": 2, "c_varchar": "2", "c_int": 2, "c_date": "2019-02-10 02:00:00"}, "version": 3, "time_extracted": "2019-01-31T15:51:48.861962Z"} 13 | {"type": "STATE", "value": {"currently_syncing": "tap_mysql_test-test_table_two", "bookmarks": {"tap_mysql_test-test_table_wo": {"initial_full_table_complete": true}}}} 14 | {"type": "ACTIVATE_VERSION", "stream": "tap_mysql_test-test_table_three", "version": 3} 15 | {"type": "STATE", "value": {"currently_syncing": "tap_mysql_test-test_table_two", "bookmarks": {"tap_mysql_test-test_table_one": {"initial_full_table_complete": true}, "tap_mysql_test-test_table_three": {"initial_full_table_complete": true}}}} 16 | {"type": "STATE", "value": {"currently_syncing": "tap_mysql_test-test_table_three", "bookmarks": {"tap_mysql_test-test_table_one": {"initial_full_table_complete": true}, "tap_mysql_test-test_table_three": {"initial_full_table_complete": true}}}} 17 | {"type": "SCHEMA", "stream": "tap_mysql_test-test_table_three", "schema": {"properties": {"c_pk": {"inclusion": "automatic", "minimum": -2147483648, "maximum": 2147483647, "type": ["null", "integer"]}, "c_varchar": {"inclusion": "available", "maxLength": 16, "type": ["null", "string"]}, "c_int": {"inclusion": "available", "minimum": -2147483648, "maximum": 2147483647, "type": ["null", "integer"]}, "c_time": {"format": "time", "inclusion": "available", "type": ["null", "string"]}}, "type": "object"}, "key_properties": ["c_pk"]} 18 | {"type": "ACTIVATE_VERSION", "stream": "tap_mysql_test-test_table_three", "version": 2} 19 | {"type": "RECORD", "stream": "tap_mysql_test-test_table_three", "record": {"c_pk": 1, "c_varchar": "1", "c_int": 1, "c_time": "04:00:00"}, "version": 2, "time_extracted": "2019-01-31T15:51:50.215998Z"} 20 | {"type": "RECORD", "stream": "tap_mysql_test-test_table_three", "record": {"c_pk": 2, "c_varchar": "2", "c_int": 2, "c_time": "07:15:00"}, "version": 2, "time_extracted": "2019-01-31T15:51:50.215998Z"} 21 | {"type": "RECORD", "stream": "tap_mysql_test-test_table_three", "record": {"c_pk": 3, "c_varchar": "3", "c_int": 3, "c_time": "23:00:03", "_sdc_deleted_at": "2019-02-10T15:51:50.215998Z"}, "version": 2, "time_extracted": "2019-01-31T15:51:50.215998Z"} 22 | {"type": "STATE", "value": {"currently_syncing": "tap_mysql_test-test_table_three", "bookmarks": {"tap_mysql_test-test_table_one": {"initial_full_table_complete": true}, "tap_mysql_test-test_table_two": {"initial_full_table_complete": true}}}} 23 | {"type": "ACTIVATE_VERSION", "stream": "tap_mysql_test-test_table_three", "version": 2} 24 | {"type": "STATE", "value": {"currently_syncing": "tap_mysql_test-test_table_three", "bookmarks": {"tap_mysql_test-test_table_one": {"initial_full_table_complete": true}, "tap_mysql_test-test_table_two": {"initial_full_table_complete": true}, "tap_mysql_test-test_table_three": {"initial_full_table_complete": true}}}} 25 | {"type": "STATE", "value": {"currently_syncing": null, "bookmarks": {"tap_mysql_test-test_table_one": {"initial_full_table_complete": true}, "tap_mysql_test-test_table_two": {"initial_full_table_complete": true}, "tap_mysql_test-test_table_three": {"initial_full_table_complete": true}}}} 26 | {"type": "SCHEMA", "stream": "tap_mysql_test-test_table_four", "schema": {"properties": {"c_pk": {"inclusion": "automatic", "minimum": -2147483648, "maximum": 2147483647, "type": ["null", "integer"]}, "c_smallint": {"inclusion": "automatic", "minimum": -32768, "maximum": 32767, "type": ["null", "integer"]}, "c_integer": {"inclusion": "automatic", "minimum": -2147483648, "maximum": 2147483647, "type": ["null", "integer"]}, "c_bigint": {"inclusion": "automatic", "minimum": -9223372036854775808, "maximum": 9223372036854775807, "type": ["null", "integer"]}, "c_nobound_int": {"inclusion": "automatic", "type": ["null", "integer"]}}, "type": "object"}, "key_properties": ["c_pk"]} 27 | {"type": "ACTIVATE_VERSION", "stream": "tap_mysql_test-test_table_four", "version": 2} 28 | {"type": "RECORD", "stream": "tap_mysql_test-test_table_four", "record": {"c_pk": 1, "c_smallint": 1, "c_integer": 1, "c_bigint": 1, "c_nobound_int": 1}, "version": 2, "time_extracted": "2019-01-31T15:51:50.215998Z"} 29 | {"type": "RECORD", "stream": "tap_mysql_test-test_table_four", "record": {"c_pk": 2, "c_smallint": 2, "c_integer": 2, "c_bigint": 2, "c_nobound_int": 2}, "version": 2, "time_extracted": "2019-01-31T15:51:50.215998Z"} 30 | {"type": "RECORD", "stream": "tap_mysql_test-test_table_four", "record": {"c_pk": 3, "c_smallint": 3, "c_integer": 3, "c_bigint": 3, "c_nobound_int": 3, "_sdc_deleted_at": "2019-02-10T15:51:50.215998Z"}, "version": 2, "time_extracted": "2019-01-31T15:51:50.215998Z"} 31 | {"type": "STATE", "value": {"currently_syncing": "tap_mysql_test-test_table_four", "bookmarks": {"tap_mysql_test-test_table_one": {"initial_full_table_complete": true}, "tap_mysql_test-test_table_two": {"initial_full_table_complete": true}}}} 32 | {"type": "ACTIVATE_VERSION", "stream": "tap_mysql_test-test_table_four", "version": 2} 33 | {"type": "STATE", "value": {"currently_syncing": "tap_mysql_test-test_table_four", "bookmarks": {"tap_mysql_test-test_table_one": {"initial_full_table_complete": true}, "tap_mysql_test-test_table_two": {"initial_full_table_complete": true}, "tap_mysql_test-test_table_four": {"initial_full_table_complete": true}}}} 34 | {"type": "STATE", "value": {"currently_syncing": null, "bookmarks": {"tap_mysql_test-test_table_one": {"initial_full_table_complete": true}, "tap_mysql_test-test_table_two": {"initial_full_table_complete": true}, "tap_mysql_test-test_table_four": {"initial_full_table_complete": true}}}} 35 | -------------------------------------------------------------------------------- /tests/integration/resources/messages-pg-logical-streams-no-records.json: -------------------------------------------------------------------------------- 1 | {"type": "SCHEMA", "stream": "logical1-logical1_edgydata", "schema": {"definitions": {"sdc_recursive_boolean_array": {"items": {"$ref": "#/definitions/sdc_recursive_boolean_array"}, "type": ["null", "boolean", "array"]}, "sdc_recursive_integer_array": {"items": {"$ref": "#/definitions/sdc_recursive_integer_array"}, "type": ["null", "integer", "array"]}, "sdc_recursive_number_array": {"items": {"$ref": "#/definitions/sdc_recursive_number_array"}, "type": ["null", "number", "array"]}, "sdc_recursive_object_array": {"items": {"$ref": "#/definitions/sdc_recursive_object_array"}, "type": ["null", "object", "array"]}, "sdc_recursive_string_array": {"items": {"$ref": "#/definitions/sdc_recursive_string_array"}, "type": ["null", "string", "array"]}, "sdc_recursive_timestamp_array": {"format": "date-time", "items": {"$ref": "#/definitions/sdc_recursive_timestamp_array"}, "type": ["null", "string", "array"]}}, "properties": {"cid": {"maximum": 2147483647, "minimum": -2147483648, "type": ["integer"]}, "cjson": {"type": ["null", "object"]}, "cjsonb": {"type": ["null", "object"]}, "ctimentz": {"format": "time", "type": ["null", "string"]}, "ctimetz": {"format": "time", "type": ["null", "string"]}, "cvarchar": {"type": ["null", "string"]}, "_sdc_deleted_at": {"type": ["null", "string"], "format": "date-time"}}, "type": "object"}, "key_properties": ["cid"], "bookmark_properties": ["lsn"]} 2 | {"type": "SCHEMA", "stream": "logical1-logical1_table1", "schema": {"definitions": {"sdc_recursive_boolean_array": {"items": {"$ref": "#/definitions/sdc_recursive_boolean_array"}, "type": ["null", "boolean", "array"]}, "sdc_recursive_integer_array": {"items": {"$ref": "#/definitions/sdc_recursive_integer_array"}, "type": ["null", "integer", "array"]}, "sdc_recursive_number_array": {"items": {"$ref": "#/definitions/sdc_recursive_number_array"}, "type": ["null", "number", "array"]}, "sdc_recursive_object_array": {"items": {"$ref": "#/definitions/sdc_recursive_object_array"}, "type": ["null", "object", "array"]}, "sdc_recursive_string_array": {"items": {"$ref": "#/definitions/sdc_recursive_string_array"}, "type": ["null", "string", "array"]}, "sdc_recursive_timestamp_array": {"format": "date-time", "items": {"$ref": "#/definitions/sdc_recursive_timestamp_array"}, "type": ["null", "string", "array"]}}, "properties": {"cid": {"maximum": 2147483647, "minimum": -2147483648, "type": ["integer"]}, "cvarchar": {"type": ["null", "string"]}, "cvarchar2": {"type": ["null", "string"]}, "_sdc_deleted_at": {"type": ["null", "string"], "format": "date-time"}}, "type": "object"}, "key_properties": ["cid"], "bookmark_properties": ["lsn"]} 3 | {"type": "SCHEMA", "stream": "logical1-logical1_table2", "schema": {"definitions": {"sdc_recursive_boolean_array": {"items": {"$ref": "#/definitions/sdc_recursive_boolean_array"}, "type": ["null", "boolean", "array"]}, "sdc_recursive_integer_array": {"items": {"$ref": "#/definitions/sdc_recursive_integer_array"}, "type": ["null", "integer", "array"]}, "sdc_recursive_number_array": {"items": {"$ref": "#/definitions/sdc_recursive_number_array"}, "type": ["null", "number", "array"]}, "sdc_recursive_object_array": {"items": {"$ref": "#/definitions/sdc_recursive_object_array"}, "type": ["null", "object", "array"]}, "sdc_recursive_string_array": {"items": {"$ref": "#/definitions/sdc_recursive_string_array"}, "type": ["null", "string", "array"]}, "sdc_recursive_timestamp_array": {"format": "date-time", "items": {"$ref": "#/definitions/sdc_recursive_timestamp_array"}, "type": ["null", "string", "array"]}}, "properties": {"cid": {"maximum": 2147483647, "minimum": -2147483648, "type": ["integer"]}, "cvarchar": {"type": ["null", "string"]}, "_sdc_deleted_at": {"type": ["null", "string"], "format": "date-time"}}, "type": "object"}, "key_properties": ["cid"], "bookmark_properties": ["lsn"]} 4 | {"type": "SCHEMA", "stream": "logical2-logical2_table1", "schema": {"definitions": {"sdc_recursive_boolean_array": {"items": {"$ref": "#/definitions/sdc_recursive_boolean_array"}, "type": ["null", "boolean", "array"]}, "sdc_recursive_integer_array": {"items": {"$ref": "#/definitions/sdc_recursive_integer_array"}, "type": ["null", "integer", "array"]}, "sdc_recursive_number_array": {"items": {"$ref": "#/definitions/sdc_recursive_number_array"}, "type": ["null", "number", "array"]}, "sdc_recursive_object_array": {"items": {"$ref": "#/definitions/sdc_recursive_object_array"}, "type": ["null", "object", "array"]}, "sdc_recursive_string_array": {"items": {"$ref": "#/definitions/sdc_recursive_string_array"}, "type": ["null", "string", "array"]}, "sdc_recursive_timestamp_array": {"format": "date-time", "items": {"$ref": "#/definitions/sdc_recursive_timestamp_array"}, "type": ["null", "string", "array"]}}, "properties": {"cid": {"maximum": 2147483647, "minimum": -2147483648, "type": ["integer"]}, "cvarchar": {"type": ["null", "string"]}, "_sdc_deleted_at": {"type": ["null", "string"], "format": "date-time"}}, "type": "object"}, "key_properties": ["cid"], "bookmark_properties": ["lsn"]} 5 | {"type": "STATE", "value": {"currently_syncing": null, "bookmarks": {"logical1-logical1_edgydata": {"last_replication_method": "LOG_BASED", "lsn": 108196176, "version": 1570922723596, "xmin": null}, "logical1-logical1_table1": {"last_replication_method": "LOG_BASED", "lsn": 108196176, "version": 1570922723618, "xmin": null}, "logical1-logical1_table2": {"last_replication_method": "LOG_BASED", "lsn": 108196176, "version": 1570922723635, "xmin": null}, "logical2-logical2_table1": {"last_replication_method": "LOG_BASED", "lsn": 108196176, "version": 1570922723651, "xmin": null}, "public-city": {"last_replication_method": "INCREMENTAL", "replication_key": "id", "version": 1570922723667, "replication_key_value": 4079}, "public-country": {"last_replication_method": "FULL_TABLE", "version": 1570922730456, "xmin": null}, "public2-wearehere": {}}}} 6 | {"type": "STATE", "value": {"currently_syncing": null, "bookmarks": {"logical1-logical1_edgydata": {"last_replication_method": "LOG_BASED", "lsn": 108196760, "version": 1570922723596, "xmin": null}, "logical1-logical1_table1": {"last_replication_method": "LOG_BASED", "lsn": 108196760, "version": 1570922723618, "xmin": null}, "logical1-logical1_table2": {"last_replication_method": "LOG_BASED", "lsn": 108196760, "version": 1570922723635, "xmin": null}, "logical2-logical2_table1": {"last_replication_method": "LOG_BASED", "lsn": 108196760, "version": 1570922723651, "xmin": null}, "public-city": {"last_replication_method": "INCREMENTAL", "replication_key": "id", "version": 1570922723667, "replication_key_value": 4079}, "public-country": {"last_replication_method": "FULL_TABLE", "version": 1570922730456, "xmin": null}, "public2-wearehere": {}}}} 7 | {"type": "STATE", "value": {"currently_syncing": null, "bookmarks": {"logical1-logical1_edgydata": {"last_replication_method": "LOG_BASED", "lsn": 108197216, "version": 1570922723596, "xmin": null}, "logical1-logical1_table1": {"last_replication_method": "LOG_BASED", "lsn": 108197216, "version": 1570922723618, "xmin": null}, "logical1-logical1_table2": {"last_replication_method": "LOG_BASED", "lsn": 108197216, "version": 1570922723635, "xmin": null}, "logical2-logical2_table1": {"last_replication_method": "LOG_BASED", "lsn": 108197216, "version": 1570922723651, "xmin": null}, "public-city": {"last_replication_method": "INCREMENTAL", "replication_key": "id", "version": 1570922723667, "replication_key_value": 4079}, "public-country": {"last_replication_method": "FULL_TABLE", "version": 1570922730456, "xmin": null}, "public2-wearehere": {}}}} 8 | {"type": "STATE", "value": {"currently_syncing": null, "bookmarks": {"logical1-logical1_edgydata": {"last_replication_method": "LOG_BASED", "lsn": 108197672, "version": 1570922723596, "xmin": null}, "logical1-logical1_table1": {"last_replication_method": "LOG_BASED", "lsn": 108197672, "version": 1570922723618, "xmin": null}, "logical1-logical1_table2": {"last_replication_method": "LOG_BASED", "lsn": 108197672, "version": 1570922723635, "xmin": null}, "logical2-logical2_table1": {"last_replication_method": "LOG_BASED", "lsn": 108197672, "version": 1570922723651, "xmin": null}, "public-city": {"last_replication_method": "INCREMENTAL", "replication_key": "id", "version": 1570922723667, "replication_key_value": 4079}, "public-country": {"last_replication_method": "FULL_TABLE", "version": 1570922730456, "xmin": null}, "public2-wearehere": {}}}} 9 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Notice 2 | To better serve Wise business and customer needs, the PipelineWise codebase needs to shrink. 3 | We have made the difficult decision that, going forward many components of PipelineWise will be removed or incorporated in the main repo. 4 | The last version before this decision is [v0.64.1](https://github.com/transferwise/pipelinewise/tree/v0.64.1) 5 | 6 | We thank all in the open-source community, that over the past 6 years, have helped to make PipelineWise a robust product for heterogeneous replication of many many Terabytes, daily 7 | 8 | # pipelinewise-target-postgres 9 | 10 | [![PyPI version](https://badge.fury.io/py/pipelinewise-target-postgres.svg)](https://badge.fury.io/py/pipelinewise-target-postgres) 11 | [![PyPI - Python Version](https://img.shields.io/pypi/pyversions/pipelinewise-target-postgres.svg)](https://pypi.org/project/pipelinewise-target-postgres/) 12 | [![License: Apache2](https://img.shields.io/badge/License-Apache2-yellow.svg)](https://opensource.org/licenses/Apache-2.0) 13 | 14 | [Singer](https://www.singer.io/) target that loads data into PostgreSQL following the [Singer spec](https://github.com/singer-io/getting-started/blob/master/docs/SPEC.md). 15 | 16 | This is a [PipelineWise](https://transferwise.github.io/pipelinewise) compatible target connector. 17 | 18 | ## How to use it 19 | 20 | The recommended method of running this target is to use it from [PipelineWise](https://transferwise.github.io/pipelinewise). When running it from PipelineWise you don't need to configure this tap with JSON files and most of things are automated. Please check the related documentation at [Target Postgres](https://transferwise.github.io/pipelinewise/connectors/targets/postgres.html) 21 | 22 | If you want to run this [Singer Target](https://singer.io) independently please read further. 23 | 24 | ### Install 25 | 26 | First, make sure Python 3 is installed on your system or follow these 27 | installation instructions for [Mac](http://docs.python-guide.org/en/latest/starting/install3/osx/) or 28 | [Ubuntu](https://www.digitalocean.com/community/tutorials/how-to-install-python-3-and-set-up-a-local-programming-environment-on-ubuntu-16-04). 29 | 30 | It's recommended to use a virtualenv: 31 | 32 | ```bash 33 | make venv 34 | ``` 35 | 36 | ### To run 37 | 38 | Like any other target that's following the singer specification: 39 | 40 | `some-singer-tap | target-postgres --config [config.json]` 41 | 42 | It's reading incoming messages from STDIN and using the properties in `config.json` to upload data into Postgres. 43 | 44 | **Note**: To avoid version conflicts run `tap` and `targets` in separate virtual environments. 45 | 46 | 47 | #### Spin up a PG DB 48 | 49 | Make use of the available docker-compose file to spin up a PG DB. 50 | 51 | ```bash 52 | docker-compose up -d --build db 53 | ``` 54 | 55 | 56 | ### Configuration settings 57 | 58 | Running the target connector requires a `config.json` file. An example with the minimal settings: 59 | 60 | ```json 61 | { 62 | "host": "localhost", 63 | "port": 5432, 64 | "user": "my_user", 65 | "password": "secret", 66 | "dbname": "target_db", 67 | "default_target_schema": "public" 68 | } 69 | ``` 70 | 71 | Full list of options in `config.json`: 72 | 73 | | Property | Type | Required? | Description | 74 | |-------------------------------------|---------|------------|---------------------------------------------------------------| 75 | | host | String | Yes | PostgreSQL host | 76 | | port | Integer | Yes | PostgreSQL port | 77 | | user | String | Yes | PostgreSQL user | 78 | | password | String | Yes | PostgreSQL password | 79 | | dbname | String | Yes | PostgreSQL database name | 80 | | batch_size_rows | Integer | | (Default: 100000) Maximum number of rows in each batch. At the end of each batch, the rows in the batch are loaded into Postgres. | 81 | | flush_all_streams | Boolean | | (Default: False) Flush and load every stream into Postgres when one batch is full. Warning: This may trigger the COPY command to use files with low number of records. | 82 | | parallelism | Integer | | (Default: 0) The number of threads used to flush tables. 0 will create a thread for each stream, up to parallelism_max. -1 will create a thread for each CPU core. Any other positive number will create that number of threads, up to parallelism_max. | 83 | | max_parallelism | Integer | | (Default: 16) Max number of parallel threads to use when flushing tables. | 84 | | default_target_schema | String | | Name of the schema where the tables will be created. If `schema_mapping` is not defined then every stream sent by the tap is loaded into this schema. | 85 | | default_target_schema_select_permission | String | | Grant USAGE privilege on newly created schemas and grant SELECT privilege on newly created 86 | | schema_mapping | Object | | Useful if you want to load multiple streams from one tap to multiple Postgres schemas.

If the tap sends the `stream_id` in `-` format then this option overwrites the `default_target_schema` value. Note, that using `schema_mapping` you can overwrite the `default_target_schema_select_permission` value to grant SELECT permissions to different groups per schemas or optionally you can create indices automatically for the replicated tables.

**Note**: This is an experimental feature and recommended to use via PipelineWise YAML files that will generate the object mapping in the right JSON format. For further info check a [PipelineWise YAML Example](https://transferwise.github.io/pipelinewise/connectors/taps/mysql.html#configuring-what-to-replicate). | 87 | | add_metadata_columns | Boolean | | (Default: False) Metadata columns add extra row level information about data ingestions, (i.e. when was the row read in source, when was inserted or deleted in postgres etc.) Metadata columns are creating automatically by adding extra columns to the tables with a column prefix `_SDC_`. The column names are following the stitch naming conventions documented at https://www.stitchdata.com/docs/data-structure/integration-schemas#sdc-columns. Enabling metadata columns will flag the deleted rows by setting the `_SDC_DELETED_AT` metadata column. Without the `add_metadata_columns` option the deleted rows from singer taps will not be recognisable in Postgres. | 88 | | hard_delete | Boolean | | (Default: False) When `hard_delete` option is true then DELETE SQL commands will be performed in Postgres to delete rows in tables. It's achieved by continuously checking the `_SDC_DELETED_AT` metadata column sent by the singer tap. Due to deleting rows requires metadata columns, `hard_delete` option automatically enables the `add_metadata_columns` option as well. | 89 | | data_flattening_max_level | Integer | | (Default: 0) Object type RECORD items from taps can be transformed to flattened columns by creating columns automatically.

When value is 0 (default) then flattening functionality is turned off. | 90 | | primary_key_required | Boolean | | (Default: True) Log based and Incremental replications on tables with no Primary Key cause duplicates when merging UPDATE events. When set to true, stop loading data if no Primary Key is defined. | 91 | | validate_records | Boolean | | (Default: False) Validate every single record message to the corresponding JSON schema. This option is disabled by default and invalid RECORD messages will fail only at load time by Postgres. Enabling this option will detect invalid records earlier but could cause performance degradation. | 92 | | temp_dir | String | | (Default: platform-dependent) Directory of temporary CSV files with RECORD messages. | 93 | 94 | ### To run tests: 95 | 96 | 1. Define environment variables that requires running the tests 97 | ``` 98 | export TARGET_POSTGRES_HOST= 99 | export TARGET_POSTGRES_PORT= 100 | export TARGET_POSTGRES_USER= 101 | export TARGET_POSTGRES_PASSWORD= 102 | export TARGET_POSTGRES_DBNAME= 103 | export TARGET_POSTGRES_SCHEMA= 104 | ``` 105 | 106 | **PS**: You can run `make env` to export pre-defined environment variables 107 | 108 | 109 | 2. Install python dependencies in a virtual env and run unit and integration tests 110 | ``` 111 | make venv 112 | ``` 113 | 114 | 3. To run unit tests: 115 | ``` 116 | make unit_test 117 | ``` 118 | 119 | 4. To run integration tests: 120 | ``` 121 | make integration_test 122 | ``` 123 | 124 | ### To run pylint: 125 | 126 | 1. Install python dependencies and run python linter 127 | ``` 128 | make venv pylint 129 | ``` 130 | 131 | ## License 132 | 133 | Apache License Version 2.0 134 | 135 | See [LICENSE](LICENSE) to see the full text. 136 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright 2012 The Obvious Corporation and contributors. 2 | 3 | 4 | Licensed under the Apache License, Version 2.0 (the "License"); 5 | you may not use this file except in compliance with the License. 6 | You may obtain a copy of the License at 7 | 8 | 9 | 10 | Unless required by applicable law or agreed to in writing, software 11 | distributed under the License is distributed on an "AS IS" BASIS, 12 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | See the License for the specific language governing permissions and 14 | limitations under the License. 15 | 16 | ``` 17 | ------------------------------------------------------------------------- 18 | Apache License 19 | Version 2.0, January 2004 20 | http://www.apache.org/licenses/ 21 | 22 | 23 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 24 | 25 | 1. Definitions. 26 | 27 | "License" shall mean the terms and conditions for use, reproduction, 28 | and distribution as defined by Sections 1 through 9 of this document. 29 | 30 | "Licensor" shall mean the copyright owner or entity authorized by 31 | the copyright owner that is granting the License. 32 | 33 | "Legal Entity" shall mean the union of the acting entity and all 34 | other entities that control, are controlled by, or are under common 35 | control with that entity. For the purposes of this definition, 36 | "control" means (i) the power, direct or indirect, to cause the 37 | direction or management of such entity, whether by contract or 38 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 39 | outstanding shares, or (iii) beneficial ownership of such entity. 40 | 41 | "You" (or "Your") shall mean an individual or Legal Entity 42 | exercising permissions granted by this License. 43 | 44 | "Source" form shall mean the preferred form for making modifications, 45 | including but not limited to software source code, documentation 46 | source, and configuration files. 47 | 48 | "Object" form shall mean any form resulting from mechanical 49 | transformation or translation of a Source form, including but 50 | not limited to compiled object code, generated documentation, 51 | and conversions to other media types. 52 | 53 | "Work" shall mean the work of authorship, whether in Source or 54 | Object form, made available under the License, as indicated by a 55 | copyright notice that is included in or attached to the work 56 | (an example is provided in the Appendix below). 57 | 58 | "Derivative Works" shall mean any work, whether in Source or Object 59 | form, that is based on (or derived from) the Work and for which the 60 | editorial revisions, annotations, elaborations, or other modifications 61 | represent, as a whole, an original work of authorship. For the purposes 62 | of this License, Derivative Works shall not include works that remain 63 | separable from, or merely link (or bind by name) to the interfaces of, 64 | the Work and Derivative Works thereof. 65 | 66 | "Contribution" shall mean any work of authorship, including 67 | the original version of the Work and any modifications or additions 68 | to that Work or Derivative Works thereof, that is intentionally 69 | submitted to Licensor for inclusion in the Work by the copyright owner 70 | or by an individual or Legal Entity authorized to submit on behalf of 71 | the copyright owner. For the purposes of this definition, "submitted" 72 | means any form of electronic, verbal, or written communication sent 73 | to the Licensor or its representatives, including but not limited to 74 | communication on electronic mailing lists, source code control systems, 75 | and issue tracking systems that are managed by, or on behalf of, the 76 | Licensor for the purpose of discussing and improving the Work, but 77 | excluding communication that is conspicuously marked or otherwise 78 | designated in writing by the copyright owner as "Not a Contribution." 79 | 80 | "Contributor" shall mean Licensor and any individual or Legal Entity 81 | on behalf of whom a Contribution has been received by Licensor and 82 | subsequently incorporated within the Work. 83 | 84 | 2. Grant of Copyright License. Subject to the terms and conditions of 85 | this License, each Contributor hereby grants to You a perpetual, 86 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 87 | copyright license to reproduce, prepare Derivative Works of, 88 | publicly display, publicly perform, sublicense, and distribute the 89 | Work and such Derivative Works in Source or Object form. 90 | 91 | 3. Grant of Patent License. Subject to the terms and conditions of 92 | this License, each Contributor hereby grants to You a perpetual, 93 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 94 | (except as stated in this section) patent license to make, have made, 95 | use, offer to sell, sell, import, and otherwise transfer the Work, 96 | where such license applies only to those patent claims licensable 97 | by such Contributor that are necessarily infringed by their 98 | Contribution(s) alone or by combination of their Contribution(s) 99 | with the Work to which such Contribution(s) was submitted. If You 100 | institute patent litigation against any entity (including a 101 | cross-claim or counterclaim in a lawsuit) alleging that the Work 102 | or a Contribution incorporated within the Work constitutes direct 103 | or contributory patent infringement, then any patent licenses 104 | granted to You under this License for that Work shall terminate 105 | as of the date such litigation is filed. 106 | 107 | 4. Redistribution. You may reproduce and distribute copies of the 108 | Work or Derivative Works thereof in any medium, with or without 109 | modifications, and in Source or Object form, provided that You 110 | meet the following conditions: 111 | 112 | (a) You must give any other recipients of the Work or 113 | Derivative Works a copy of this License; and 114 | 115 | (b) You must cause any modified files to carry prominent notices 116 | stating that You changed the files; and 117 | 118 | (c) You must retain, in the Source form of any Derivative Works 119 | that You distribute, all copyright, patent, trademark, and 120 | attribution notices from the Source form of the Work, 121 | excluding those notices that do not pertain to any part of 122 | the Derivative Works; and 123 | 124 | (d) If the Work includes a "NOTICE" text file as part of its 125 | distribution, then any Derivative Works that You distribute must 126 | include a readable copy of the attribution notices contained 127 | within such NOTICE file, excluding those notices that do not 128 | pertain to any part of the Derivative Works, in at least one 129 | of the following places: within a NOTICE text file distributed 130 | as part of the Derivative Works; within the Source form or 131 | documentation, if provided along with the Derivative Works; or, 132 | within a display generated by the Derivative Works, if and 133 | wherever such third-party notices normally appear. The contents 134 | of the NOTICE file are for informational purposes only and 135 | do not modify the License. You may add Your own attribution 136 | notices within Derivative Works that You distribute, alongside 137 | or as an addendum to the NOTICE text from the Work, provided 138 | that such additional attribution notices cannot be construed 139 | as modifying the License. 140 | 141 | You may add Your own copyright statement to Your modifications and 142 | may provide additional or different license terms and conditions 143 | for use, reproduction, or distribution of Your modifications, or 144 | for any such Derivative Works as a whole, provided Your use, 145 | reproduction, and distribution of the Work otherwise complies with 146 | the conditions stated in this License. 147 | 148 | 5. Submission of Contributions. Unless You explicitly state otherwise, 149 | any Contribution intentionally submitted for inclusion in the Work 150 | by You to the Licensor shall be under the terms and conditions of 151 | this License, without any additional terms or conditions. 152 | Notwithstanding the above, nothing herein shall supersede or modify 153 | the terms of any separate license agreement you may have executed 154 | with Licensor regarding such Contributions. 155 | 156 | 6. Trademarks. This License does not grant permission to use the trade 157 | names, trademarks, service marks, or product names of the Licensor, 158 | except as required for reasonable and customary use in describing the 159 | origin of the Work and reproducing the content of the NOTICE file. 160 | 161 | 7. Disclaimer of Warranty. Unless required by applicable law or 162 | agreed to in writing, Licensor provides the Work (and each 163 | Contributor provides its Contributions) on an "AS IS" BASIS, 164 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 165 | implied, including, without limitation, any warranties or conditions 166 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 167 | PARTICULAR PURPOSE. You are solely responsible for determining the 168 | appropriateness of using or redistributing the Work and assume any 169 | risks associated with Your exercise of permissions under this License. 170 | 171 | 8. Limitation of Liability. In no event and under no legal theory, 172 | whether in tort (including negligence), contract, or otherwise, 173 | unless required by applicable law (such as deliberate and grossly 174 | negligent acts) or agreed to in writing, shall any Contributor be 175 | liable to You for damages, including any direct, indirect, special, 176 | incidental, or consequential damages of any character arising as a 177 | result of this License or out of the use or inability to use the 178 | Work (including but not limited to damages for loss of goodwill, 179 | work stoppage, computer failure or malfunction, or any and all 180 | other commercial damages or losses), even if such Contributor 181 | has been advised of the possibility of such damages. 182 | 183 | 9. Accepting Warranty or Additional Liability. While redistributing 184 | the Work or Derivative Works thereof, You may choose to offer, 185 | and charge a fee for, acceptance of support, warranty, indemnity, 186 | or other liability obligations and/or rights consistent with this 187 | License. However, in accepting such obligations, You may act only 188 | on Your own behalf and on Your sole responsibility, not on behalf 189 | of any other Contributor, and only if You agree to indemnify, 190 | defend, and hold each Contributor harmless for any liability 191 | incurred by, or claims asserted against, such Contributor by reason 192 | of your accepting any such warranty or additional liability. 193 | 194 | END OF TERMS AND CONDITIONS 195 | ``` 196 | -------------------------------------------------------------------------------- /tests/unit/test_db_sync.py: -------------------------------------------------------------------------------- 1 | import unittest 2 | 3 | import target_postgres 4 | 5 | 6 | class TestUnit(unittest.TestCase): 7 | """ 8 | Unit Tests 9 | """ 10 | @classmethod 11 | def setUp(self): 12 | self.config = {} 13 | 14 | def test_config_validation(self): 15 | """Test configuration validator""" 16 | validator = target_postgres.db_sync.validate_config 17 | empty_config = {} 18 | minimal_config = { 19 | 'host': "dummy-value", 20 | 'port': 5432, 21 | 'user': "dummy-value", 22 | 'password': "dummy-value", 23 | 'dbname': "dummy-value", 24 | 'default_target_schema': "dummy-value" 25 | } 26 | 27 | # Config validator returns a list of errors 28 | # If the list is empty then the configuration is valid otherwise invalid 29 | 30 | # Empty configuration should fail - (nr_of_errors >= 0) 31 | self.assertGreater(len(validator(empty_config)), 0) 32 | 33 | # Minimal configuration should pass - (nr_of_errors == 0) 34 | self.assertEqual(len(validator(minimal_config)), 0) 35 | 36 | # Configuration without schema references - (nr_of_errors >= 0) 37 | config_with_no_schema = minimal_config.copy() 38 | config_with_no_schema.pop('default_target_schema') 39 | self.assertGreater(len(validator(config_with_no_schema)), 0) 40 | 41 | # Configuration with schema mapping - (nr_of_errors >= 0) 42 | config_with_schema_mapping = minimal_config.copy() 43 | config_with_schema_mapping.pop('default_target_schema') 44 | config_with_schema_mapping['schema_mapping'] = { 45 | "dummy_stream": { 46 | "target_schema": "dummy_schema" 47 | } 48 | } 49 | self.assertEqual(len(validator(config_with_schema_mapping)), 0) 50 | 51 | 52 | def test_column_type_mapping(self): 53 | """Test JSON type to Postgres column type mappings""" 54 | mapper = target_postgres.db_sync.column_type 55 | 56 | # Incoming JSON schema types 57 | json_str = {"type": ["string"] } 58 | json_str_or_null = {"type": ["string", "null"] } 59 | json_dt = {"type": ["string"] , "format": "date-time"} 60 | json_dt_or_null = {"type": ["string", "null"] , "format": "date-time"} 61 | json_t = {"type": ["string"] , "format": "time"} 62 | json_t_or_null = {"type": ["string", "null"] , "format": "time"} 63 | json_num = {"type": ["number"] } 64 | json_smallint = {"type": ["integer"] , "maximum": 32767, "minimum": -32768} 65 | json_int = {"type": ["integer"] , "maximum": 2147483647, "minimum": -2147483648} 66 | json_bigint = {"type": ["integer"] , "maximum": 9223372036854775807, "minimum": -9223372036854775808} 67 | json_nobound_int = {"type": ["integer"] } 68 | json_int_or_str = {"type": ["integer", "string"] } 69 | json_bool = {"type": ["boolean"] } 70 | json_obj = {"type": ["object"] } 71 | json_arr = {"type": ["array"] } 72 | 73 | # Mapping from JSON schema types to Postgres column types 74 | self.assertEqual(mapper(json_str) , 'character varying') 75 | self.assertEqual(mapper(json_str_or_null) , 'character varying') 76 | self.assertEqual(mapper(json_dt) , 'timestamp without time zone') 77 | self.assertEqual(mapper(json_dt_or_null) , 'timestamp without time zone') 78 | self.assertEqual(mapper(json_t) , 'time without time zone') 79 | self.assertEqual(mapper(json_t_or_null) , 'time without time zone') 80 | self.assertEqual(mapper(json_num) , 'double precision') 81 | self.assertEqual(mapper(json_smallint) , 'smallint') 82 | self.assertEqual(mapper(json_int) , 'integer') 83 | self.assertEqual(mapper(json_bigint) , 'bigint') 84 | self.assertEqual(mapper(json_nobound_int) , 'numeric') 85 | self.assertEqual(mapper(json_int_or_str) , 'character varying') 86 | self.assertEqual(mapper(json_bool) , 'boolean') 87 | self.assertEqual(mapper(json_obj) , 'jsonb') 88 | self.assertEqual(mapper(json_arr) , 'jsonb') 89 | 90 | def test_stream_name_to_dict(self): 91 | """Test identifying catalog, schema and table names from fully qualified stream and table names""" 92 | # Singer stream name format (Default '-' separator) 93 | assert \ 94 | target_postgres.db_sync.stream_name_to_dict('my_table') == \ 95 | {"catalog_name": None, "schema_name": None, "table_name": "my_table"} 96 | 97 | # Singer stream name format (Default '-' separator) 98 | assert \ 99 | target_postgres.db_sync.stream_name_to_dict('my_schema-my_table') == \ 100 | {"catalog_name": None, "schema_name": "my_schema", "table_name": "my_table"} 101 | 102 | # Singer stream name format (Default '-' separator) 103 | assert \ 104 | target_postgres.db_sync.stream_name_to_dict('my_catalog-my_schema-my_table') == \ 105 | {"catalog_name": "my_catalog", "schema_name": "my_schema", "table_name": "my_table"} 106 | 107 | # Redshift table format (Custom '.' separator) 108 | assert \ 109 | target_postgres.db_sync.stream_name_to_dict('my_table', separator='.') == \ 110 | {"catalog_name": None, "schema_name": None, "table_name": "my_table"} 111 | 112 | # Redshift table format (Custom '.' separator) 113 | assert \ 114 | target_postgres.db_sync.stream_name_to_dict('my_schema.my_table', separator='.') == \ 115 | {"catalog_name": None, "schema_name": "my_schema", "table_name": "my_table"} 116 | 117 | # Redshift table format (Custom '.' separator) 118 | assert \ 119 | target_postgres.db_sync.stream_name_to_dict('my_catalog.my_schema.my_table', separator='.') == \ 120 | {"catalog_name": "my_catalog", "schema_name": "my_schema", "table_name": "my_table"} 121 | 122 | 123 | def test_flatten_schema(self): 124 | """Test flattening of SCHEMA messages""" 125 | flatten_schema = target_postgres.db_sync.flatten_schema 126 | 127 | # Schema with no object properties should be empty dict 128 | schema_with_no_properties = {"type": "object"} 129 | assert flatten_schema(schema_with_no_properties) == {} 130 | 131 | not_nested_schema = { 132 | "type": "object", 133 | "properties": { 134 | "c_pk": {"type": ["null", "integer"]}, 135 | "c_varchar": {"type": ["null", "string"]}, 136 | "c_int": {"type": ["null", "integer"]}}} 137 | # NO FLATTENNING - Schema with simple properties should be a plain dictionary 138 | assert flatten_schema(not_nested_schema) == not_nested_schema['properties'] 139 | 140 | nested_schema_with_no_properties = { 141 | "type": "object", 142 | "properties": { 143 | "c_pk": {"type": ["null", "integer"]}, 144 | "c_varchar": {"type": ["null", "string"]}, 145 | "c_int": {"type": ["null", "integer"]}, 146 | "c_obj": {"type": ["null", "object"]}}} 147 | # NO FLATTENNING - Schema with object type property but without further properties should be a plain dictionary 148 | assert flatten_schema(nested_schema_with_no_properties) == nested_schema_with_no_properties['properties'] 149 | 150 | nested_schema_with_properties = { 151 | "type": "object", 152 | "properties": { 153 | "c_pk": {"type": ["null", "integer"]}, 154 | "c_varchar": {"type": ["null", "string"]}, 155 | "c_int": {"type": ["null", "integer"]}, 156 | "c_obj": { 157 | "type": ["null", "object"], 158 | "properties": { 159 | "nested_prop1": {"type": ["null", "string"]}, 160 | "nested_prop2": {"type": ["null", "string"]}, 161 | "nested_prop3": { 162 | "type": ["null", "object"], 163 | "properties": { 164 | "multi_nested_prop1": {"type": ["null", "string"]}, 165 | "multi_nested_prop2": {"type": ["null", "string"]} 166 | } 167 | } 168 | } 169 | } 170 | } 171 | } 172 | # NO FLATTENNING - Schema with object type property but without further properties should be a plain dictionary 173 | # No flattening (default) 174 | assert flatten_schema(nested_schema_with_properties) == nested_schema_with_properties['properties'] 175 | 176 | # NO FLATTENNING - Schema with object type property but without further properties should be a plain dictionary 177 | # max_level: 0 : No flattening (default) 178 | assert flatten_schema(nested_schema_with_properties, max_level=0) == nested_schema_with_properties['properties'] 179 | 180 | # FLATTENNING - Schema with object type property but without further properties should be a dict with flattened properties 181 | assert \ 182 | flatten_schema(nested_schema_with_properties, max_level=1) == \ 183 | { 184 | 'c_pk': {'type': ['null', 'integer']}, 185 | 'c_varchar': {'type': ['null', 'string']}, 186 | 'c_int': {'type': ['null', 'integer']}, 187 | 'c_obj__nested_prop1': {'type': ['null', 'string']}, 188 | 'c_obj__nested_prop2': {'type': ['null', 'string']}, 189 | 'c_obj__nested_prop3': { 190 | 'type': ['null', 'object'], 191 | "properties": { 192 | "multi_nested_prop1": {"type": ["null", "string"]}, 193 | "multi_nested_prop2": {"type": ["null", "string"]} 194 | } 195 | } 196 | } 197 | 198 | # FLATTENNING - Schema with object type property but without further properties should be a dict with flattened properties 199 | assert \ 200 | flatten_schema(nested_schema_with_properties, max_level=10) == \ 201 | { 202 | 'c_pk': {'type': ['null', 'integer']}, 203 | 'c_varchar': {'type': ['null', 'string']}, 204 | 'c_int': {'type': ['null', 'integer']}, 205 | 'c_obj__nested_prop1': {'type': ['null', 'string']}, 206 | 'c_obj__nested_prop2': {'type': ['null', 'string']}, 207 | 'c_obj__nested_prop3__multi_nested_prop1': {'type': ['null', 'string']}, 208 | 'c_obj__nested_prop3__multi_nested_prop2': {'type': ['null', 'string']} 209 | } 210 | 211 | def test_flatten_record(self): 212 | """Test flattening of RECORD messages""" 213 | flatten_record = target_postgres.db_sync.flatten_record 214 | 215 | empty_record = {} 216 | # Empty record should be empty dict 217 | assert flatten_record(empty_record) == {} 218 | 219 | not_nested_record = {"c_pk": 1, "c_varchar": "1", "c_int": 1} 220 | # NO FLATTENNING - Record with simple properties should be a plain dictionary 221 | assert flatten_record(not_nested_record) == not_nested_record 222 | 223 | nested_record = { 224 | "c_pk": 1, 225 | "c_varchar": "1", 226 | "c_int": 1, 227 | "c_obj": { 228 | "nested_prop1": "value_1", 229 | "nested_prop2": "value_2", 230 | "nested_prop3": { 231 | "multi_nested_prop1": "multi_value_1", 232 | "multi_nested_prop2": "multi_value_2", 233 | }}} 234 | 235 | # NO FLATTENNING - No flattening (default) 236 | assert \ 237 | flatten_record(nested_record) == \ 238 | { 239 | "c_pk": 1, 240 | "c_varchar": "1", 241 | "c_int": 1, 242 | "c_obj": '{"nested_prop1": "value_1", "nested_prop2": "value_2", "nested_prop3": {"multi_nested_prop1": "multi_value_1", "multi_nested_prop2": "multi_value_2"}}' 243 | } 244 | 245 | # NO FLATTENNING 246 | # max_level: 0 : No flattening (default) 247 | assert \ 248 | flatten_record(nested_record, max_level=0) == \ 249 | { 250 | "c_pk": 1, 251 | "c_varchar": "1", 252 | "c_int": 1, 253 | "c_obj": '{"nested_prop1": "value_1", "nested_prop2": "value_2", "nested_prop3": {"multi_nested_prop1": "multi_value_1", "multi_nested_prop2": "multi_value_2"}}' 254 | } 255 | 256 | # SEMI FLATTENNING 257 | # max_level: 1 : Semi-flattening (default) 258 | assert \ 259 | flatten_record(nested_record, max_level=1) == \ 260 | { 261 | "c_pk": 1, 262 | "c_varchar": "1", 263 | "c_int": 1, 264 | "c_obj__nested_prop1": "value_1", 265 | "c_obj__nested_prop2": "value_2", 266 | "c_obj__nested_prop3": '{"multi_nested_prop1": "multi_value_1", "multi_nested_prop2": "multi_value_2"}' 267 | } 268 | 269 | # FLATTENNING 270 | assert \ 271 | flatten_record(nested_record, max_level=10) == \ 272 | { 273 | "c_pk": 1, 274 | "c_varchar": "1", 275 | "c_int": 1, 276 | "c_obj__nested_prop1": "value_1", 277 | "c_obj__nested_prop2": "value_2", 278 | "c_obj__nested_prop3__multi_nested_prop1": "multi_value_1", 279 | "c_obj__nested_prop3__multi_nested_prop2": "multi_value_2" 280 | } 281 | 282 | def test_flatten_record_with_flatten_schema(self): 283 | flatten_record = target_postgres.db_sync.flatten_record 284 | 285 | flatten_schema = { 286 | "id": { 287 | "type": [ 288 | "object", 289 | "array", 290 | "null" 291 | ] 292 | } 293 | } 294 | 295 | test_cases = [ 296 | ( 297 | True, 298 | { 299 | "id": 1, 300 | "data": "xyz" 301 | }, 302 | { 303 | "id": "1", 304 | "data": "xyz" 305 | } 306 | ), 307 | ( 308 | False, 309 | { 310 | "id": 1, 311 | "data": "xyz" 312 | }, 313 | { 314 | "id": 1, 315 | "data": "xyz" 316 | } 317 | ) 318 | ] 319 | 320 | for idx, (should_use_flatten_schema, record, expected_output) in enumerate(test_cases): 321 | output = flatten_record(record, flatten_schema if should_use_flatten_schema else None) 322 | assert output == expected_output 323 | -------------------------------------------------------------------------------- /target_postgres/__init__.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | import argparse 4 | import io 5 | import json 6 | import os 7 | import sys 8 | import copy 9 | from datetime import datetime 10 | from decimal import Decimal 11 | from tempfile import mkstemp 12 | 13 | from joblib import Parallel, delayed, parallel_backend 14 | from jsonschema import Draft7Validator, FormatChecker 15 | from singer import get_logger 16 | 17 | from target_postgres.db_sync import DbSync 18 | 19 | LOGGER = get_logger('target_postgres') 20 | 21 | DEFAULT_BATCH_SIZE_ROWS = 100000 22 | DEFAULT_PARALLELISM = 0 # 0 The number of threads used to flush tables 23 | DEFAULT_MAX_PARALLELISM = 16 # Don't use more than this number of threads by default when flushing streams in parallel 24 | 25 | 26 | class RecordValidationException(Exception): 27 | """Exception to raise when record validation failed""" 28 | 29 | 30 | class InvalidValidationOperationException(Exception): 31 | """Exception to raise when internal JSON schema validation process failed""" 32 | 33 | 34 | def float_to_decimal(value): 35 | """Walk the given data structure and turn all instances of float into 36 | double.""" 37 | if isinstance(value, float): 38 | return Decimal(str(value)) 39 | if isinstance(value, list): 40 | return [float_to_decimal(child) for child in value] 41 | if isinstance(value, dict): 42 | return {k: float_to_decimal(v) for k, v in value.items()} 43 | return value 44 | 45 | 46 | def add_metadata_columns_to_schema(schema_message): 47 | """Metadata _sdc columns according to the stitch documentation at 48 | https://www.stitchdata.com/docs/data-structure/integration-schemas#sdc-columns 49 | 50 | Metadata columns gives information about data injections 51 | """ 52 | extended_schema_message = schema_message 53 | extended_schema_message['schema']['properties']['_sdc_extracted_at'] = {'type': ['null', 'string'], 54 | 'format': 'date-time'} 55 | extended_schema_message['schema']['properties']['_sdc_batched_at'] = {'type': ['null', 'string'], 56 | 'format': 'date-time'} 57 | extended_schema_message['schema']['properties']['_sdc_deleted_at'] = {'type': ['null', 'string']} 58 | 59 | return extended_schema_message 60 | 61 | 62 | def add_metadata_values_to_record(record_message): 63 | """Populate metadata _sdc columns from incoming record message 64 | The location of the required attributes are fixed in the stream 65 | """ 66 | extended_record = record_message['record'] 67 | extended_record['_sdc_extracted_at'] = record_message.get('time_extracted') 68 | extended_record['_sdc_batched_at'] = datetime.now().isoformat() 69 | extended_record['_sdc_deleted_at'] = record_message.get('record', {}).get('_sdc_deleted_at') 70 | 71 | return extended_record 72 | 73 | 74 | def emit_state(state): 75 | """Emit state message to standard output then it can be 76 | consumed by other components""" 77 | if state is not None: 78 | line = json.dumps(state) 79 | LOGGER.debug('Emitting state %s', line) 80 | sys.stdout.write("{}\n".format(line)) 81 | sys.stdout.flush() 82 | 83 | 84 | # pylint: disable=too-many-locals,too-many-branches,too-many-statements,invalid-name,consider-iterating-dictionary 85 | def persist_lines(config, lines) -> None: 86 | """Read singer messages and process them line by line""" 87 | state = None 88 | flushed_state = None 89 | schemas = {} 90 | key_properties = {} 91 | validators = {} 92 | records_to_load = {} 93 | row_count = {} 94 | stream_to_sync = {} 95 | total_row_count = {} 96 | batch_size_rows = config.get('batch_size_rows', DEFAULT_BATCH_SIZE_ROWS) 97 | 98 | # Loop over lines from stdin 99 | for line in lines: 100 | try: 101 | o = json.loads(line) 102 | except json.decoder.JSONDecodeError: 103 | LOGGER.error('Unable to parse:\n%s', line) 104 | raise 105 | 106 | if 'type' not in o: 107 | raise Exception("Line is missing required key 'type': {}".format(line)) 108 | t = o['type'] 109 | 110 | if t == 'RECORD': 111 | if 'stream' not in o: 112 | raise Exception("Line is missing required key 'stream': {}".format(line)) 113 | if o['stream'] not in schemas: 114 | raise Exception( 115 | "A record for stream {} was encountered before a corresponding schema".format(o['stream'])) 116 | 117 | # Get schema for this record's stream 118 | stream = o['stream'] 119 | 120 | # Validate record 121 | if config.get('validate_records'): 122 | try: 123 | validators[stream].validate(float_to_decimal(o['record'])) 124 | except Exception as ex: 125 | if type(ex).__name__ == "InvalidOperation": 126 | raise InvalidValidationOperationException( 127 | f"Data validation failed and cannot load to destination. RECORD: {o['record']}\n" 128 | "multipleOf validations that allows long precisions are not supported (i.e. with 15 digits" 129 | "or more) Try removing 'multipleOf' methods from JSON schema.") from ex 130 | raise RecordValidationException( 131 | f"Record does not pass schema validation. RECORD: {o['record']}") from ex 132 | 133 | primary_key_string = stream_to_sync[stream].record_primary_key_string(o['record']) 134 | if not primary_key_string: 135 | primary_key_string = 'RID-{}'.format(total_row_count[stream]) 136 | 137 | if stream not in records_to_load: 138 | records_to_load[stream] = {} 139 | 140 | # increment row count only when a new PK is encountered in the current batch 141 | if primary_key_string not in records_to_load[stream]: 142 | row_count[stream] += 1 143 | total_row_count[stream] += 1 144 | 145 | # append record 146 | if config.get('add_metadata_columns') or config.get('hard_delete'): 147 | records_to_load[stream][primary_key_string] = add_metadata_values_to_record(o) 148 | else: 149 | records_to_load[stream][primary_key_string] = o['record'] 150 | 151 | row_count[stream] = len(records_to_load[stream]) 152 | 153 | if row_count[stream] >= batch_size_rows: 154 | # flush all streams, delete records if needed, reset counts and then emit current state 155 | if config.get('flush_all_streams'): 156 | filter_streams = None 157 | else: 158 | filter_streams = [stream] 159 | 160 | # Flush and return a new state dict with new positions only for the flushed streams 161 | flushed_state = flush_streams(records_to_load, 162 | row_count, 163 | stream_to_sync, 164 | config, 165 | state, 166 | flushed_state, 167 | filter_streams=filter_streams) 168 | 169 | # emit last encountered state 170 | emit_state(copy.deepcopy(flushed_state)) 171 | 172 | elif t == 'STATE': 173 | LOGGER.debug('Setting state to %s', o['value']) 174 | state = o['value'] 175 | 176 | # Initially set flushed state 177 | if not flushed_state: 178 | flushed_state = copy.deepcopy(state) 179 | 180 | elif t == 'SCHEMA': 181 | if 'stream' not in o: 182 | raise Exception("Line is missing required key 'stream': {}".format(line)) 183 | stream = o['stream'] 184 | 185 | schemas[stream] = float_to_decimal(o['schema']) 186 | validators[stream] = Draft7Validator(schemas[stream], format_checker=FormatChecker()) 187 | 188 | # flush records from previous stream SCHEMA 189 | if row_count.get(stream, 0) > 0: 190 | flushed_state = flush_streams(records_to_load, row_count, stream_to_sync, config, state, flushed_state) 191 | 192 | # emit latest encountered state 193 | emit_state(flushed_state) 194 | 195 | # key_properties key must be available in the SCHEMA message. 196 | if 'key_properties' not in o: 197 | raise Exception("key_properties field is required") 198 | 199 | # Log based and Incremental replications on tables with no Primary Key 200 | # cause duplicates when merging UPDATE events. 201 | # Stop loading data by default if no Primary Key. 202 | # 203 | # If you want to load tables with no Primary Key: 204 | # 1) Set ` 'primary_key_required': false ` in the target-postgres config.json 205 | # or 206 | # 2) Use fastsync [postgres-to-postgres, mysql-to-postgres, etc.] 207 | if config.get('primary_key_required', True) and len(o['key_properties']) == 0: 208 | LOGGER.critical("Primary key is set to mandatory but not defined in the [%s] stream", stream) 209 | raise Exception("key_properties field is required") 210 | 211 | key_properties[stream] = o['key_properties'] 212 | 213 | if config.get('add_metadata_columns') or config.get('hard_delete'): 214 | stream_to_sync[stream] = DbSync(config, add_metadata_columns_to_schema(o)) 215 | else: 216 | stream_to_sync[stream] = DbSync(config, o) 217 | 218 | stream_to_sync[stream].create_schema_if_not_exists() 219 | stream_to_sync[stream].sync_table() 220 | 221 | row_count[stream] = 0 222 | total_row_count[stream] = 0 223 | 224 | elif t == 'ACTIVATE_VERSION': 225 | LOGGER.debug('ACTIVATE_VERSION message') 226 | 227 | # Initially set flushed state 228 | if not flushed_state: 229 | flushed_state = copy.deepcopy(state) 230 | 231 | else: 232 | raise Exception("Unknown message type {} in message {}" 233 | .format(o['type'], o)) 234 | 235 | # if some bucket has records that need to be flushed but haven't reached batch size 236 | # then flush all buckets. 237 | if sum(row_count.values()) > 0: 238 | # flush all streams one last time, delete records if needed, reset counts and then emit current state 239 | flushed_state = flush_streams(records_to_load, row_count, stream_to_sync, config, state, flushed_state) 240 | 241 | # emit latest state 242 | emit_state(copy.deepcopy(flushed_state)) 243 | 244 | 245 | # pylint: disable=too-many-arguments 246 | def flush_streams( 247 | streams, 248 | row_count, 249 | stream_to_sync, 250 | config, 251 | state, 252 | flushed_state, 253 | filter_streams=None): 254 | """ 255 | Flushes all buckets and resets records count to 0 as well as empties records to load list 256 | :param streams: dictionary with records to load per stream 257 | :param row_count: dictionary with row count per stream 258 | :param stream_to_sync: Postgres db sync instance per stream 259 | :param config: dictionary containing the configuration 260 | :param state: dictionary containing the original state from tap 261 | :param flushed_state: dictionary containing updated states only when streams got flushed 262 | :param filter_streams: Keys of streams to flush from the streams dict. Default is every stream 263 | :return: State dict with flushed positions 264 | """ 265 | parallelism = config.get("parallelism", DEFAULT_PARALLELISM) 266 | max_parallelism = config.get("max_parallelism", DEFAULT_MAX_PARALLELISM) 267 | 268 | # Parallelism 0 means auto parallelism: 269 | # 270 | # Auto parallelism trying to flush streams efficiently with auto defined number 271 | # of threads where the number of threads is the number of streams that need to 272 | # be loaded but it's not greater than the value of max_parallelism 273 | if parallelism == 0: 274 | n_streams_to_flush = len(streams.keys()) 275 | if n_streams_to_flush > max_parallelism: 276 | parallelism = max_parallelism 277 | else: 278 | parallelism = n_streams_to_flush 279 | 280 | # Select the required streams to flush 281 | if filter_streams: 282 | streams_to_flush = filter_streams 283 | else: 284 | streams_to_flush = streams.keys() 285 | 286 | # Single-host, thread-based parallelism 287 | with parallel_backend('threading', n_jobs=parallelism): 288 | Parallel()(delayed(load_stream_batch)( 289 | stream=stream, 290 | records_to_load=streams[stream], 291 | row_count=row_count, 292 | db_sync=stream_to_sync[stream], 293 | delete_rows=config.get('hard_delete'), 294 | temp_dir=config.get('temp_dir') 295 | ) for stream in streams_to_flush) 296 | 297 | # reset flushed stream records to empty to avoid flushing same records 298 | for stream in streams_to_flush: 299 | streams[stream] = {} 300 | 301 | # Update flushed streams 302 | if filter_streams: 303 | # update flushed_state position if we have state information for the stream 304 | if state is not None and stream in state.get('bookmarks', {}): 305 | # Create bookmark key if not exists 306 | if 'bookmarks' not in flushed_state: 307 | flushed_state['bookmarks'] = {} 308 | # Copy the stream bookmark from the latest state 309 | flushed_state['bookmarks'][stream] = copy.deepcopy(state['bookmarks'][stream]) 310 | 311 | # If we flush every bucket use the latest state 312 | else: 313 | flushed_state = copy.deepcopy(state) 314 | 315 | # Return with state message with flushed positions 316 | return flushed_state 317 | 318 | 319 | # pylint: disable=too-many-arguments 320 | def load_stream_batch(stream, records_to_load, row_count, db_sync, delete_rows=False, temp_dir=None): 321 | """Load a batch of records and do post load operations, like creating 322 | or deleting rows""" 323 | # Load into Postgres 324 | if row_count[stream] > 0: 325 | flush_records(stream, records_to_load, row_count[stream], db_sync, temp_dir) 326 | 327 | # Load finished, create indices if required 328 | db_sync.create_indices(stream) 329 | 330 | # Delete soft-deleted, flagged rows - where _sdc_deleted at is not null 331 | if delete_rows: 332 | db_sync.delete_rows(stream) 333 | 334 | # reset row count for the current stream 335 | row_count[stream] = 0 336 | 337 | 338 | # pylint: disable=unused-argument 339 | def flush_records(stream, records_to_load, row_count, db_sync, temp_dir=None): 340 | """Take a list of records and load into database""" 341 | if temp_dir: 342 | temp_dir = os.path.expanduser(temp_dir) 343 | os.makedirs(temp_dir, exist_ok=True) 344 | 345 | size_bytes = 0 346 | csv_fd, csv_file = mkstemp(suffix='.csv', prefix=f'{stream}_', dir=temp_dir) 347 | with open(csv_fd, 'w+b') as f: 348 | for record in records_to_load.values(): 349 | csv_line = db_sync.record_to_csv_line(record) 350 | f.write(bytes(csv_line + '\n', 'UTF-8')) 351 | 352 | size_bytes = os.path.getsize(csv_file) 353 | db_sync.load_csv(csv_file, row_count, size_bytes) 354 | 355 | # Delete temp file 356 | os.remove(csv_file) 357 | 358 | 359 | def main(): 360 | """Main entry point""" 361 | arg_parser = argparse.ArgumentParser() 362 | arg_parser.add_argument('-c', '--config', help='Config file') 363 | args = arg_parser.parse_args() 364 | 365 | if args.config: 366 | with open(args.config) as config_input: 367 | config = json.load(config_input) 368 | else: 369 | config = {} 370 | 371 | # Consume singer messages 372 | singer_messages = io.TextIOWrapper(sys.stdin.buffer, encoding='utf-8') 373 | persist_lines(config, singer_messages) 374 | 375 | LOGGER.debug("Exiting normally") 376 | 377 | 378 | if __name__ == '__main__': 379 | main() 380 | -------------------------------------------------------------------------------- /.pylintrc: -------------------------------------------------------------------------------- 1 | # Based on Apache 2.0 licensed code from https://github.com/ClusterHQ/flocker 2 | 3 | [MASTER] 4 | 5 | # Specify a configuration file. 6 | #rcfile= 7 | 8 | # Python code to execute, usually for sys.path manipulation such as 9 | # pygtk.require(). 10 | # init-hook= 11 | 12 | # Add files or directories to the blacklist. They should be base names, not paths. 13 | ignore= 14 | 15 | # Pickle collected data for later comparisons. 16 | persistent=no 17 | 18 | # List of plugins (as comma separated values of python modules names) to load, 19 | # usually to register additional checkers. 20 | load-plugins= 21 | 22 | # Use multiple processes to speed up Pylint. 23 | # DO NOT CHANGE THIS VALUES >1 HIDE RESULTS!!!!! 24 | jobs=1 25 | 26 | # Allow loading of arbitrary C extensions. Extensions are imported into the 27 | # active Python interpreter and may run arbitrary code. 28 | unsafe-load-any-extension=no 29 | 30 | # A comma-separated list of package or module names from where C extensions may 31 | # be loaded. Extensions are loading into the active Python interpreter and may 32 | # run arbitrary code 33 | extension-pkg-whitelist=ujson 34 | 35 | # Allow optimization of some AST trees. This will activate a peephole AST 36 | # optimizer, which will apply various small optimizations. For instance, it can 37 | # be used to obtain the result of joining multiple strings with the addition 38 | # operator. Joining a lot of strings can lead to a maximum recursion error in 39 | # Pylint and this flag can prevent that. It has one side effect, the resulting 40 | # AST will be different than the one from reality. 41 | optimize-ast=no 42 | 43 | 44 | [MESSAGES CONTROL] 45 | 46 | # Only show warnings with the listed confidence levels. Leave empty to show 47 | # all. Valid levels: HIGH, INFERENCE, INFERENCE_FAILURE, UNDEFINED 48 | confidence= 49 | 50 | # Enable the message, report, category or checker with the given id(s). You can 51 | # either give multiple identifier separated by comma (,) or put this option 52 | # multiple time. See also the "--disable" option for examples. 53 | disable=wrong-import-order, 54 | broad-except, 55 | missing-module-docstring, 56 | 57 | 58 | enable=import-error, 59 | import-self, 60 | reimported, 61 | wildcard-import, 62 | misplaced-future, 63 | deprecated-module, 64 | unpacking-non-sequence, 65 | invalid-all-object, 66 | undefined-all-variable, 67 | used-before-assignment, 68 | cell-var-from-loop, 69 | global-variable-undefined, 70 | redefine-in-handler, 71 | unused-import, 72 | unused-wildcard-import, 73 | global-variable-not-assigned, 74 | undefined-loop-variable, 75 | global-statement, 76 | global-at-module-level, 77 | bad-open-mode, 78 | redundant-unittest-assert, 79 | boolean-datetime 80 | deprecated-method, 81 | anomalous-unicode-escape-in-string, 82 | anomalous-backslash-in-string, 83 | not-in-loop, 84 | continue-in-finally, 85 | abstract-class-instantiated, 86 | star-needs-assignment-target, 87 | duplicate-argument-name, 88 | return-in-init, 89 | too-many-star-expressions, 90 | nonlocal-and-global, 91 | return-outside-function, 92 | return-arg-in-generator, 93 | invalid-star-assignment-target, 94 | bad-reversed-sequence, 95 | nonexistent-operator, 96 | yield-outside-function, 97 | init-is-generator, 98 | nonlocal-without-binding, 99 | lost-exception, 100 | assert-on-tuple, 101 | dangerous-default-value, 102 | duplicate-key, 103 | useless-else-on-loop 104 | expression-not-assigned, 105 | confusing-with-statement, 106 | unnecessary-lambda, 107 | pointless-statement, 108 | pointless-string-statement, 109 | unnecessary-pass, 110 | unreachable, 111 | eval-used, 112 | exec-used, 113 | using-constant-test, 114 | bad-super-call, 115 | missing-super-argument, 116 | slots-on-old-class, 117 | super-on-old-class, 118 | property-on-old-class, 119 | not-an-iterable, 120 | not-a-mapping, 121 | format-needs-mapping, 122 | truncated-format-string, 123 | missing-format-string-key, 124 | mixed-format-string, 125 | too-few-format-args, 126 | bad-str-strip-call, 127 | too-many-format-args, 128 | bad-format-character, 129 | format-combined-specification, 130 | bad-format-string-key, 131 | bad-format-string, 132 | missing-format-attribute, 133 | missing-format-argument-key, 134 | unused-format-string-argument 135 | unused-format-string-key, 136 | invalid-format-index, 137 | bad-indentation, 138 | mixed-indentation, 139 | unnecessary-semicolon, 140 | lowercase-l-suffix, 141 | invalid-encoded-data, 142 | unpacking-in-except, 143 | import-star-module-level, 144 | long-suffix, 145 | old-octal-literal, 146 | old-ne-operator, 147 | backtick, 148 | old-raise-syntax, 149 | metaclass-assignment, 150 | next-method-called, 151 | dict-iter-method, 152 | dict-view-method, 153 | indexing-exception, 154 | raising-string, 155 | using-cmp-argument, 156 | cmp-method, 157 | coerce-method, 158 | delslice-method, 159 | getslice-method, 160 | hex-method, 161 | nonzero-method, 162 | t-method, 163 | setslice-method, 164 | old-division, 165 | logging-format-truncated, 166 | logging-too-few-args, 167 | logging-too-many-args, 168 | logging-unsupported-format, 169 | logging-format-interpolation, 170 | invalid-unary-operand-type, 171 | unsupported-binary-operation, 172 | not-callable, 173 | redundant-keyword-arg, 174 | assignment-from-no-return, 175 | assignment-from-none, 176 | not-context-manager, 177 | repeated-keyword, 178 | missing-kwoa, 179 | no-value-for-parameter, 180 | invalid-sequence-index, 181 | invalid-slice-index, 182 | unexpected-keyword-arg, 183 | unsupported-membership-test, 184 | unsubscriptable-object, 185 | access-member-before-definition, 186 | method-hidden, 187 | assigning-non-slot, 188 | duplicate-bases, 189 | inconsistent-mro, 190 | inherit-non-class, 191 | invalid-slots, 192 | invalid-slots-object, 193 | no-method-argument, 194 | no-self-argument, 195 | unexpected-special-method-signature, 196 | non-iterator-returned, 197 | arguments-differ, 198 | signature-differs, 199 | bad-staticmethod-argument, 200 | non-parent-init-called, 201 | bad-except-order, 202 | catching-non-exception, 203 | bad-exception-context, 204 | notimplemented-raised, 205 | raising-bad-type, 206 | raising-non-exception, 207 | misplaced-bare-raise, 208 | duplicate-except, 209 | nonstandard-exception, 210 | binary-op-exception, 211 | bare-except, 212 | not-async-context-manager, 213 | yield-inside-async-function 214 | 215 | # Needs investigation: 216 | # abstract-method (might be indicating a bug? probably not though) 217 | # protected-access (requires some refactoring) 218 | # attribute-defined-outside-init (requires some refactoring) 219 | # super-init-not-called (requires some cleanup) 220 | 221 | # Things we'd like to enable someday: 222 | # redefined-builtin (requires a bunch of work to clean up our code first) 223 | # redefined-outer-name (requires a bunch of work to clean up our code first) 224 | # undefined-variable (re-enable when pylint fixes https://github.com/PyCQA/pylint/issues/760) 225 | # no-name-in-module (giving us spurious warnings https://github.com/PyCQA/pylint/issues/73) 226 | # unused-argument (need to clean up or code a lot, e.g. prefix unused_?) 227 | # function-redefined (@overload causes lots of spurious warnings) 228 | # too-many-function-args (@overload causes spurious warnings... I think) 229 | # parameter-unpacking (needed for eventual Python 3 compat) 230 | # print-statement (needed for eventual Python 3 compat) 231 | # filter-builtin-not-iterating (Python 3) 232 | # map-builtin-not-iterating (Python 3) 233 | # range-builtin-not-iterating (Python 3) 234 | # zip-builtin-not-iterating (Python 3) 235 | # many others relevant to Python 3 236 | # unused-variable (a little work to cleanup, is all) 237 | 238 | # ... 239 | [REPORTS] 240 | 241 | # Set the output format. Available formats are text, parseable, colorized, msvs 242 | # (visual studio) and html. You can also give a reporter class, eg 243 | # mypackage.mymodule.MyReporterClass. 244 | output-format=parseable 245 | 246 | # Put messages in a separate file for each module / package specified on the 247 | # command line instead of printing them on stdout. Reports (if any) will be 248 | # written in a file name "pylint_global.[txt|html]". 249 | files-output=no 250 | 251 | # Tells whether to display a full report or only the messages 252 | reports=no 253 | 254 | # Python expression which should return a note less than 10 (10 is the highest 255 | # note). You have access to the variables errors warning, statement which 256 | # respectively contain the number of errors / warnings messages and the total 257 | # number of statements analyzed. This is used by the global evaluation report 258 | # (RP0004). 259 | evaluation=10.0 - ((float(5 * error + warning + refactor + convention) / statement) * 10) 260 | 261 | # Template used to display messages. This is a python new-style format string 262 | # used to format the message information. See doc for all details 263 | #msg-template= 264 | 265 | 266 | [LOGGING] 267 | 268 | # Logging modules to check that the string format arguments are in logging 269 | # function parameter format 270 | logging-modules=logging 271 | 272 | 273 | [FORMAT] 274 | 275 | # Maximum number of characters on a single line. 276 | max-line-length=120 277 | 278 | # Regexp for a line that is allowed to be longer than the limit. 279 | ignore-long-lines=^\s*(# )??$ 280 | 281 | # Allow the body of an if to be on the same line as the test if there is no 282 | # else. 283 | single-line-if-stmt=no 284 | 285 | # List of optional constructs for which whitespace checking is disabled. `dict- 286 | # separator` is used to allow tabulation in dicts, etc.: {1 : 1,\n222: 2}. 287 | # `trailing-comma` allows a space between comma and closing bracket: (a, ). 288 | # `empty-line` allows space-only lines. 289 | no-space-check=trailing-comma,dict-separator 290 | 291 | # Maximum number of lines in a module 292 | max-module-lines=1000 293 | 294 | # String used as indentation unit. This is usually " " (4 spaces) or "\t" (1 295 | # tab). 296 | indent-string=' ' 297 | 298 | # Number of spaces of indent required inside a hanging or continued line. 299 | indent-after-paren=4 300 | 301 | # Expected format of line ending, e.g. empty (any line ending), LF or CRLF. 302 | expected-line-ending-format= 303 | 304 | 305 | [TYPECHECK] 306 | 307 | # Tells whether missing members accessed in mixin class should be ignored. A 308 | # mixin class is detected if its name ends with "mixin" (case insensitive). 309 | ignore-mixin-members=yes 310 | 311 | # List of module names for which member attributes should not be checked 312 | # (useful for modules/projects where namespaces are manipulated during runtime 313 | # and thus existing member attributes cannot be deduced by static analysis. It 314 | # supports qualified module names, as well as Unix pattern matching. 315 | ignored-modules= 316 | 317 | # List of classes names for which member attributes should not be checked 318 | # (useful for classes with attributes dynamically set). This supports can work 319 | # with qualified names. 320 | ignored-classes= 321 | 322 | # List of members which are set dynamically and missed by pylint inference 323 | # system, and so shouldn't trigger E1101 when accessed. Python regular 324 | # expressions are accepted. 325 | generated-members= 326 | 327 | 328 | [VARIABLES] 329 | 330 | # Tells whether we should check for unused import in __init__ files. 331 | init-import=no 332 | 333 | # A regular expression matching the name of dummy variables (i.e. expectedly 334 | # not used). 335 | dummy-variables-rgx=_$|dummy 336 | 337 | # List of additional names supposed to be defined in builtins. Remember that 338 | # you should avoid to define new builtins when possible. 339 | additional-builtins= 340 | 341 | # List of strings which can identify a callback function by name. A callback 342 | # name must start or end with one of those strings. 343 | callbacks=cb_,_cb 344 | 345 | 346 | [SIMILARITIES] 347 | 348 | # Minimum lines number of a similarity. 349 | min-similarity-lines=4 350 | 351 | # Ignore comments when computing similarities. 352 | ignore-comments=yes 353 | 354 | # Ignore docstrings when computing similarities. 355 | ignore-docstrings=yes 356 | 357 | # Ignore imports when computing similarities. 358 | ignore-imports=no 359 | 360 | 361 | [SPELLING] 362 | 363 | # Spelling dictionary name. Available dictionaries: none. To make it working 364 | # install python-enchant package. 365 | spelling-dict= 366 | 367 | # List of comma separated words that should not be checked. 368 | spelling-ignore-words= 369 | 370 | # A path to a file that contains private dictionary; one word per line. 371 | spelling-private-dict-file= 372 | 373 | # Tells whether to store unknown words to indicated private dictionary in 374 | # --spelling-private-dict-file option instead of raising a message. 375 | spelling-store-unknown-words=no 376 | 377 | 378 | [MISCELLANEOUS] 379 | 380 | # List of note tags to take in consideration, separated by a comma. 381 | notes=FIXME,XXX,TODO 382 | 383 | 384 | [BASIC] 385 | 386 | # List of builtins function names that should not be used, separated by a comma 387 | bad-functions=map,filter,input 388 | 389 | # Good variable names which should always be accepted, separated by a comma 390 | good-names=i,j,k,ex,Run,_ 391 | 392 | # Bad variable names which should always be refused, separated by a comma 393 | bad-names=foo,bar,baz,toto,tutu,tata 394 | 395 | # Colon-delimited sets of names that determine each other's naming style when 396 | # the name regexes allow several styles. 397 | name-group= 398 | 399 | # Include a hint for the correct naming format with invalid-name 400 | include-naming-hint=no 401 | 402 | # Regular expression matching correct function names 403 | function-rgx=[a-z_][a-z0-9_]{2,40}$ 404 | 405 | # Naming hint for function names 406 | function-name-hint=[a-z_][a-z0-9_]{2,40}$ 407 | 408 | # Regular expression matching correct variable names 409 | variable-rgx=[a-z_][a-z0-9_]{2,30}$ 410 | 411 | # Naming hint for variable names 412 | variable-name-hint=[a-z_][a-z0-9_]{2,30}$ 413 | 414 | # Regular expression matching correct constant names 415 | const-rgx=(([A-Z_][A-Z0-9_]*)|(__.*__))$ 416 | 417 | # Naming hint for constant names 418 | const-name-hint=(([A-Z_][A-Z0-9_]*)|(__.*__))$ 419 | 420 | # Regular expression matching correct attribute names 421 | attr-rgx=[a-z_][a-z0-9_]{2,30}$ 422 | 423 | # Naming hint for attribute names 424 | attr-name-hint=[a-z_][a-z0-9_]{2,30}$ 425 | 426 | # Regular expression matching correct argument names 427 | argument-rgx=[a-z_][a-z0-9_]{2,30}$ 428 | 429 | # Naming hint for argument names 430 | argument-name-hint=[a-z_][a-z0-9_]{2,30}$ 431 | 432 | # Regular expression matching correct class attribute names 433 | class-attribute-rgx=([A-Za-z_][A-Za-z0-9_]{2,30}|(__.*__))$ 434 | 435 | # Naming hint for class attribute names 436 | class-attribute-name-hint=([A-Za-z_][A-Za-z0-9_]{2,30}|(__.*__))$ 437 | 438 | # Regular expression matching correct inline iteration names 439 | inlinevar-rgx=[A-Za-z_][A-Za-z0-9_]*$ 440 | 441 | # Naming hint for inline iteration names 442 | inlinevar-name-hint=[A-Za-z_][A-Za-z0-9_]*$ 443 | 444 | # Regular expression matching correct class names 445 | class-rgx=[A-Z_][a-zA-Z0-9]+$ 446 | 447 | # Naming hint for class names 448 | class-name-hint=[A-Z_][a-zA-Z0-9]+$ 449 | 450 | # Regular expression matching correct module names 451 | module-rgx=(([a-z_][a-z0-9_]*)|([A-Z][a-zA-Z0-9]+))$ 452 | 453 | # Naming hint for module names 454 | module-name-hint=(([a-z_][a-z0-9_]*)|([A-Z][a-zA-Z0-9]+))$ 455 | 456 | # Regular expression matching correct method names 457 | method-rgx=[a-z_][a-z0-9_]{2,30}$ 458 | 459 | # Naming hint for method names 460 | method-name-hint=[a-z_][a-z0-9_]{2,30}$ 461 | 462 | # Regular expression which should only match function or class names that do 463 | # not require a docstring. 464 | no-docstring-rgx=^_ 465 | 466 | # Minimum line length for functions/classes that require docstrings, shorter 467 | # ones are exempt. 468 | docstring-min-length=-1 469 | 470 | 471 | [ELIF] 472 | 473 | # Maximum number of nested blocks for function / method body 474 | max-nested-blocks=5 475 | 476 | 477 | [IMPORTS] 478 | 479 | # Deprecated modules which should not be used, separated by a comma 480 | deprecated-modules=regsub,TERMIOS,Bastion,rexec 481 | 482 | # Create a graph of every (i.e. internal and external) dependencies in the 483 | # given file (report RP0402 must not be disabled) 484 | import-graph= 485 | 486 | # Create a graph of external dependencies in the given file (report RP0402 must 487 | # not be disabled) 488 | ext-import-graph= 489 | 490 | # Create a graph of internal dependencies in the given file (report RP0402 must 491 | # not be disabled) 492 | int-import-graph= 493 | 494 | 495 | [DESIGN] 496 | 497 | # Maximum number of arguments for function / method 498 | max-args=5 499 | 500 | # Argument names that match this expression will be ignored. Default to name 501 | # with leading underscore 502 | ignored-argument-names=_.* 503 | 504 | # Maximum number of locals for function / method body 505 | max-locals=15 506 | 507 | # Maximum number of return / yield for function / method body 508 | max-returns=6 509 | 510 | # Maximum number of branch for function / method body 511 | max-branches=12 512 | 513 | # Maximum number of statements in function / method body 514 | max-statements=50 515 | 516 | # Maximum number of parents for a class (see R0901). 517 | max-parents=7 518 | 519 | # Maximum number of attributes for a class (see R0902). 520 | max-attributes=7 521 | 522 | # Minimum number of public methods for a class (see R0903). 523 | min-public-methods=2 524 | 525 | # Maximum number of public methods for a class (see R0904). 526 | max-public-methods=20 527 | 528 | # Maximum number of boolean expressions in a if statement 529 | max-bool-expr=5 530 | 531 | 532 | [CLASSES] 533 | 534 | # List of method names used to declare (i.e. assign) instance attributes. 535 | defining-attr-methods=__init__,__new__,setUp 536 | 537 | # List of valid names for the first argument in a class method. 538 | valid-classmethod-first-arg=cls 539 | 540 | # List of valid names for the first argument in a metaclass class method. 541 | valid-metaclass-classmethod-first-arg=mcs 542 | 543 | # List of member names, which should be excluded from the protected access 544 | # warning. 545 | exclude-protected=_asdict,_fields,_replace,_source,_make 546 | 547 | 548 | [EXCEPTIONS] 549 | 550 | # Exceptions that will emit a warning when being caught. Defaults to 551 | # "Exception" 552 | overgeneral-exceptions=Exception 553 | -------------------------------------------------------------------------------- /tests/unit/resources/logical-streams.json: -------------------------------------------------------------------------------- 1 | {"type": "SCHEMA", "stream": "logical1-logical1_table2", "schema": {"definitions": {"sdc_recursive_boolean_array": {"items": {"$ref": "#/definitions/sdc_recursive_boolean_array"}, "type": ["null", "boolean", "array"]}, "sdc_recursive_integer_array": {"items": {"$ref": "#/definitions/sdc_recursive_integer_array"}, "type": ["null", "integer", "array"]}, "sdc_recursive_number_array": {"items": {"$ref": "#/definitions/sdc_recursive_number_array"}, "type": ["null", "number", "array"]}, "sdc_recursive_object_array": {"items": {"$ref": "#/definitions/sdc_recursive_object_array"}, "type": ["null", "object", "array"]}, "sdc_recursive_string_array": {"items": {"$ref": "#/definitions/sdc_recursive_string_array"}, "type": ["null", "string", "array"]}, "sdc_recursive_timestamp_array": {"format": "date-time", "items": {"$ref": "#/definitions/sdc_recursive_timestamp_array"}, "type": ["null", "string", "array"]}}, "properties": {"cid": {"maximum": 2147483647, "minimum": -2147483648, "type": ["integer"]}, "cvarchar": {"type": ["null", "string"]}, "_sdc_deleted_at": {"type": ["null", "string"], "format": "date-time"}}, "type": "object"}, "key_properties": ["cid"], "bookmark_properties": ["lsn"]} 2 | {"type": "RECORD", "stream": "logical1-logical1_table2", "record": {"cid": 1, "cvarchar": "inserted row", "_sdc_deleted_at": null}, "version": 1570922723635, "time_extracted": "2019-10-13T14:06:31.838328Z"} 3 | {"type": "RECORD", "stream": "logical1-logical1_table2", "record": {"cid": 2, "cvarchar": "inserted row", "_sdc_deleted_at": null}, "version": 1570922723635, "time_extracted": "2019-10-13T14:06:31.838328Z"} 4 | {"type": "RECORD", "stream": "logical1-logical1_table2", "record": {"cid": 3, "cvarchar": "inserted row", "_sdc_deleted_at": null}, "version": 1570922723635, "time_extracted": "2019-10-13T14:06:31.838328Z"} 5 | {"type": "STATE", "value": {"currently_syncing": null, "bookmarks": {"logical1-logical1_edgydata": {"last_replication_method": "LOG_BASED", "lsn": 108200520, "version": 1570922723596, "xmin": null}, "logical1-logical1_table1": {"last_replication_method": "LOG_BASED", "lsn": 108200520, "version": 1570922723618, "xmin": null}, "logical1-logical1_table2": {"last_replication_method": "LOG_BASED", "lsn": 108200520, "version": 1570922723635, "xmin": null}, "logical2-logical2_table1": {"last_replication_method": "LOG_BASED", "lsn": 108200520, "version": 1570922723651, "xmin": null}, "public-city": {"last_replication_method": "INCREMENTAL", "replication_key": "id", "version": 1570922723667, "replication_key_value": 4079}, "public-country": {"last_replication_method": "FULL_TABLE", "version": 1570922730456, "xmin": null}, "public2-wearehere": {}}}} 6 | {"type": "RECORD", "stream": "logical1-logical1_table2", "record": {"cid": 4, "cvarchar": "delete later", "_sdc_deleted_at": null}, "version": 1570922723635, "time_extracted": "2019-10-13T14:06:31.838328Z"} 7 | {"type": "RECORD", "stream": "logical1-logical1_table2", "record": {"cid": 5, "cvarchar": "inserted row", "_sdc_deleted_at": null}, "version": 1570922723635, "time_extracted": "2019-10-13T14:06:31.838328Z"} 8 | {"type": "RECORD", "stream": "logical1-logical1_table2", "record": {"cid": 6, "cvarchar": "delete later", "_sdc_deleted_at": null}, "version": 1570922723635, "time_extracted": "2019-10-13T14:06:31.838328Z"} 9 | {"type": "STATE", "value": {"currently_syncing": null, "bookmarks": {"logical1-logical1_edgydata": {"last_replication_method": "LOG_BASED", "lsn": 108200928, "version": 1570922723596, "xmin": null}, "logical1-logical1_table1": {"last_replication_method": "LOG_BASED", "lsn": 108200928, "version": 1570922723618, "xmin": null}, "logical1-logical1_table2": {"last_replication_method": "LOG_BASED", "lsn": 108200928, "version": 1570922723635, "xmin": null}, "logical2-logical2_table1": {"last_replication_method": "LOG_BASED", "lsn": 108200928, "version": 1570922723651, "xmin": null}, "public-city": {"last_replication_method": "INCREMENTAL", "replication_key": "id", "version": 1570922723667, "replication_key_value": 4079}, "public-country": {"last_replication_method": "FULL_TABLE", "version": 1570922730456, "xmin": null}, "public2-wearehere": {}}}} 10 | {"type": "RECORD", "stream": "logical1-logical1_table2", "record": {"cid": 7, "cvarchar": "inserted row", "_sdc_deleted_at": null}, "version": 1570922723635, "time_extracted": "2019-10-13T14:06:31.838328Z"} 11 | {"type": "RECORD", "stream": "logical1-logical1_table2", "record": {"cid": 8, "cvarchar": "inserted row", "_sdc_deleted_at": null}, "version": 1570922723635, "time_extracted": "2019-10-13T14:06:31.838328Z"} 12 | {"type": "STATE", "value": {"currently_syncing": null, "bookmarks": {"logical1-logical1_edgydata": {"last_replication_method": "LOG_BASED", "lsn": 108201336, "version": 1570922723596, "xmin": null}, "logical1-logical1_table1": {"last_replication_method": "LOG_BASED", "lsn": 108201336, "version": 1570922723618, "xmin": null}, "logical1-logical1_table2": {"last_replication_method": "LOG_BASED", "lsn": 108201336, "version": 1570922723635, "xmin": null}, "logical2-logical2_table1": {"last_replication_method": "LOG_BASED", "lsn": 108201336, "version": 1570922723651, "xmin": null}, "public-city": {"last_replication_method": "INCREMENTAL", "replication_key": "id", "version": 1570922723667, "replication_key_value": 4079}, "public-country": {"last_replication_method": "FULL_TABLE", "version": 1570922730456, "xmin": null}, "public2-wearehere": {}}}} 13 | {"type": "RECORD", "stream": "logical1-logical1_table2", "record": {"cid": 9, "cvarchar": "inserted row", "_sdc_deleted_at": null}, "version": 1570922723635, "time_extracted": "2019-10-13T14:06:31.838328Z"} 14 | {"type": "RECORD", "stream": "logical1-logical1_table2", "record": {"cid": 10, "cvarchar": "inserted row", "_sdc_deleted_at": null}, "version": 1570922723635, "time_extracted": "2019-10-13T14:06:31.838328Z"} 15 | {"type": "STATE", "value": {"currently_syncing": null, "bookmarks": {"logical1-logical1_edgydata": {"last_replication_method": "LOG_BASED", "lsn": 108236664, "version": 1570922723596, "xmin": null}, "logical1-logical1_table1": {"last_replication_method": "LOG_BASED", "lsn": 108236664, "version": 1570922723618, "xmin": null}, "logical1-logical1_table2": {"last_replication_method": "LOG_BASED", "lsn": 108236664, "version": 1570922723635, "xmin": null}, "logical2-logical2_table1": {"last_replication_method": "LOG_BASED", "lsn": 108236664, "version": 1570922723651, "xmin": null}, "public-city": {"last_replication_method": "INCREMENTAL", "replication_key": "id", "version": 1570922723667, "replication_key_value": 4079}, "public-country": {"last_replication_method": "FULL_TABLE", "version": 1570922730456, "xmin": null}, "public2-wearehere": {}}}} 16 | {"type": "RECORD", "stream": "logical1-logical1_table2", "record": {"cid": 4, "_sdc_deleted_at": "2019-10-13T14:06:31.838328+00:00"}, "version": 1570922723635, "time_extracted": "2019-10-13T14:06:31.838328Z"} 17 | {"type": "RECORD", "stream": "logical1-logical1_table2", "record": {"cid": 6, "_sdc_deleted_at": "2019-10-13T14:06:31.838328+00:00"}, "version": 1570922723635, "time_extracted": "2019-10-13T14:06:31.838328Z"} 18 | {"type": "STATE", "value": {"currently_syncing": null, "bookmarks": {"logical1-logical1_edgydata": {"last_replication_method": "LOG_BASED", "lsn": 108237120, "version": 1570922723596, "xmin": null}, "logical1-logical1_table1": {"last_replication_method": "LOG_BASED", "lsn": 108237120, "version": 1570922723618, "xmin": null}, "logical1-logical1_table2": {"last_replication_method": "LOG_BASED", "lsn": 108237120, "version": 1570922723635, "xmin": null}, "logical2-logical2_table1": {"last_replication_method": "LOG_BASED", "lsn": 108237120, "version": 1570922723651, "xmin": null}, "public-city": {"last_replication_method": "INCREMENTAL", "replication_key": "id", "version": 1570922723667, "replication_key_value": 4079}, "public-country": {"last_replication_method": "FULL_TABLE", "version": 1570922730456, "xmin": null}, "public2-wearehere": {}}}} 19 | {"type": "RECORD", "stream": "logical1-logical1_table2", "record": {"cid": 1, "cvarchar": "updated row", "_sdc_deleted_at": null}, "version": 1570922723635, "time_extracted": "2019-10-13T14:06:31.838328Z"} 20 | {"type": "RECORD", "stream": "logical1-logical1_table2", "record": {"cid": 2, "cvarchar": "updated row", "_sdc_deleted_at": null}, "version": 1570922723635, "time_extracted": "2019-10-13T14:06:31.838328Z"} 21 | {"type": "RECORD", "stream": "logical1-logical1_table2", "record": {"cid": 3, "cvarchar": "updated row", "_sdc_deleted_at": null}, "version": 1570922723635, "time_extracted": "2019-10-13T14:06:31.838328Z"} 22 | {"type": "STATE", "value": {"currently_syncing": null, "bookmarks": {"logical1-logical1_edgydata": {"last_replication_method": "LOG_BASED", "lsn": 108237336, "version": 1570922723596, "xmin": null}, "logical1-logical1_table1": {"last_replication_method": "LOG_BASED", "lsn": 108237336, "version": 1570922723618, "xmin": null}, "logical1-logical1_table2": {"last_replication_method": "LOG_BASED", "lsn": 108237336, "version": 1570922723635, "xmin": null}, "logical2-logical2_table1": {"last_replication_method": "LOG_BASED", "lsn": 108237336, "version": 1570922723651, "xmin": null}, "public-city": {"last_replication_method": "INCREMENTAL", "replication_key": "id", "version": 1570922723667, "replication_key_value": 4079}, "public-country": {"last_replication_method": "FULL_TABLE", "version": 1570922730456, "xmin": null}, "public2-wearehere": {}}}} 23 | {"type": "RECORD", "stream": "logical1-logical1_table2", "record": {"cid": 5, "cvarchar": "updated row", "_sdc_deleted_at": null}, "version": 1570922723635, "time_extracted": "2019-10-13T14:06:31.838328Z"} 24 | {"type": "RECORD", "stream": "logical1-logical1_table2", "record": {"cid": 7, "cvarchar": "updated row", "_sdc_deleted_at": null}, "version": 1570922723635, "time_extracted": "2019-10-13T14:06:31.838328Z"} 25 | {"type": "RECORD", "stream": "logical1-logical1_table2", "record": {"cid": 8, "cvarchar": "updated row", "_sdc_deleted_at": null}, "version": 1570922723635, "time_extracted": "2019-10-13T14:06:31.838328Z"} 26 | {"type": "STATE", "value": {"currently_syncing": null, "bookmarks": {"logical1-logical1_edgydata": {"last_replication_method": "LOG_BASED", "lsn": 108237600, "version": 1570922723596, "xmin": null}, "logical1-logical1_table1": {"last_replication_method": "LOG_BASED", "lsn": 108237600, "version": 1570922723618, "xmin": null}, "logical1-logical1_table2": {"last_replication_method": "LOG_BASED", "lsn": 108237600, "version": 1570922723635, "xmin": null}, "logical2-logical2_table1": {"last_replication_method": "LOG_BASED", "lsn": 108237600, "version": 1570922723651, "xmin": null}, "public-city": {"last_replication_method": "INCREMENTAL", "replication_key": "id", "version": 1570922723667, "replication_key_value": 4079}, "public-country": {"last_replication_method": "FULL_TABLE", "version": 1570922730456, "xmin": null}, "public2-wearehere": {}}}} 27 | {"type": "RECORD", "stream": "logical1-logical1_table2", "record": {"cid": 9, "cvarchar": "updated row", "_sdc_deleted_at": null}, "version": 1570922723635, "time_extracted": "2019-10-13T14:06:31.838328Z"} 28 | {"type": "RECORD", "stream": "logical1-logical1_table2", "record": {"cid": 10, "cvarchar": "updated row", "_sdc_deleted_at": null}, "version": 1570922723635, "time_extracted": "2019-10-13T14:06:31.838328Z"} 29 | {"type": "RECORD", "stream": "logical1-logical1_table2", "record": {"cid": 11, "cvarchar": "inserted row", "_sdc_deleted_at": null}, "version": 1570922723635, "time_extracted": "2019-10-13T14:06:31.838328Z"} 30 | {"type": "STATE", "value": {"currently_syncing": null, "bookmarks": {"logical1-logical1_edgydata": {"last_replication_method": "LOG_BASED", "lsn": 108237864, "version": 1570922723596, "xmin": null}, "logical1-logical1_table1": {"last_replication_method": "LOG_BASED", "lsn": 108237864, "version": 1570922723618, "xmin": null}, "logical1-logical1_table2": {"last_replication_method": "LOG_BASED", "lsn": 108237864, "version": 1570922723635, "xmin": null}, "logical2-logical2_table1": {"last_replication_method": "LOG_BASED", "lsn": 108237864, "version": 1570922723651, "xmin": null}, "public-city": {"last_replication_method": "INCREMENTAL", "replication_key": "id", "version": 1570922723667, "replication_key_value": 4079}, "public-country": {"last_replication_method": "FULL_TABLE", "version": 1570922730456, "xmin": null}, "public2-wearehere": {}}}} 31 | {"type": "RECORD", "stream": "logical1-logical1_table2", "record": {"cid": 12, "cvarchar": "inserted row", "_sdc_deleted_at": null}, "version": 1570922723635, "time_extracted": "2019-10-13T14:06:31.838328Z"} 32 | {"type": "RECORD", "stream": "logical1-logical1_table2", "record": {"cid": 13, "cvarchar": "inserted row", "_sdc_deleted_at": null}, "version": 1570922723635, "time_extracted": "2019-10-13T14:06:31.838328Z"} 33 | {"type": "STATE", "value": {"currently_syncing": null, "bookmarks": {"logical1-logical1_edgydata": {"last_replication_method": "LOG_BASED", "lsn": 108238224, "version": 1570922723596, "xmin": null}, "logical1-logical1_table1": {"last_replication_method": "LOG_BASED", "lsn": 108238224, "version": 1570922723618, "xmin": null}, "logical1-logical1_table2": {"last_replication_method": "LOG_BASED", "lsn": 108238224, "version": 1570922723635, "xmin": null}, "logical2-logical2_table1": {"last_replication_method": "LOG_BASED", "lsn": 108238224, "version": 1570922723651, "xmin": null}, "public-city": {"last_replication_method": "INCREMENTAL", "replication_key": "id", "version": 1570922723667, "replication_key_value": 4079}, "public-country": {"last_replication_method": "FULL_TABLE", "version": 1570922730456, "xmin": null}, "public2-wearehere": {}}}} 34 | {"type": "RECORD", "stream": "logical1-logical1_table2", "record": {"cid": 14, "cvarchar": "inserted row", "_sdc_deleted_at": null}, "version": 1570922723635, "time_extracted": "2019-10-13T14:06:31.838328Z"} 35 | {"type": "RECORD", "stream": "logical1-logical1_table2", "record": {"cid": 15, "cvarchar": "inserted row", "_sdc_deleted_at": null}, "version": 1570922723635, "time_extracted": "2019-10-13T14:06:31.838328Z"} 36 | {"type": "RECORD", "stream": "logical1-logical1_table2", "record": {"cid": 16, "cvarchar": "inserted row", "_sdc_deleted_at": null}, "version": 1570922723635, "time_extracted": "2019-10-13T14:06:31.838328Z"} 37 | {"type": "RECORD", "stream": "logical1-logical1_table2", "record": {"cid": 17, "cvarchar": "inserted row", "_sdc_deleted_at": null}, "version": 1570922723635, "time_extracted": "2019-10-13T14:06:31.838328Z"} 38 | {"type": "STATE", "value": {"currently_syncing": null, "bookmarks": {"logical1-logical1_edgydata": {"last_replication_method": "LOG_BASED", "lsn": 108238768, "version": 1570922723596, "xmin": null}, "logical1-logical1_table1": {"last_replication_method": "LOG_BASED", "lsn": 108238768, "version": 1570922723618, "xmin": null}, "logical1-logical1_table2": {"last_replication_method": "LOG_BASED", "lsn": 108238768, "version": 1570922723635, "xmin": null}, "logical2-logical2_table1": {"last_replication_method": "LOG_BASED", "lsn": 108238768, "version": 1570922723651, "xmin": null}, "public-city": {"last_replication_method": "INCREMENTAL", "replication_key": "id", "version": 1570922723667, "replication_key_value": 4079}, "public-country": {"last_replication_method": "FULL_TABLE", "version": 1570922730456, "xmin": null}, "public2-wearehere": {}}}} 39 | {"type": "RECORD", "stream": "logical1-logical1_table2", "record": {"cid": 18, "cvarchar": "inserted row", "_sdc_deleted_at": null}, "version": 1570922723635, "time_extracted": "2019-10-13T14:06:31.838328Z"} 40 | {"type": "RECORD", "stream": "logical1-logical1_table2", "record": {"cid": 19, "cvarchar": "inserted row", "_sdc_deleted_at": null}, "version": 1570922723635, "time_extracted": "2019-10-13T14:06:31.838328Z"} 41 | {"type": "RECORD", "stream": "logical1-logical1_table2", "record": {"cid": 20, "cvarchar": "inserted row", "_sdc_deleted_at": null}, "version": 1570922723635, "time_extracted": "2019-10-13T14:06:31.838328Z"} 42 | {"type": "STATE", "value": {"currently_syncing": null, "bookmarks": {"logical1-logical1_edgydata": {"last_replication_method": "LOG_BASED", "lsn": 108239176, "version": 1570922723596, "xmin": null}, "logical1-logical1_table1": {"last_replication_method": "LOG_BASED", "lsn": 108239176, "version": 1570922723618, "xmin": null}, "logical1-logical1_table2": {"last_replication_method": "LOG_BASED", "lsn": 108239176, "version": 1570922723635, "xmin": null}, "logical2-logical2_table1": {"last_replication_method": "LOG_BASED", "lsn": 108239176, "version": 1570922723651, "xmin": null}, "public-city": {"last_replication_method": "INCREMENTAL", "replication_key": "id", "version": 1570922723667, "replication_key_value": 4079}, "public-country": {"last_replication_method": "FULL_TABLE", "version": 1570922730456, "xmin": null}, "public2-wearehere": {}}}} 43 | {"type": "RECORD", "stream": "logical1-logical1_table2", "record": {"cid": 11, "_sdc_deleted_at": "2019-10-13T14:06:31.838328+00:00"}, "version": 1570922723635, "time_extracted": "2019-10-13T14:06:31.838328Z"} 44 | {"type": "RECORD", "stream": "logical1-logical1_table2", "record": {"cid": 12, "_sdc_deleted_at": "2019-10-13T14:06:31.838328+00:00"}, "version": 1570922723635, "time_extracted": "2019-10-13T14:06:31.838328Z"} 45 | {"type": "RECORD", "stream": "logical1-logical1_table2", "record": {"cid": 13, "_sdc_deleted_at": "2019-10-13T14:06:31.838328+00:00"}, "version": 1570922723635, "time_extracted": "2019-10-13T14:06:31.838328Z"} 46 | {"type": "STATE", "value": {"currently_syncing": null, "bookmarks": {"logical1-logical1_edgydata": {"last_replication_method": "LOG_BASED", "lsn": 108239512, "version": 1570922723596, "xmin": null}, "logical1-logical1_table1": {"last_replication_method": "LOG_BASED", "lsn": 108239512, "version": 1570922723618, "xmin": null}, "logical1-logical1_table2": {"last_replication_method": "LOG_BASED", "lsn": 108239512, "version": 1570922723635, "xmin": null}, "logical2-logical2_table1": {"last_replication_method": "LOG_BASED", "lsn": 108239512, "version": 1570922723651, "xmin": null}, "public-city": {"last_replication_method": "INCREMENTAL", "replication_key": "id", "version": 1570922723667, "replication_key_value": 4079}, "public-country": {"last_replication_method": "FULL_TABLE", "version": 1570922730456, "xmin": null}, "public2-wearehere": {}}}} 47 | {"type": "RECORD", "stream": "logical1-logical1_table2", "record": {"cid": 14, "_sdc_deleted_at": "2019-10-13T14:06:31.838328+00:00"}, "version": 1570922723635, "time_extracted": "2019-10-13T14:06:31.838328Z"} 48 | {"type": "RECORD", "stream": "logical1-logical1_table2", "record": {"cid": 15, "_sdc_deleted_at": "2019-10-13T14:06:31.838328+00:00"}, "version": 1570922723635, "time_extracted": "2019-10-13T14:06:31.838328Z"} 49 | {"type": "RECORD", "stream": "logical1-logical1_table2", "record": {"cid": 16, "_sdc_deleted_at": "2019-10-13T14:06:31.838328+00:00"}, "version": 1570922723635, "time_extracted": "2019-10-13T14:06:31.838328Z"} 50 | {"type": "STATE", "value": {"currently_syncing": null, "bookmarks": {"logical1-logical1_edgydata": {"last_replication_method": "LOG_BASED", "lsn": 108239704, "version": 1570922723596, "xmin": null}, "logical1-logical1_table1": {"last_replication_method": "LOG_BASED", "lsn": 108239704, "version": 1570922723618, "xmin": null}, "logical1-logical1_table2": {"last_replication_method": "LOG_BASED", "lsn": 108239704, "version": 1570922723635, "xmin": null}, "logical2-logical2_table1": {"last_replication_method": "LOG_BASED", "lsn": 108239704, "version": 1570922723651, "xmin": null}, "public-city": {"last_replication_method": "INCREMENTAL", "replication_key": "id", "version": 1570922723667, "replication_key_value": 4079}, "public-country": {"last_replication_method": "FULL_TABLE", "version": 1570922730456, "xmin": null}, "public2-wearehere": {}}}} 51 | {"type": "RECORD", "stream": "logical1-logical1_table2", "record": {"cid": 17, "_sdc_deleted_at": "2019-10-13T14:06:31.838328+00:00"}, "version": 1570922723635, "time_extracted": "2019-10-13T14:06:31.838328Z"} 52 | {"type": "RECORD", "stream": "logical1-logical1_table2", "record": {"cid": 18, "_sdc_deleted_at": "2019-10-13T14:06:31.838328+00:00"}, "version": 1570922723635, "time_extracted": "2019-10-13T14:06:31.838328Z"} 53 | {"type": "RECORD", "stream": "logical1-logical1_table2", "record": {"cid": 19, "_sdc_deleted_at": "2019-10-13T14:06:31.838328+00:00"}, "version": 1570922723635, "time_extracted": "2019-10-13T14:06:31.838328Z"} 54 | {"type": "STATE", "value": {"currently_syncing": null, "bookmarks": {"logical1-logical1_edgydata": {"last_replication_method": "LOG_BASED", "lsn": 108239896, "version": 1570922723596, "xmin": null}, "logical1-logical1_table1": {"last_replication_method": "LOG_BASED", "lsn": 108239896, "version": 1570922723618, "xmin": null}, "logical1-logical1_table2": {"last_replication_method": "LOG_BASED", "lsn": 108239896, "version": 1570922723635, "xmin": null}, "logical2-logical2_table1": {"last_replication_method": "LOG_BASED", "lsn": 108239896, "version": 1570922723651, "xmin": null}, "public-city": {"last_replication_method": "INCREMENTAL", "replication_key": "id", "version": 1570922723667, "replication_key_value": 4079}, "public-country": {"last_replication_method": "FULL_TABLE", "version": 1570922730456, "xmin": null}, "public2-wearehere": {}}}} 55 | {"type": "RECORD", "stream": "logical1-logical1_table2", "record": {"cid": 20, "_sdc_deleted_at": "2019-10-13T14:06:31.838328+00:00"}, "version": 1570922723635, "time_extracted": "2019-10-13T14:06:31.838328Z"} 56 | {"type": "STATE", "value": {"currently_syncing": null, "bookmarks": {"logical1-logical1_edgydata": {"last_replication_method": "LOG_BASED", "lsn": 108240192, "version": 1570922723596, "xmin": null}, "logical1-logical1_table1": {"last_replication_method": "LOG_BASED", "lsn": 108240192, "version": 1570922723618, "xmin": null}, "logical1-logical1_table2": {"last_replication_method": "LOG_BASED", "lsn": 108240192, "version": 1570922723635, "xmin": null}, "logical2-logical2_table1": {"last_replication_method": "LOG_BASED", "lsn": 108240192, "version": 1570922723651, "xmin": null}, "public-city": {"last_replication_method": "INCREMENTAL", "replication_key": "id", "version": 1570922723667, "replication_key_value": 4079}, "public-country": {"last_replication_method": "FULL_TABLE", "version": 1570922730456, "xmin": null}, "public2-wearehere": {}}}} 57 | {"type": "STATE", "value": {"currently_syncing": null, "bookmarks": {"logical1-logical1_edgydata": {"last_replication_method": "LOG_BASED", "lsn": 108240696, "version": 1570922723596, "xmin": null}, "logical1-logical1_table1": {"last_replication_method": "LOG_BASED", "lsn": 108240696, "version": 1570922723618, "xmin": null}, "logical1-logical1_table2": {"last_replication_method": "LOG_BASED", "lsn": 108240696, "version": 1570922723635, "xmin": null}, "logical2-logical2_table1": {"last_replication_method": "LOG_BASED", "lsn": 108240696, "version": 1570922723651, "xmin": null}, "public-city": {"last_replication_method": "INCREMENTAL", "replication_key": "id", "version": 1570922723667, "replication_key_value": 4079}, "public-country": {"last_replication_method": "FULL_TABLE", "version": 1570922730456, "xmin": null}, "public2-wearehere": {}}}} 58 | {"type": "STATE", "value": {"currently_syncing": null, "bookmarks": {"logical1-logical1_edgydata": {"last_replication_method": "LOG_BASED", "lsn": 108240796, "version": 1570922723596, "xmin": null}, "logical1-logical1_table1": {"last_replication_method": "LOG_BASED", "lsn": 108240796, "version": 1570922723618, "xmin": null}, "logical1-logical1_table2": {"last_replication_method": "LOG_BASED", "lsn": 108240796, "version": 1570922723635, "xmin": null}, "logical2-logical2_table1": {"last_replication_method": "LOG_BASED", "lsn": 108240796, "version": 1570922723651, "xmin": null}, "public-city": {"last_replication_method": "INCREMENTAL", "replication_key": "id", "version": 1570922723667, "replication_key_value": 4079}, "public-country": {"last_replication_method": "FULL_TABLE", "version": 1570922730456, "xmin": null}, "public2-wearehere": {}}}} 59 | {"type": "STATE", "value": {"currently_syncing": null, "bookmarks": {"logical1-logical1_edgydata": {"last_replication_method": "LOG_BASED", "lsn": 108240872, "version": 1570922723596, "xmin": null}, "logical1-logical1_table1": {"last_replication_method": "LOG_BASED", "lsn": 108240872, "version": 1570922723618, "xmin": null}, "logical1-logical1_table2": {"last_replication_method": "LOG_BASED", "lsn": 108240872, "version": 1570922723635, "xmin": null}, "logical2-logical2_table1": {"last_replication_method": "LOG_BASED", "lsn": 108240872, "version": 1570922723651, "xmin": null}, "public-city": {"last_replication_method": "INCREMENTAL", "replication_key": "id", "version": 1570922723667, "replication_key_value": 4079}, "public-country": {"last_replication_method": "FULL_TABLE", "version": 1570922730456, "xmin": null}, "public2-wearehere": {}}}} 60 | -------------------------------------------------------------------------------- /target_postgres/db_sync.py: -------------------------------------------------------------------------------- 1 | import json 2 | import sys 3 | import psycopg2 4 | import psycopg2.extras 5 | import inflection 6 | import re 7 | import uuid 8 | import itertools 9 | import time 10 | from collections.abc import MutableMapping 11 | from singer import get_logger 12 | 13 | 14 | # pylint: disable=missing-function-docstring,missing-class-docstring 15 | def validate_config(config): 16 | errors = [] 17 | required_config_keys = [ 18 | 'host', 19 | 'port', 20 | 'user', 21 | 'password', 22 | 'dbname' 23 | ] 24 | 25 | # Check if mandatory keys exist 26 | for k in required_config_keys: 27 | if not config.get(k, None): 28 | errors.append("Required key is missing from config: [{}]".format(k)) 29 | 30 | # Check target schema config 31 | config_default_target_schema = config.get('default_target_schema', None) 32 | config_schema_mapping = config.get('schema_mapping', None) 33 | if not config_default_target_schema and not config_schema_mapping: 34 | errors.append("Neither 'default_target_schema' (string) nor 'schema_mapping' (object) keys set in config.") 35 | 36 | return errors 37 | 38 | 39 | # pylint: disable=fixme 40 | def column_type(schema_property): 41 | property_type = schema_property['type'] 42 | property_format = schema_property['format'] if 'format' in schema_property else None 43 | col_type = 'character varying' 44 | if 'object' in property_type or 'array' in property_type: 45 | col_type = 'jsonb' 46 | 47 | # Every date-time JSON value is currently mapped to TIMESTAMP WITHOUT TIME ZONE 48 | # 49 | # TODO: Detect if timezone postfix exists in the JSON and find if TIMESTAMP WITHOUT TIME ZONE or 50 | # TIMESTAMP WITH TIME ZONE is the better column type 51 | elif property_format == 'date-time': 52 | col_type = 'timestamp without time zone' 53 | elif property_format == 'time': 54 | col_type = 'time without time zone' 55 | elif 'number' in property_type: 56 | col_type = 'double precision' 57 | elif 'integer' in property_type and 'string' in property_type: 58 | col_type = 'character varying' 59 | elif 'integer' in property_type: 60 | if 'maximum' in schema_property: 61 | if schema_property['maximum'] <= 32767: 62 | col_type = 'smallint' 63 | elif schema_property['maximum'] <= 2147483647: 64 | col_type = 'integer' 65 | elif schema_property['maximum'] <= 9223372036854775807: 66 | col_type = 'bigint' 67 | else: 68 | col_type = 'numeric' 69 | elif 'boolean' in property_type: 70 | col_type = 'boolean' 71 | 72 | get_logger('target_postgres').debug("schema_property: %s -> col_type: %s", schema_property, col_type) 73 | 74 | return col_type 75 | 76 | 77 | def safe_column_name(name): 78 | return '"{}"'.format(name).lower() 79 | 80 | 81 | def column_clause(name, schema_property): 82 | return '{} {}'.format(safe_column_name(name), column_type(schema_property)) 83 | 84 | 85 | def flatten_key(k, parent_key, sep): 86 | full_key = parent_key + [k] 87 | inflected_key = full_key.copy() 88 | reducer_index = 0 89 | while len(sep.join(inflected_key)) >= 63 and reducer_index < len(inflected_key): 90 | reduced_key = re.sub(r'[a-z]', '', inflection.camelize(inflected_key[reducer_index])) 91 | inflected_key[reducer_index] = \ 92 | (reduced_key if len(reduced_key) > 1 else inflected_key[reducer_index][0:3]).lower() 93 | reducer_index += 1 94 | 95 | return sep.join(inflected_key) 96 | 97 | 98 | # pylint: disable=dangerous-default-value,invalid-name 99 | def flatten_schema(d, parent_key=[], sep='__', level=0, max_level=0): 100 | items = [] 101 | 102 | if 'properties' not in d: 103 | return {} 104 | 105 | for k, v in d['properties'].items(): 106 | new_key = flatten_key(k, parent_key, sep) 107 | if 'type' in v.keys(): 108 | if 'object' in v['type'] and 'properties' in v and level < max_level: 109 | items.extend(flatten_schema(v, parent_key + [k], sep=sep, level=level + 1, max_level=max_level).items()) 110 | else: 111 | items.append((new_key, v)) 112 | else: 113 | if len(v.values()) > 0: 114 | if list(v.values())[0][0]['type'] == 'string': 115 | list(v.values())[0][0]['type'] = ['null', 'string'] 116 | items.append((new_key, list(v.values())[0][0])) 117 | elif list(v.values())[0][0]['type'] == 'array': 118 | list(v.values())[0][0]['type'] = ['null', 'array'] 119 | items.append((new_key, list(v.values())[0][0])) 120 | elif list(v.values())[0][0]['type'] == 'object': 121 | list(v.values())[0][0]['type'] = ['null', 'object'] 122 | items.append((new_key, list(v.values())[0][0])) 123 | 124 | key_func = lambda item: item[0] 125 | sorted_items = sorted(items, key=key_func) 126 | for k, g in itertools.groupby(sorted_items, key=key_func): 127 | if len(list(g)) > 1: 128 | raise ValueError('Duplicate column name produced in schema: {}'.format(k)) 129 | 130 | return dict(sorted_items) 131 | 132 | 133 | # pylint: disable=redefined-outer-name 134 | def _should_json_dump_value(key, value, flatten_schema=None): 135 | if isinstance(value, (dict, list)): 136 | return True 137 | 138 | if flatten_schema and key in flatten_schema and 'type' in flatten_schema[key]\ 139 | and set(flatten_schema[key]['type']) == {'null', 'object', 'array'}: 140 | return True 141 | 142 | return False 143 | 144 | 145 | # pylint: disable-msg=too-many-arguments 146 | def flatten_record(d, flatten_schema=None, parent_key=[], sep='__', level=0, max_level=0): 147 | items = [] 148 | for k, v in d.items(): 149 | new_key = flatten_key(k, parent_key, sep) 150 | if isinstance(v, MutableMapping) and level < max_level: 151 | items.extend(flatten_record(v, flatten_schema, parent_key + [k], sep=sep, level=level + 1, 152 | max_level=max_level).items()) 153 | else: 154 | items.append((new_key, json.dumps(v) if _should_json_dump_value(k, v, flatten_schema) else v)) 155 | return dict(items) 156 | 157 | 158 | def primary_column_names(stream_schema_message): 159 | return [safe_column_name(p) for p in stream_schema_message['key_properties']] 160 | 161 | 162 | def stream_name_to_dict(stream_name, separator='-'): 163 | catalog_name = None 164 | schema_name = None 165 | table_name = stream_name 166 | 167 | # Schema and table name can be derived from stream if it's in - format 168 | s = stream_name.split(separator) 169 | if len(s) == 2: 170 | schema_name = s[0] 171 | table_name = s[1] 172 | if len(s) > 2: 173 | catalog_name = s[0] 174 | schema_name = s[1] 175 | table_name = '_'.join(s[2:]) 176 | 177 | return { 178 | 'catalog_name': catalog_name, 179 | 'schema_name': schema_name, 180 | 'table_name': table_name 181 | } 182 | 183 | 184 | # pylint: disable=too-many-public-methods,too-many-instance-attributes 185 | class DbSync: 186 | def __init__(self, connection_config, stream_schema_message=None): 187 | """ 188 | connection_config: Postgres connection details 189 | 190 | stream_schema_message: An instance of the DbSync class is typically used to load 191 | data only from a certain singer tap stream. 192 | 193 | The stream_schema_message holds the destination schema 194 | name and the JSON schema that will be used to 195 | validate every RECORDS messages that comes from the stream. 196 | Schema validation happening before creating CSV and before 197 | uploading data into Postgres. 198 | 199 | If stream_schema_message is not defined then we can use 200 | the DbSync instance as a generic purpose connection to 201 | Postgres and can run individual queries. For example 202 | collecting catalog information from Postgres for caching 203 | purposes. 204 | """ 205 | self.connection_config = connection_config 206 | self.stream_schema_message = stream_schema_message 207 | 208 | # logger to be used across the class's methods 209 | self.logger = get_logger('target_postgres') 210 | 211 | # Validate connection configuration 212 | config_errors = validate_config(connection_config) 213 | 214 | # Exit if config has errors 215 | if len(config_errors) > 0: 216 | self.logger.error("Invalid configuration:\n * %s", '\n * '.join(config_errors)) 217 | sys.exit(1) 218 | 219 | self.schema_name = None 220 | self.grantees = None 221 | 222 | # Init stream schema 223 | if stream_schema_message is not None: 224 | # Define initial list of indices to created 225 | self.hard_delete = self.connection_config.get('hard_delete') 226 | if self.hard_delete: 227 | self.indices = ['_sdc_deleted_at'] 228 | else: 229 | self.indices = [] 230 | 231 | # Define target schema name. 232 | # -------------------------- 233 | # Target schema name can be defined in multiple ways: 234 | # 235 | # 1: 'default_target_schema' key : Target schema is the same for every incoming stream if 236 | # not specified explicitly for a given stream in the `schema_mapping` object 237 | # 2: 'schema_mapping' key : Target schema defined explicitly for a given stream. 238 | # Example config.json: 239 | # "schema_mapping": { 240 | # "my_tap_stream_id": { 241 | # "target_schema": "my_postgres_schema", 242 | # "target_schema_select_permissions": [ "role_with_select_privs" ], 243 | # "indices": ["column_1", "column_2s"] 244 | # } 245 | # } 246 | config_default_target_schema = self.connection_config.get('default_target_schema', '').strip() 247 | config_schema_mapping = self.connection_config.get('schema_mapping', {}) 248 | 249 | stream_name = stream_schema_message['stream'] 250 | stream_schema_name = stream_name_to_dict(stream_name)['schema_name'] 251 | stream_table_name = stream_name_to_dict(stream_name)['table_name'] 252 | if config_schema_mapping and stream_schema_name in config_schema_mapping: 253 | self.schema_name = config_schema_mapping[stream_schema_name].get('target_schema') 254 | 255 | # Get indices to create for the target table 256 | indices = config_schema_mapping[stream_schema_name].get('indices', {}) 257 | if stream_table_name in indices: 258 | self.indices.extend(indices.get(stream_table_name, [])) 259 | 260 | elif config_default_target_schema: 261 | self.schema_name = config_default_target_schema 262 | 263 | if not self.schema_name: 264 | raise Exception("Target schema name not defined in config. Neither 'default_target_schema' (string)" 265 | "nor 'schema_mapping' (object) defines target schema for {} stream." 266 | .format(stream_name)) 267 | 268 | # Define grantees 269 | # --------------- 270 | # Grantees can be defined in multiple ways: 271 | # 272 | # 1: 'default_target_schema_select_permissions' key : USAGE and SELECT privileges will be granted on 273 | # every table to a given role for every incoming stream if not specified explicitly in the 274 | # `schema_mapping` object 275 | # 2: 'target_schema_select_permissions' key : Roles to grant USAGE and SELECT privileges defined 276 | # explicitly for a given stream. 277 | # Example config.json: 278 | # "schema_mapping": { 279 | # "my_tap_stream_id": { 280 | # "target_schema": "my_postgres_schema", 281 | # "target_schema_select_permissions": [ "role_with_select_privs" ] 282 | # } 283 | # } 284 | self.grantees = self.connection_config.get('default_target_schema_select_permissions') 285 | if config_schema_mapping and stream_schema_name in config_schema_mapping: 286 | self.grantees = config_schema_mapping[stream_schema_name].get('target_schema_select_permissions', 287 | self.grantees) 288 | 289 | self.data_flattening_max_level = self.connection_config.get('data_flattening_max_level', 0) 290 | self.flatten_schema = flatten_schema(stream_schema_message['schema'], 291 | max_level=self.data_flattening_max_level) 292 | 293 | def open_connection(self): 294 | conn_string = "host='{}' dbname='{}' user='{}' password='{}' port='{}'".format( 295 | self.connection_config['host'], 296 | self.connection_config['dbname'], 297 | self.connection_config['user'], 298 | self.connection_config['password'], 299 | self.connection_config['port'] 300 | ) 301 | 302 | if 'ssl' in self.connection_config and self.connection_config['ssl'] == 'true': 303 | conn_string += " sslmode='require'" 304 | 305 | return psycopg2.connect(conn_string) 306 | 307 | def query(self, query, params=None): 308 | self.logger.debug("Running query: %s", query) 309 | with self.open_connection() as connection: 310 | with connection.cursor(cursor_factory=psycopg2.extras.DictCursor) as cur: 311 | cur.execute( 312 | query, 313 | params 314 | ) 315 | 316 | if cur.rowcount > 0: 317 | return cur.fetchall() 318 | 319 | return [] 320 | 321 | def table_name(self, stream_name, is_temporary=False, without_schema=False): 322 | stream_dict = stream_name_to_dict(stream_name) 323 | table_name = stream_dict['table_name'] 324 | pg_table_name = table_name.replace('.', '_').replace('-', '_').lower() 325 | 326 | if is_temporary: 327 | return 'tmp_{}'.format(str(uuid.uuid4()).replace('-', '_')) 328 | 329 | if without_schema: 330 | return f'"{pg_table_name.lower()}"' 331 | 332 | return f'{self.schema_name}."{pg_table_name.lower()}"' 333 | 334 | def record_primary_key_string(self, record): 335 | if len(self.stream_schema_message['key_properties']) == 0: 336 | return None 337 | flatten = flatten_record(record, self.flatten_schema, max_level=self.data_flattening_max_level) 338 | try: 339 | key_props = [str(flatten[p]) for p in self.stream_schema_message['key_properties']] 340 | except Exception as exc: 341 | self.logger.info("Cannot find %s primary key(s) in record: %s", 342 | self.stream_schema_message['key_properties'], 343 | flatten) 344 | raise exc 345 | return ','.join(key_props) 346 | 347 | def record_to_csv_line(self, record): 348 | flatten = flatten_record(record, self.flatten_schema, max_level=self.data_flattening_max_level) 349 | return ','.join( 350 | [ 351 | json.dumps(flatten[name], ensure_ascii=False) 352 | if name in flatten and (flatten[name] == 0 or flatten[name]) else '' 353 | for name in self.flatten_schema 354 | ] 355 | ) 356 | 357 | def load_csv(self, file, count, size_bytes): 358 | stream_schema_message = self.stream_schema_message 359 | stream = stream_schema_message['stream'] 360 | self.logger.info("Loading %d rows into '%s'", count, self.table_name(stream, False)) 361 | 362 | with self.open_connection() as connection: 363 | with connection.cursor(cursor_factory=psycopg2.extras.DictCursor) as cur: 364 | inserts = 0 365 | updates = 0 366 | 367 | temp_table = self.table_name(stream_schema_message['stream'], is_temporary=True) 368 | cur.execute(self.create_table_query(table_name=temp_table, is_temporary=True)) 369 | 370 | copy_sql = "COPY {} ({}) FROM STDIN WITH (FORMAT CSV, ESCAPE '\\')".format( 371 | temp_table, 372 | ', '.join(self.column_names()) 373 | ) 374 | self.logger.debug(copy_sql) 375 | with open(file, "rb") as f: 376 | cur.copy_expert(copy_sql, f) 377 | if len(self.stream_schema_message['key_properties']) > 0: 378 | cur.execute(self.update_from_temp_table(temp_table)) 379 | updates = cur.rowcount 380 | cur.execute(self.insert_from_temp_table(temp_table)) 381 | inserts = cur.rowcount 382 | 383 | self.logger.info('Loading into %s: %s', 384 | self.table_name(stream, False), 385 | json.dumps({'inserts': inserts, 'updates': updates, 'size_bytes': size_bytes})) 386 | 387 | # pylint: disable=duplicate-string-formatting-argument 388 | def insert_from_temp_table(self, temp_table): 389 | stream_schema_message = self.stream_schema_message 390 | columns = self.column_names() 391 | table = self.table_name(stream_schema_message['stream']) 392 | 393 | if len(stream_schema_message['key_properties']) == 0: 394 | return """INSERT INTO {} ({}) 395 | (SELECT s.* FROM {} s) 396 | """.format(table, 397 | ', '.join(columns), 398 | temp_table) 399 | 400 | return """INSERT INTO {} ({}) 401 | (SELECT s.* FROM {} s LEFT OUTER JOIN {} t ON {} WHERE {}) 402 | """.format(table, 403 | ', '.join(columns), 404 | temp_table, 405 | table, 406 | self.primary_key_condition('t'), 407 | self.primary_key_null_condition('t')) 408 | 409 | def update_from_temp_table(self, temp_table): 410 | stream_schema_message = self.stream_schema_message 411 | columns = self.column_names() 412 | table = self.table_name(stream_schema_message['stream']) 413 | 414 | return """UPDATE {} SET {} FROM {} s 415 | WHERE {} 416 | """.format(table, 417 | ', '.join(['{}=s.{}'.format(c, c) for c in columns]), 418 | temp_table, 419 | self.primary_key_condition(table)) 420 | 421 | def primary_key_condition(self, right_table): 422 | stream_schema_message = self.stream_schema_message 423 | names = primary_column_names(stream_schema_message) 424 | return ' AND '.join(['s.{} = {}.{}'.format(c, right_table, c) for c in names]) 425 | 426 | def primary_key_null_condition(self, right_table): 427 | stream_schema_message = self.stream_schema_message 428 | names = primary_column_names(stream_schema_message) 429 | return ' AND '.join(['{}.{} is null'.format(right_table, c) for c in names]) 430 | 431 | def column_names(self): 432 | return [safe_column_name(name) for name in self.flatten_schema] 433 | 434 | def create_table_query(self, table_name=None, is_temporary=False): 435 | stream_schema_message = self.stream_schema_message 436 | columns = [ 437 | column_clause( 438 | name, 439 | schema 440 | ) 441 | for (name, schema) in self.flatten_schema.items() 442 | ] 443 | 444 | primary_key = ["PRIMARY KEY ({})".format(', '.join(primary_column_names(stream_schema_message)))] \ 445 | if len(stream_schema_message['key_properties']) > 0 else [] 446 | 447 | if not table_name: 448 | gen_table_name = self.table_name(stream_schema_message['stream'], is_temporary=is_temporary) 449 | 450 | return 'CREATE {}TABLE IF NOT EXISTS {} ({})'.format( 451 | 'TEMP ' if is_temporary else '', 452 | table_name if table_name else gen_table_name, 453 | ', '.join(columns + primary_key) 454 | ) 455 | 456 | def grant_usage_on_schema(self, schema_name, grantee): 457 | query = "GRANT USAGE ON SCHEMA {} TO GROUP {}".format(schema_name, grantee) 458 | self.logger.info("Granting USAGE privilege on '%s' schema to '%s'... %s", schema_name, grantee, query) 459 | self.query(query) 460 | 461 | def grant_select_on_all_tables_in_schema(self, schema_name, grantee): 462 | query = "GRANT SELECT ON ALL TABLES IN SCHEMA {} TO GROUP {}".format(schema_name, grantee) 463 | self.logger.info("Granting SELECT ON ALL TABLES privilege on '%s' schema to '%s'... %s", 464 | schema_name, 465 | grantee, 466 | query) 467 | self.query(query) 468 | 469 | @classmethod 470 | def grant_privilege(cls, schema, grantees, grant_method): 471 | if isinstance(grantees, list): 472 | for grantee in grantees: 473 | grant_method(schema, grantee) 474 | elif isinstance(grantees, str): 475 | grant_method(schema, grantees) 476 | 477 | def create_index(self, stream, column): 478 | table = self.table_name(stream) 479 | table_without_schema = self.table_name(stream, without_schema=True) 480 | index_name = 'i_{}_{}'.format(table_without_schema[:30].replace(' ', '').replace('"', ''), 481 | column.replace(',', '_')) 482 | query = "CREATE INDEX IF NOT EXISTS {} ON {} ({})".format(index_name, table, column) 483 | self.logger.info("Creating index on '%s' table on '%s' column(s)... %s", table, column, query) 484 | self.query(query) 485 | 486 | def create_indices(self, stream): 487 | if isinstance(self.indices, list): 488 | for index in self.indices: 489 | self.create_index(stream, index) 490 | 491 | def delete_rows(self, stream): 492 | table = self.table_name(stream) 493 | query = "DELETE FROM {} WHERE _sdc_deleted_at IS NOT NULL RETURNING _sdc_deleted_at".format(table) 494 | self.logger.info("Deleting rows from '%s' table... %s", table, query) 495 | self.logger.info("DELETE %s", len(self.query(query))) 496 | 497 | def create_schema_if_not_exists(self, table_columns_cache=None): 498 | schema_name = self.schema_name 499 | schema_rows = 0 500 | 501 | # table_columns_cache is an optional pre-collected list of available objects in postgres 502 | if table_columns_cache: 503 | schema_rows = list(filter(lambda x: x['TABLE_SCHEMA'] == schema_name, table_columns_cache)) 504 | # Query realtime if not pre-collected 505 | else: 506 | schema_rows = self.query( 507 | 'SELECT LOWER(schema_name) schema_name FROM information_schema.schemata WHERE LOWER(schema_name) = %s', 508 | (schema_name.lower(),) 509 | ) 510 | 511 | if len(schema_rows) == 0: 512 | query = "CREATE SCHEMA IF NOT EXISTS {}".format(schema_name) 513 | self.logger.info("Schema '%s' does not exist. Creating... %s", schema_name, query) 514 | self.query(query) 515 | 516 | self.grant_privilege(schema_name, self.grantees, self.grant_usage_on_schema) 517 | 518 | def get_tables(self): 519 | return self.query( 520 | 'SELECT table_name FROM information_schema.tables WHERE table_schema = %s', 521 | (self.schema_name,) 522 | ) 523 | 524 | def get_table_columns(self, table_name): 525 | return self.query("""SELECT column_name, data_type 526 | FROM information_schema.columns 527 | WHERE lower(table_name) = %s AND lower(table_schema) = %s""", (table_name.replace("\"", "").lower(), 528 | self.schema_name.lower())) 529 | 530 | def update_columns(self): 531 | stream_schema_message = self.stream_schema_message 532 | stream = stream_schema_message['stream'] 533 | table_name = self.table_name(stream, without_schema=True) 534 | columns = self.get_table_columns(table_name) 535 | columns_dict = {column['column_name'].lower(): column for column in columns} 536 | 537 | columns_to_add = [ 538 | column_clause( 539 | name, 540 | properties_schema 541 | ) 542 | for (name, properties_schema) in self.flatten_schema.items() 543 | if name.lower() not in columns_dict 544 | ] 545 | 546 | for column in columns_to_add: 547 | self.add_column(column, stream) 548 | 549 | columns_to_replace = [ 550 | (safe_column_name(name), column_clause( 551 | name, 552 | properties_schema 553 | )) 554 | for (name, properties_schema) in self.flatten_schema.items() 555 | if name.lower() in columns_dict and 556 | columns_dict[name.lower()]['data_type'].lower() != column_type(properties_schema).lower() 557 | ] 558 | 559 | for (column_name, column) in columns_to_replace: 560 | self.version_column(column_name, stream) 561 | self.add_column(column, stream) 562 | 563 | def drop_column(self, column_name, stream): 564 | drop_column = "ALTER TABLE {} DROP COLUMN {}".format(self.table_name(stream), column_name) 565 | self.logger.info('Dropping column: %s', drop_column) 566 | self.query(drop_column) 567 | 568 | def version_column(self, column_name, stream): 569 | version_column = "ALTER TABLE {} RENAME COLUMN {} TO \"{}_{}\"".format(self.table_name(stream, False), 570 | column_name, 571 | column_name.replace("\"", ""), 572 | time.strftime("%Y%m%d_%H%M")) 573 | self.logger.info('Versioning column: %s', version_column) 574 | self.query(version_column) 575 | 576 | def add_column(self, column, stream): 577 | add_column = "ALTER TABLE {} ADD COLUMN {}".format(self.table_name(stream), column) 578 | self.logger.info('Adding column: %s', add_column) 579 | self.query(add_column) 580 | 581 | def sync_table(self): 582 | stream_schema_message = self.stream_schema_message 583 | stream = stream_schema_message['stream'] 584 | table_name = self.table_name(stream, without_schema=True) 585 | found_tables = [table for table in (self.get_tables()) if f'"{table["table_name"].lower()}"' == table_name] 586 | if len(found_tables) == 0: 587 | query = self.create_table_query() 588 | self.logger.info("Table '%s' does not exist. Creating... %s", table_name, query) 589 | self.query(query) 590 | 591 | self.grant_privilege(self.schema_name, self.grantees, self.grant_select_on_all_tables_in_schema) 592 | else: 593 | self.logger.info("Table '%s' exists", table_name) 594 | self.update_columns() 595 | -------------------------------------------------------------------------------- /tests/integration/resources/messages-with-long-texts.json: -------------------------------------------------------------------------------- 1 | {"type": "STATE", "value": {"currently_syncing": "tap_mysql_test-test_table_long_texts"}} 2 | {"type": "SCHEMA", "stream": "tap_mysql_test-test_table_long_texts", "schema": {"properties": {"c_pk": {"inclusion": "automatic", "minimum": -2147483648, "maximum": 2147483647, "type": ["null", "integer"]}, "c_varchar": {"inclusion": "available", "maxLength": 65535, "type": ["null", "string"]}, "c_int": {"inclusion": "available", "minimum": -2147483648, "maximum": 2147483647, "type": ["null", "integer"]}}, "type": "object"}, "key_properties": ["c_pk"]} 3 | {"type": "ACTIVATE_VERSION", "stream": "tap_mysql_test-test_table_long_texts", "version": 1} 4 | {"type": "RECORD", "stream": "tap_mysql_test-test_table_long_texts", "record": {"c_pk": 1, "c_varchar": "Up to 128 characters: Lorem ipsum dolor sit amet, consectetuer adipiscing elit.", "c_int": 1}, "version": 1, "time_extracted": "2019-01-31T15:51:50.215998Z"} 5 | {"type": "RECORD", "stream": "tap_mysql_test-test_table_long_texts", "record": {"c_pk": 2, "c_varchar": "Up to 256 characters: Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Donec quam felis, ultricies.", "c_int": 2}, "version": 1, "time_extracted": "2019-01-31T15:51:50.215998Z"} 6 | {"type": "RECORD", "stream": "tap_mysql_test-test_table_long_texts", "record": {"c_pk": 3, "c_varchar": "Up to 1024 characters: Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Donec quam felis, ultricies nec, pellentesque eu, pretium quis, sem. Nulla consequat massa quis enim. Donec pede justo, fringilla vel, aliquet nec, vulputate eget, arcu. In enim justo, rhoncus ut, imperdiet a, venenatis vitae, justo. Nullam dictum felis eu pede mollis pretium. Integer tincidunt. Cras dapibus. Vivamus elementum semper nisi. Aenean vulputate eleifend tellus. Aenean leo ligula, porttitor eu, consequat vitae, eleifend ac, enim. Aliquam lorem ante, dapibus in, viverra quis, feugiat a, tellus. Phasellus viverra nulla ut metus varius laoreet. Quisque rutrum. Aenean imperdiet. Etiam ultricies nisi vel augue. Curabitur ullamcorper ultricies nisi. Nam eget dui. Etiam rhoncus. Maecenas tempus, tellus eget condimentum rhoncus, sem quam semper libero, sit amet adipiscing sem neque sed ipsum.", "c_int": 3}, "version": 1, "time_extracted": "2019-01-31T15:51:50.215998Z"} 7 | {"type": "RECORD", "stream": "tap_mysql_test-test_table_long_texts", "record": {"c_pk": 4, "c_varchar": "Up to 4096 characters: orem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Donec quam felis, ultricies nec, pellentesque eu, pretium quis, sem. Nulla consequat massa quis enim. Donec pede justo, fringilla vel, aliquet nec, vulputate eget, arcu. In enim justo, rhoncus ut, imperdiet a, venenatis vitae, justo. Nullam dictum felis eu pede mollis pretium. Integer tincidunt. Cras dapibus. Vivamus elementum semper nisi. Aenean vulputate eleifend tellus. Aenean leo ligula, porttitor eu, consequat vitae, eleifend ac, enim. Aliquam lorem ante, dapibus in, viverra quis, feugiat a, tellus. Phasellus viverra nulla ut metus varius laoreet. Quisque rutrum. Aenean imperdiet. Etiam ultricies nisi vel augue. Curabitur ullamcorper ultricies nisi. Nam eget dui. Etiam rhoncus. Maecenas tempus, tellus eget condimentum rhoncus, sem quam semper libero, sit amet adipiscing sem neque sed ipsum. Nam quam nunc, blandit vel, luctus pulvinar, hendrerit id, lorem. Maecenas nec odio et ante tincidunt tempus. Donec vitae sapien ut libero venenatis faucibus. Nullam quis ante. Etiam sit amet orci eget eros faucibus tincidunt. Duis leo. Sed fringilla mauris sit amet nibh. Donec sodales sagittis magna. Sed consequat, leo eget bibendum sodales, augue velit cursus nunc, quis gravida magna mi a libero. Fusce vulputate eleifend sapien. Vestibulum purus quam, scelerisque ut, mollis sed, nonummy id, metus. Nullam accumsan lorem in dui. Cras ultricies mi eu turpis hendrerit fringilla. Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia Curae; In ac dui quis mi consectetuer lacinia. Nam pretium turpis et arcu. Duis arcu tortor, suscipit eget, imperdiet nec, imperdiet iaculis, ipsum. Sed aliquam ultrices mauris. Integer ante arcu, accumsan a, consectetuer eget, posuere ut, mauris. Praesent adipiscing. Phasellus ullamcorper ipsum rutrum nunc. Nunc nonummy metus. Vestibulum volutpat pretium libero. Cras id dui. Aenean ut eros et nisl sagittis vestibulum. Nullam nulla eros, ultricies sit amet, nonummy id, imperdiet feugiat, pede. Sed lectus. Donec mollis hendrerit risus. Phasellus nec sem in justo pellentesque facilisis. Etiam imperdiet imperdiet orci. Nunc nec neque. Phasellus leo dolor, tempus non, auctor et, hendrerit quis, nisi. Curabitur ligula sapien, tincidunt non, euismod vitae, posuere imperdiet, leo. Maecenas malesuada. Praesent congue erat at massa. Sed cursus turpis vitae tortor. Donec posuere vulputate arcu. Phasellus accumsan cursus velit. Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia Curae; Sed aliquam, nisi quis porttitor congue, elit erat euismod orci, ac placerat dolor lectus quis orci. Phasellus consectetuer vestibulum elit. Aenean tellus metus, bibendum sed, posuere ac, mattis non, nunc. Vestibulum fringilla pede sit amet augue. In turpis. Pellentesque posuere. Praesent turpis. Aenean posuere, tortor sed cursus feugiat, nunc augue blandit nunc, eu sollicitudin urna dolor sagittis lacus. Donec elit libero, sodales nec, volutpat a, suscipit non, turpis. Nullam sagittis. Suspendisse pulvinar, augue ac venenatis condimentum, sem libero volutpat nibh, nec pellentesque velit pede quis nunc. Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia Curae; Fusce id purus. Ut varius tincidunt libero. Phasellus dolor. Maecenas vestibulum mollis diam. Pellentesque ut neque. Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. In dui magna, posuere eget, vestibulum et, tempor auctor, justo. In ac felis quis tortor malesuada pretium. Pellentesque auctor neque nec urna. Proin sapien ipsum, porta a, auctor quis, euismod ut, mi. Aenean viverra rhoncus pede. Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. Ut non enim eleifend felis pretium feugiat. Vivamus quis mi. Phasellus a est.", "c_int": 4, "_sdc_deleted_at": "2019-02-10T15:51:50.215998Z"}, "version": 1, "time_extracted": "2019-01-31T15:51:50.215998Z"} 8 | {"type": "RECORD", "stream": "tap_mysql_test-test_table_long_texts", "record": {"c_pk": 5, "c_varchar": "Up to 32786 characters: Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Donec quam felis, ultricies nec, pellentesque eu, pretium quis, sem. Nulla consequat massa quis enim. Donec pede justo, fringilla vel, aliquet nec, vulputate eget, arcu. In enim justo, rhoncus ut, imperdiet a, venenatis vitae, justo. Nullam dictum felis eu pede mollis pretium. Integer tincidunt. Cras dapibus. Vivamus elementum semper nisi. Aenean vulputate eleifend tellus. Aenean leo ligula, porttitor eu, consequat vitae, eleifend ac, enim. Aliquam lorem ante, dapibus in, viverra quis, feugiat a, tellus. Phasellus viverra nulla ut metus varius laoreet. Quisque rutrum. Aenean imperdiet. Etiam ultricies nisi vel augue. Curabitur ullamcorper ultricies nisi. Nam eget dui. Etiam rhoncus. Maecenas tempus, tellus eget condimentum rhoncus, sem quam semper libero, sit amet adipiscing sem neque sed ipsum. Nam quam nunc, blandit vel, luctus pulvinar, hendrerit id, lorem. Maecenas nec odio et ante tincidunt tempus. Donec vitae sapien ut libero venenatis faucibus. Nullam quis ante. Etiam sit amet orci eget eros faucibus tincidunt. Duis leo. Sed fringilla mauris sit amet nibh. Donec sodales sagittis magna. Sed consequat, leo eget bibendum sodales, augue velit cursus nunc, quis gravida magna mi a libero. Fusce vulputate eleifend sapien. Vestibulum purus quam, scelerisque ut, mollis sed, nonummy id, metus. Nullam accumsan lorem in dui. Cras ultricies mi eu turpis hendrerit fringilla. Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia Curae; In ac dui quis mi consectetuer lacinia. Nam pretium turpis et arcu. Duis arcu tortor, suscipit eget, imperdiet nec, imperdiet iaculis, ipsum. Sed aliquam ultrices mauris. Integer ante arcu, accumsan a, consectetuer eget, posuere ut, mauris. Praesent adipiscing. Phasellus ullamcorper ipsum rutrum nunc. Nunc nonummy metus. Vestibulum volutpat pretium libero. Cras id dui. Aenean ut eros et nisl sagittis vestibulum. Nullam nulla eros, ultricies sit amet, nonummy id, imperdiet feugiat, pede. Sed lectus. Donec mollis hendrerit risus. Phasellus nec sem in justo pellentesque facilisis. Etiam imperdiet imperdiet orci. Nunc nec neque. Phasellus leo dolor, tempus non, auctor et, hendrerit quis, nisi. Curabitur ligula sapien, tincidunt non, euismod vitae, posuere imperdiet, leo. Maecenas malesuada. Praesent congue erat at massa. Sed cursus turpis vitae tortor. Donec posuere vulputate arcu. Phasellus accumsan cursus velit. Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia Curae; Sed aliquam, nisi quis porttitor congue, elit erat euismod orci, ac placerat dolor lectus quis orci. Phasellus consectetuer vestibulum elit. Aenean tellus metus, bibendum sed, posuere ac, mattis non, nunc. Vestibulum fringilla pede sit amet augue. In turpis. Pellentesque posuere. Praesent turpis. Aenean posuere, tortor sed cursus feugiat, nunc augue blandit nunc, eu sollicitudin urna dolor sagittis lacus. Donec elit libero, sodales nec, volutpat a, suscipit non, turpis. Nullam sagittis. Suspendisse pulvinar, augue ac venenatis condimentum, sem libero volutpat nibh, nec pellentesque velit pede quis nunc. Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia Curae; Fusce id purus. Ut varius tincidunt libero. Phasellus dolor. Maecenas vestibulum mollis diam. Pellentesque ut neque. Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. In dui magna, posuere eget, vestibulum et, tempor auctor, justo. In ac felis quis tortor malesuada pretium. Pellentesque auctor neque nec urna. Proin sapien ipsum, porta a, auctor quis, euismod ut, mi. Aenean viverra rhoncus pede. Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. Ut non enim eleifend felis pretium feugiat. Vivamus quis mi. Phasellus a est. Phasellus magna. In hac habitasse platea dictumst. Curabitur at lacus ac velit ornare lobortis. Curabitur a felis in nunc fringilla tristique. Morbi mattis ullamcorper velit. Phasellus gravida semper nisi. Nullam vel sem. Pellentesque libero tortor, tincidunt et, tincidunt eget, semper nec, quam. Sed hendrerit. Morbi ac felis. Nunc egestas, augue at pellentesque laoreet, felis eros vehicula leo, at malesuada velit leo quis pede. Donec interdum, metus et hendrerit aliquet, dolor diam sagittis ligula, eget egestas libero turpis vel mi. Nunc nulla. Fusce risus nisl, viverra et, tempor et, pretium in, sapien. Donec venenatis vulputate lorem. Morbi nec metus. Phasellus blandit leo ut odio. Maecenas ullamcorper, dui et placerat feugiat, eros pede varius nisi, condimentum viverra felis nunc et lorem. Sed magna purus, fermentum eu, tincidunt eu, varius ut, felis. In auctor lobortis lacus. Quisque libero metus, condimentum nec, tempor a, commodo mollis, magna. Vestibulum ullamcorper mauris at ligula. Fusce fermentum. Nullam cursus lacinia erat. Praesent blandit laoreet nibh. Fusce convallis metus id felis luctus adipiscing. Pellentesque egestas, neque sit amet convallis pulvinar, justo nulla eleifend augue, ac auctor orci leo non est. Quisque id mi. Ut tincidunt tincidunt erat. Etiam feugiat lorem non metus. Vestibulum dapibus nunc ac augue. Curabitur vestibulum aliquam leo. Praesent egestas neque eu enim. In hac habitasse platea dictumst. Fusce a quam. Etiam ut purus mattis mauris sodales aliquam. Curabitur nisi. Quisque malesuada placerat nisl. Nam ipsum risus, rutrum vitae, vestibulum eu, molestie vel, lacus. Sed augue ipsum, egestas nec, vestibulum et, malesuada adipiscing, dui. Vestibulum facilisis, purus nec pulvinar iaculis, ligula mi congue nunc, vitae euismod ligula urna in dolor. Mauris sollicitudin fermentum libero. Praesent nonummy mi in odio. Nunc interdum lacus sit amet orci. Vestibulum rutrum, mi nec elementum vehicula, eros quam gravida nisl, id fringilla neque ante vel mi. Morbi mollis tellus ac sapien. Phasellus volutpat, metus eget egestas mollis, lacus lacus blandit dui, id egestas quam mauris ut lacus. Fusce vel dui. Sed in libero ut nibh placerat accumsan. Proin faucibus arcu quis ante. In consectetuer turpis ut velit. Nulla sit amet est. Praesent metus tellus, elementum eu, semper a, adipiscing nec, purus. Cras risus ipsum, faucibus ut, ullamcorper id, varius ac, leo. Suspendisse feugiat. Suspendisse enim turpis, dictum sed, iaculis a, condimentum nec, nisi. Praesent nec nisl a purus blandit viverra. Praesent ac massa at ligula laoreet iaculis. Nulla neque dolor, sagittis eget, iaculis quis, molestie non, velit. Mauris turpis nunc, blandit et, volutpat molestie, porta ut, ligula. Fusce pharetra convallis urna. Quisque ut nisi. Donec mi odio, faucibus at, scelerisque quis, convallis in, nisi. Suspendisse non nisl sit amet velit hendrerit rutrum. Ut leo. Ut a nisl id ante tempus hendrerit. Proin pretium, leo ac pellentesque mollis, felis nunc ultrices eros, sed gravida augue augue mollis justo. Suspendisse eu ligula. Nulla facilisi. Donec id justo. Praesent porttitor, nulla vitae posuere iaculis, arcu nisl dignissim dolor, a pretium mi sem ut ipsum. Curabitur suscipit suscipit tellus. Praesent vestibulum dapibus nibh. Etiam iaculis nunc ac metus. Ut id nisl quis enim dignissim sagittis. Etiam sollicitudin, ipsum eu pulvinar rutrum, tellus ipsum laoreet sapien, quis venenatis ante odio sit amet eros. Proin magna. Duis vel nibh at velit scelerisque suscipit. Curabitur turpis. Vestibulum suscipit nulla quis orci. Fusce ac felis sit amet ligula pharetra condimentum. Maecenas egestas arcu quis ligula mattis placerat. Duis lobortis massa imperdiet quam. Suspendisse potenti. Pellentesque commodo eros a enim. Vestibulum turpis sem, aliquet eget, lobortis pellentesque, rutrum eu, nisl. Sed libero. Aliquam erat volutpat. Etiam vitae tortor. Morbi vestibulum volutpat enim. Aliquam eu nunc. Nunc sed turpis. Sed mollis, eros et ultrices tempus, mauris ipsum aliquam libero, non adipiscing dolor urna a orci. Nulla porta dolor. Class aptent taciti sociosqu ad litora torquent per conubia nostra, per inceptos hymenaeos. Pellentesque dapibus hendrerit tortor. Praesent egestas tristique nibh. Sed a libero. Cras varius. Donec vitae orci sed dolor rutrum auctor. Fusce egestas elit eget lorem. Suspendisse nisl elit, rhoncus eget, elementum ac, condimentum eget, diam. Nam at tortor in tellus interdum sagittis. Aliquam lobortis. Donec orci lectus, aliquam ut, faucibus non, euismod id, nulla. Curabitur blandit mollis lacus. Nam adipiscing. Vestibulum eu odio. Vivamus laoreet. Nullam tincidunt adipiscing enim. Phasellus tempus. Proin viverra, ligula sit amet ultrices semper, ligula arcu tristique sapien, a accumsan nisi mauris ac eros. Fusce neque. Suspendisse faucibus, nunc et pellentesque egestas, lacus ante convallis tellus, vitae iaculis lacus elit id tortor. Vivamus aliquet elit ac nisl. Fusce fermentum odio nec arcu. Vivamus euismod mauris. In ut quam vitae odio lacinia tincidunt. Praesent ut ligula non mi varius sagittis. Cras sagittis. Praesent ac sem eget est egestas volutpat. Vivamus consectetuer hendrerit lacus. Cras non dolor. Vivamus in erat ut urna cursus vestibulum. Fusce commodo aliquam arcu. Nam commodo suscipit quam. Quisque id odio. Praesent venenatis metus at tortor pulvinar varius. Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Donec quam felis, ultricies nec, pellentesque eu, pretium quis, sem. Nulla consequat massa quis enim. Donec pede justo, fringilla vel, aliquet nec, vulputate eget, arcu. In enim justo, rhoncus ut, imperdiet a, venenatis vitae, justo. Nullam dictum felis eu pede mollis pretium. Integer tincidunt. Cras dapibus. Vivamus elementum semper nisi. Aenean vulputate eleifend tellus. Aenean leo ligula, porttitor eu, consequat vitae, eleifend ac, enim. Aliquam lorem ante, dapibus in, viverra quis, feugiat a, tellus. Phasellus viverra nulla ut metus varius laoreet. Quisque rutrum. Aenean imperdiet. Etiam ultricies nisi vel augue. Curabitur ullamcorper ultricies nisi. Nam eget dui. Etiam rhoncus. Maecenas tempus, tellus eget condimentum rhoncus, sem quam semper libero, sit amet adipiscing sem neque sed ipsum. Nam quam nunc, blandit vel, luctus pulvinar, hendrerit id, lorem. Maecenas nec odio et ante tincidunt tempus. Donec vitae sapien ut libero venenatis faucibus. Nullam quis ante. Etiam sit amet orci eget eros faucibus tincidunt. Duis leo. Sed fringilla mauris sit amet nibh. Donec sodales sagittis magna. Sed consequat, leo eget bibendum sodales, augue velit cursus nunc, quis gravida magna mi a libero. Fusce vulputate eleifend sapien. Vestibulum purus quam, scelerisque ut, mollis sed, nonummy id, metus. Nullam accumsan lorem in dui. Cras ultricies mi eu turpis hendrerit fringilla. Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia Curae; In ac dui quis mi consectetuer lacinia. Nam pretium turpis et arcu. Duis arcu tortor, suscipit eget, imperdiet nec, imperdiet iaculis, ipsum. Sed aliquam ultrices mauris. Integer ante arcu, accumsan a, consectetuer eget, posuere ut, mauris. Praesent adipiscing. Phasellus ullamcorper ipsum rutrum nunc. Nunc nonummy metus. Vestibulum volutpat pretium libero. Cras id dui. Aenean ut eros et nisl sagittis vestibulum. Nullam nulla eros, ultricies sit amet, nonummy id, imperdiet feugiat, pede. Sed lectus. Donec mollis hendrerit risus. Phasellus nec sem in justo pellentesque facilisis. Etiam imperdiet imperdiet orci. Nunc nec neque. Phasellus leo dolor, tempus non, auctor et, hendrerit quis, nisi. Curabitur ligula sapien, tincidunt non, euismod vitae, posuere imperdiet, leo. Maecenas malesuada. Praesent congue erat at massa. Sed cursus turpis vitae tortor. Donec posuere vulputate arcu. Phasellus accumsan cursus velit. Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia Curae; Sed aliquam, nisi quis porttitor congue, elit erat euismod orci, ac placerat dolor lectus quis orci. Phasellus consectetuer vestibulum elit. Aenean tellus metus, bibendum sed, posuere ac, mattis non, nunc. Vestibulum fringilla pede sit amet augue. In turpis. Pellentesque posuere. Praesent turpis. Aenean posuere, tortor sed cursus feugiat, nunc augue blandit nunc, eu sollicitudin urna dolor sagittis lacus. Donec elit libero, sodales nec, volutpat a, suscipit non, turpis. Nullam sagittis. Suspendisse pulvinar, augue ac venenatis condimentum, sem libero volutpat nibh, nec pellentesque velit pede quis nunc. Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia Curae; Fusce id purus. Ut varius tincidunt libero. Phasellus dolor. Maecenas vestibulum mollis diam. Pellentesque ut neque. Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. In dui magna, posuere eget, vestibulum et, tempor auctor, justo. In ac felis quis tortor malesuada pretium. Pellentesque auctor neque nec urna. Proin sapien ipsum, porta a, auctor quis, euismod ut, mi. Aenean viverra rhoncus pede. Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. Ut non enim eleifend felis pretium feugiat. Vivamus quis mi. Phasellus a est. Phasellus magna. In hac habitasse platea dictumst. Curabitur at lacus ac velit ornare lobortis. Curabitur a felis in nunc fringilla tristique. Morbi mattis ullamcorper velit. Phasellus gravida semper nisi. Nullam vel sem. Pellentesque libero tortor, tincidunt et, tincidunt eget, semper nec, quam. Sed hendrerit. Morbi ac felis. Nunc egestas, augue at pellentesque laoreet, felis eros vehicula leo, at malesuada velit leo quis pede. Donec interdum, metus et hendrerit aliquet, dolor diam sagittis ligula, eget egestas libero turpis vel mi. Nunc nulla. Fusce risus nisl, viverra et, tempor et, pretium in, sapien. Donec venenatis vulputate lorem. Morbi nec metus. Phasellus blandit leo ut odio. Maecenas ullamcorper, dui et placerat feugiat, eros pede varius nisi, condimentum viverra felis nunc et lorem. Sed magna purus, fermentum eu, tincidunt eu, varius ut, felis. In auctor lobortis lacus. Quisque libero metus, condimentum nec, tempor a, commodo mollis, magna. Vestibulum ullamcorper mauris at ligula. Fusce fermentum. Nullam cursus lacinia erat. Praesent blandit laoreet nibh. Fusce convallis metus id felis luctus adipiscing. Pellentesque egestas, neque sit amet convallis pulvinar, justo nulla eleifend augue, ac auctor orci leo non est. Quisque id mi. Ut tincidunt tincidunt erat. Etiam feugiat lorem non metus. Vestibulum dapibus nunc ac augue. Curabitur vestibulum aliquam leo. Praesent egestas neque eu enim. In hac habitasse platea dictumst. Fusce a quam. Etiam ut purus mattis mauris sodales aliquam. Curabitur nisi. Quisque malesuada placerat nisl. Nam ipsum risus, rutrum vitae, vestibulum eu, molestie vel, lacus. Sed augue ipsum, egestas nec, vestibulum et, malesuada adipiscing, dui. Vestibulum facilisis, purus nec pulvinar iaculis, ligula mi congue nunc, vitae euismod ligula urna in dolor. Mauris sollicitudin fermentum libero. Praesent nonummy mi in odio. Nunc interdum lacus sit amet orci. Vestibulum rutrum, mi nec elementum vehicula, eros quam gravida nisl, id fringilla neque ante vel mi. Morbi mollis tellus ac sapien. Phasellus volutpat, metus eget egestas mollis, lacus lacus blandit dui, id egestas quam mauris ut lacus. Fusce vel dui. Sed in libero ut nibh placerat accumsan. Proin faucibus arcu quis ante. In consectetuer turpis ut velit. Nulla sit amet est. Praesent metus tellus, elementum eu, semper a, adipiscing nec, purus. Cras risus ipsum, faucibus ut, ullamcorper id, varius ac, leo. Suspendisse feugiat. Suspendisse enim turpis, dictum sed, iaculis a, condimentum nec, nisi. Praesent nec nisl a purus blandit viverra. Praesent ac massa at ligula laoreet iaculis. Nulla neque dolor, sagittis eget, iaculis quis, molestie non, velit. Mauris turpis nunc, blandit et, volutpat molestie, porta ut, ligula. Fusce pharetra convallis urna. Quisque ut nisi. Donec mi odio, faucibus at, scelerisque quis, convallis in, nisi. Suspendisse non nisl sit amet velit hendrerit rutrum. Ut leo. Ut a nisl id ante tempus hendrerit. Proin pretium, leo ac pellentesque mollis, felis nunc ultrices eros, sed gravida augue augue mollis justo. Suspendisse eu ligula. Nulla facilisi. Donec id justo. Praesent porttitor, nulla vitae posuere iaculis, arcu nisl dignissim dolor, a pretium mi sem ut ipsum. Curabitur suscipit suscipit tellus. Praesent vestibulum dapibus nibh. Etiam iaculis nunc ac metus. Ut id nisl quis enim dignissim sagittis. Etiam sollicitudin, ipsum eu pulvinar rutrum, tellus ipsum laoreet sapien, quis venenatis ante odio sit amet eros. Proin magna. Duis vel nibh at velit scelerisque suscipit. Curabitur turpis. Vestibulum suscipit nulla quis orci. Fusce ac felis sit amet ligula pharetra condimentum. Maecenas egestas arcu quis ligula mattis placerat. Duis lobortis massa imperdiet quam. Suspendisse potenti. Pellentesque commodo eros a enim. Vestibulum turpis sem, aliquet eget, lobortis pellentesque, rutrum eu, nisl. Sed libero. Aliquam erat volutpat. Etiam vitae tortor. Morbi vestibulum volutpat enim. Aliquam eu nunc. Nunc sed turpis. Sed mollis, eros et ultrices tempus, mauris ipsum aliquam libero, non adipiscing dolor urna a orci. Nulla porta dolor. Class aptent taciti sociosqu ad litora torquent per conubia nostra, per inceptos hymenaeos. Pellentesque dapibus hendrerit tortor. Praesent egestas tristique nibh. Sed a libero. Cras varius. Donec vitae orci sed dolor rutrum auctor. Fusce egestas elit eget lorem. Suspendisse nisl elit, rhoncus eget, elementum ac, condimentum eget, diam. Nam at tortor in tellus interdum sagittis. Aliquam lobortis. Donec orci lectus, aliquam ut, faucibus non, euismod id, nulla. Curabitur blandit mollis lacus. Nam adipiscing. Vestibulum eu odio. Vivamus laoreet. Nullam tincidunt adipiscing enim. Phasellus tempus. Proin viverra, ligula sit amet ultrices semper, ligula arcu tristique sapien, a accumsan nisi mauris ac eros. Fusce neque. Suspendisse faucibus, nunc et pellentesque egestas, lacus ante convallis tellus, vitae iaculis lacus elit id tortor. Vivamus aliquet elit ac nisl. Fusce fermentum odio nec arcu. Vivamus euismod mauris. In ut quam vitae odio lacinia tincidunt. Praesent ut ligula non mi varius sagittis. Cras sagittis. Praesent ac sem eget est egestas volutpat. Vivamus consectetuer hendrerit lacus. Cras non dolor. Vivamus in erat ut urna cursus vestibulum. Fusce commodo aliquam arcu. Nam commodo suscipit quam. Quisque id odio. Praesent venenatis metus at tortor pulvinar varius. Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Donec quam felis, ultricies nec, pellentesque eu, pretium quis, sem. Nulla consequat massa quis enim. Donec pede justo, fringilla vel, aliquet nec, vulputate eget, arcu. In enim justo, rhoncus ut, imperdiet a, venenatis vitae, justo. Nullam dictum felis eu pede mollis pretium. Integer tincidunt. Cras dapibus. Vivamus elementum semper nisi. Aenean vulputate eleifend tellus. Aenean leo ligula, porttitor eu, consequat vitae, eleifend ac, enim. Aliquam lorem ante, dapibus in, viverra quis, feugiat a, tellus. Phasellus viverra nulla ut metus varius laoreet. Quisque rutrum. Aenean imperdiet. Etiam ultricies nisi vel augue. Curabitur ullamcorper ultricies nisi. Nam eget dui. Etiam rhoncus. Maecenas tempus, tellus eget condimentum rhoncus, sem quam semper libero, sit amet adipiscing sem neque sed ipsum. Nam quam nunc, blandit vel, luctus pulvinar, hendrerit id, lorem. Maecenas nec odio et ante tincidunt tempus. Donec vitae sapien ut libero venenatis faucibus. Nullam quis ante. Etiam sit amet orci eget eros faucibus tincidunt. Duis leo. Sed fringilla mauris sit amet nibh. Donec sodales sagittis magna. Sed consequat, leo eget bibendum sodales, augue velit cursus nunc, quis gravida magna mi a libero. Fusce vulputate eleifend sapien. Vestibulum purus quam, scelerisque ut, mollis sed, nonummy id, metus. Nullam accumsan lorem in dui. Cras ultricies mi eu turpis hendrerit fringilla. Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia Curae; In ac dui quis mi consectetuer lacinia. Nam pretium turpis et arcu. Duis arcu tortor, suscipit eget, imperdiet nec, imperdiet iaculis, ipsum. Sed aliquam ultrices mauris. Integer ante arcu, accumsan a, consectetuer eget, posuere ut, mauris. Praesent adipiscing. Phasellus ullamcorper ipsum rutrum nunc. Nunc nonummy metus. Vestibulum volutpat pretium libero. Cras id dui. Aenean ut eros et nisl sagittis vestibulum. Nullam nulla eros, ultricies sit amet, nonummy id, imperdiet feugiat, pede. Sed lectus. Donec mollis hendrerit risus. Phasellus nec sem in justo pellentesque facilisis. Etiam imperdiet imperdiet orci. Nunc nec neque. Phasellus leo dolor, tempus non, auctor et, hendrerit quis, nisi. Curabitur ligula sapien, tincidunt non, euismod vitae, posuere imperdiet, leo. Maecenas malesuada. Praesent congue erat at massa. Sed cursus turpis vitae tortor. Donec posuere vulputate arcu. Phasellus accumsan cursus velit. Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia Curae; Sed aliquam, nisi quis porttitor congue, elit erat euismod orci, ac placerat dolor lectus quis orci. Phasellus consectetuer vestibulum elit. Aenean tellus metus, bibendum sed, posuere ac, mattis non, nunc. Vestibulum fringilla pede sit amet augue. In turpis. Pellentesque posuere. Praesent turpis. Aenean posuere, tortor sed cursus feugiat, nunc augue blandit nunc, eu sollicitudin urna dolor sagittis lacus. Donec elit libero, sodales nec, volutpat a, suscipit non, turpis. Nullam sagittis. Suspendisse pulvinar, augue ac venenatis condimentum, sem libero volutpat nibh, nec pellentesque velit pede quis nunc. Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia Curae; Fusce id purus. Ut varius tincidunt libero. Phasellus dolor. Maecenas vestibulum mollis diam. Pellentesque ut neque. Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. In dui magna, posuere eget, vestibulum et, tempor auctor, justo. In ac felis quis tortor malesuada pretium. Pellentesque auctor neque nec urna. Proin sapien ipsum, porta a, auctor quis, euismod ut, mi. Aenean viverra rhoncus pede. Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. Ut non enim eleifend felis pretium feugiat. Vivamus quis mi. Phasellus a est. Phasellus magna. In hac habitasse platea dictumst. Curabitur at lacus ac velit ornare lobortis. Curabitur a felis in nunc fringilla tristique. Morbi mattis ullamcorper velit. Phasellus gravida semper nisi. Nullam vel sem. Pellentesque libero tortor, tincidunt et, tincidunt eget, semper nec, quam. Sed hendrerit. Morbi ac felis. Nunc egestas, augue at pellentesque laoreet, felis eros vehicula leo, at malesuada velit leo quis pede. Donec interdum, metus et hendrerit aliquet, dolor diam sagittis ligula, eget egestas libero turpis vel mi. Nunc nulla. Fusce risus nisl, viverra et, tempor et, pretium in, sapien. Donec venenatis vulputate lorem. Morbi nec metus. Phasellus blandit leo ut odio. Maecenas ullamcorper, dui et placerat feugiat, eros pede varius nisi, condimentum viverra felis nunc et lorem. Sed magna purus, fermentum eu, tincidunt eu, varius ut, felis. In auctor lobortis lacus. Quisque libero metus, condimentum nec, tempor a, commodo mollis, magna. Vestibulum ullamcorper mauris at ligula. Fusce fermentum. Nullam cursus lacinia erat. Praesent blandit laoreet nibh. Fusce convallis metus id felis luctus adipiscing. Pellentesque egestas, neque sit amet convallis pulvinar, justo nulla eleifend augue, ac auctor orci leo non est. Quisque id mi. Ut tincidunt tincidunt erat. Etiam feugiat lorem non metus. Vestibulum dapibus nunc ac augue. Curabitur vestibulum aliquam leo. Praesent egestas neque eu enim. In hac habitasse platea dictumst. Fusce a quam. Etiam ut purus mattis mauris sodales aliquam. Curabitur nisi. Quisque malesuada placerat nisl. Nam ipsum risus, rutrum vitae, vestibulum eu, molestie vel, lacus. Sed augue ipsum, egestas nec, vestibulum et, malesuada adipiscing, dui. Vestibulum facilisis, purus nec pulvinar iaculis, ligula mi congue nunc, vitae euismod ligula urna in dolor. Mauris sollicitudin fermentum libero. Praesent nonummy mi in odio. Nunc interdum lacus sit amet orci. Vestibulum rutrum, mi nec elementum vehicula, eros quam gravida nisl, id fringilla neque ante vel mi. Morbi mollis tellus ac sapien. Phasellus volutpat, metus eget egestas mollis, lacus lacus blandit dui, id egestas quam mauris ut lacus. Fusce vel dui. Sed in libero ut nibh placerat accumsan. Proin faucibus arcu quis ante. In consectetuer turpis ut velit. Nulla sit amet est. Praesent metus tellus, elementum eu, semper a, adipiscing nec, purus. Cras risus ipsum, faucibus ut, ullamcorper id, varius ac, leo. Suspendisse feugiat. Suspendisse enim turpis, dictum sed, iaculis a, condimentum nec, nisi. Praesent nec nisl a purus blandit viverra. Praesent ac massa at ligula laoreet iaculis. Nulla neque dolor, sagittis eget, iaculis quis, molestie non, velit. Mauris turpis nunc, blandit et, volutpat molestie, porta ut, ligula. Fusce pharetra convallis urna. Quisque ut nisi. Donec mi odio, faucibus at, scelerisque quis, convallis in, nisi. Suspendisse non nisl sit amet velit hendrerit rutrum. Ut leo. Ut a nisl id ante tempus hendrerit. Proin pretium, leo ac pellentesque mollis, felis nunc ultrices eros, sed gravida augue augue mollis justo. Suspendisse eu ligula. Nulla facilisi. Donec id justo. Praesent porttitor, nulla vitae posuere iaculis, arcu nisl dignissim dolor, a pretium mi sem ut ipsum. Curabitur suscipit suscipit tellus. Praesent vestibulum dapibus nibh. Etiam iaculis nunc ac metus. Ut id nisl quis enim dignissim sagittis. Etiam sollicitudin, ipsum eu pulvinar rutrum, tellus ipsum laoreet sapien, quis venenatis ante odio sit amet eros. Proin magna. Duis vel nibh at velit scelerisque suscipit. Curabitur turpis. Vestibulum suscipit nulla quis orci. Fusce ac felis sit amet ligula pharetra condimentum. Maecenas egestas arcu quis ligula mattis placerat. Duis lobortis massa imperdiet quam. Suspendisse potenti. Pellentesque commodo eros a enim. Vestibulum turpis sem, aliquet eget, lobortis pellentesque, rutrum eu, nisl. Sed libero. Aliquam erat volutpat. Etiam vitae tortor. Morbi vestibulum volutpat enim. Aliquam eu nunc. Nunc sed turpis. Sed mollis, eros et ultrices tempus, mauris ipsum aliquam libero, non adipiscing dolor urna a orci. Nulla porta dolor. Class aptent taciti sociosqu ad litora torquent per conubia nostra, per inceptos hymenaeos. Pellentesque dapibus hendrerit tortor. Praesent egestas tristique nibh. Sed a libero. Cras varius. Donec vitae orci sed dolor rutrum auctor. Fusce egestas elit eget lorem. Suspendisse nisl elit, rhoncus eget, elementum ac, condimentum eget, diam. Nam at tortor in tellus interdum sagittis. Aliquam lobortis. Donec orci lectus, aliquam ut, faucibus non, euismod id, nulla. Curabitur blandit mollis lacus. Nam adipiscing. Vestibulum eu odio. Vivamus laoreet. Nullam tincidunt adipiscing enim. Phasellus tempus. Proin viverra, ligula sit amet ultrices semper, ligula arcu tristique sapien, a accumsan nisi mauris ac eros. Fusce neque. Suspendisse faucibus, nunc et pellentesque egestas, lacus ante convallis tellus, vitae iaculis lacus elit id tortor. Vivamus aliquet elit ac nisl. Fusce fermentum odio nec arcu. Vivamus euismod mauris. In ut quam vitae odio lacinia tincidunt. Praesent ut ligula non mi varius sagittis. Cras sagittis. Praesent ac sem eget est egestas volutpat. Vivamus consectetuer hendrerit lacus. Cras non dolor. Vivamus in erat ut urna cursus vestibulum. Fusce commodo aliquam arcu. Nam commodo suscipit quam. Quisque id odio. Praesent venenatis metus at tortor pulvinar varius. Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Donec quam felis, ultricies nec, pellentesque eu, pretium quis, sem. Nulla consequat massa quis enim. Donec pede justo, fringilla vel, aliquet nec, vulputate eget, arcu. In enim justo, rhoncus ut, imperdiet a, venenatis vitae, justo. Nullam dictum felis eu pede mollis pretium. Integer tincidunt. Cras dapibus. Vivamus elementum semper nisi. Aenean vulputate eleifend tellus. Aenean leo ligula, porttitor eu, consequat vitae, eleifend ac, enim. Aliquam lorem ante, dapibus in, viverra quis, feugiat a, tellus. Phasellus viverra nulla ut metus varius laoreet. Quisque rutrum. Aenean imperdiet. Etiam ultricies nisi vel augue. Curabitur ullamcorper ultricies nisi. Nam eget dui. Etiam rhoncus. Maecenas tempus, tellus eget condimentum rhoncus, sem quam semper libero, sit amet adipiscing sem neque sed ipsum. Nam quam nunc, blandit vel, luctus pulvinar, hendrerit id, lorem. Maecenas nec odio et ante tincidunt tempus. Donec vitae sapien ut libero venenatis faucibus. Nullam quis ante. Etiam sit amet orci eget eros faucibus tincidunt. Duis leo. Sed fringilla mauris sit amet nibh. Donec sodales sagittis magna. Sed consequat, leo eget bibendum sodales, augue velit cursus nunc, quis gravida magna mi a libero. Fusce vulputate eleifend sapien. Vestibulum purus quam, scelerisque ut, mollis sed, nonummy id, metus. Nullam accumsan lorem in dui. Cras ultricies mi eu turpis hendrerit fringilla. Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia Curae; In ac dui quis mi consectetuer lacinia. Nam pretium turpis et arcu. Duis arcu tortor, suscipit eget, imperdiet nec, imperdiet iaculis, ipsum. Sed aliquam ultrices mauris. Integer ante arcu, accumsan a, consectetuer eget, posuere ut, mauris. Praesent adipiscing. Phasellus ullamcorper ipsum rutrum nunc. Nunc nonummy metus. Vestibulum volutpat pretium libero. Cras id dui. Aenean ut eros et nisl sagittis vestibulum. Nullam nulla eros, ultricies sit amet, nonummy id, imperdiet feugiat, pede. Sed lectus. Donec mollis hendrerit risus. Phasellus nec sem in justo pellentesque facilisis. Etiam imperdiet imperdiet orci. Nunc nec neque. Phasellus leo dolor, tempus non, auctor et, hendrerit quis, nisi. Curabitur ligula sapien, tincidunt non, euismod vitae, posuere imperdiet, leo. Maecenas malesuada. Praesent congue erat at massa. Sed cursus turpis vitae tortor. Donec posuere vulputate arcu. Phasellus accumsan cursus velit. Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia Curae; Sed aliquam, nisi quis porttitor congue, elit erat euismod orci, ac placerat dolor lectus quis orci. Phasellus consectetuer vestibulum elit. Aenean tellus metus, bibendum sed, posuere ac, mattis non, nunc. Vestibulum fringilla pede sit amet augue. In turpis. Pellentesque posuere. Praesent turpis. Aenean posuere, tortor sed cursus feugiat, nunc augue blandit nunc, eu sollicitudin urna dolor sagittis lacus. Donec elit libero, sodales nec, volutpat a, suscipit non, turpis. Nullam sagittis. Suspendisse pulvinar, augue ac venenatis condimentum, sem libero volutpat nibh, nec pellentesque velit pede quis nunc. Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia Curae; Fusce id purus. Ut varius tincidunt libero. Phasellus dolor. Maecenas vestibulum mollis diam. Pellentesque ut neque. Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas. In dui magna, posuere eget, vestibulum et, tempor auctor, justo. In ac felis quis tortor malesuada pretium. Pellentesque auctor neque nec urna. Proin sapien ipsum, porta a, auctor quis, euismod ut, mi. Aenean viverra rhoncus pede. ", "c_int": 5, "_sdc_deleted_at": "2019-02-10T15:51:50.215998Z"}, "version": 1, "time_extracted": "2019-01-31T15:51:50.215998Z"} 9 | {"type": "STATE", "value": {"currently_syncing": "tap_mysql_test-test_table_long_texts", "bookmarks": {"tap_mysql_test-test_table_one": {"initial_full_table_complete": true}, "tap_mysql_test-test_table_two": {"initial_full_table_complete": true}}}} 10 | {"type": "ACTIVATE_VERSION", "stream": "tap_mysql_test-test_table_long_texts", "version": 1} 11 | {"type": "STATE", "value": {"currently_syncing": null, "bookmarks": {"tap_mysql_test-test_table_long_texts": {"initial_full_table_complete": true}}}} 12 | --------------------------------------------------------------------------------