├── .github └── workflows │ └── python-publish.yml ├── .gitignore ├── CODE_OF_CONDUCT.md ├── DOCUMENTATION.md ├── Docs ├── Autodiscovery.md ├── BigQuery.md ├── ColumnMapping.md ├── CommonScenarios │ └── HistoricalData.md ├── ConfigExamples.md ├── Configuration.md ├── ConfigurationCode.md ├── ConfigurationSettings.md ├── ConvertToWatermark.md ├── DataExpiration.md ├── DataMaintenance.md ├── Destinations.md ├── Examples │ ├── Configuration_Code_Example.ipynb │ ├── Fabric-Sync-Installer-Billing-Export.ipynb │ └── Watermark_Upgrade.ipynb ├── Fabric.md ├── Installation.md ├── LoadGroups.md ├── Logging.md ├── ObjectTypes.md ├── Optimizations.md ├── ReleaseLog.md ├── RotateCredentials.md ├── Scheduling.md ├── Synchronization.md ├── TableConfiguration.md ├── UpdateBQConnector.md └── images │ └── load_groups.png ├── LICENSE ├── Notebooks └── v2.0.0 │ ├── Fabric-Sync-Environment.ipynb │ ├── Fabric-Sync-Installer.ipynb │ ├── Fabric-Sync-Maintenance-Environment.ipynb │ ├── Fabric-Sync-Maintenance.ipynb │ └── Fabric-Sync.ipynb ├── Packages └── FabricSync │ └── FabricSync │ ├── BQ │ ├── APIClient.py │ ├── Auth.py │ ├── BigQueryAPI.py │ ├── Constants.py │ ├── Core.py │ ├── Enum.py │ ├── Exceptions.py │ ├── Experimental.py │ ├── Expiration.py │ ├── FileSystem.py │ ├── GoogleStorageAPI.py │ ├── Http.py │ ├── Lakehouse.py │ ├── Loader.py │ ├── Logging.py │ ├── Maintenance.py │ ├── Metadata.py │ ├── Metastore.py │ ├── Mirror.py │ ├── Model │ │ ├── Config.py │ │ ├── Core.py │ │ ├── Maintenance.py │ │ ├── Schedule.py │ │ └── __init__.py │ ├── ModelValidation.py │ ├── Schedule.py │ ├── SessionManager.py │ ├── Setup.py │ ├── Sync.py │ ├── SyncCore.py │ ├── SyncUtils.py │ ├── Threading.py │ ├── Utils.py │ ├── Validation.py │ ├── Warnings.py │ └── __init__.py │ ├── Meta.py │ └── __init__.py ├── README.md ├── SECURITY.md ├── SUPPORT.md ├── Setup └── v2.0.0 │ └── Data │ └── bq_data_types.csv ├── fabricsync.png ├── pyproject.toml ├── requirements.txt └── uv.lock /.github/workflows/python-publish.yml: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/.github/workflows/python-publish.yml -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/.gitignore -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/CODE_OF_CONDUCT.md -------------------------------------------------------------------------------- /DOCUMENTATION.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/DOCUMENTATION.md -------------------------------------------------------------------------------- /Docs/Autodiscovery.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/Docs/Autodiscovery.md -------------------------------------------------------------------------------- /Docs/BigQuery.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/Docs/BigQuery.md -------------------------------------------------------------------------------- /Docs/ColumnMapping.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/Docs/ColumnMapping.md -------------------------------------------------------------------------------- /Docs/CommonScenarios/HistoricalData.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/Docs/CommonScenarios/HistoricalData.md -------------------------------------------------------------------------------- /Docs/ConfigExamples.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/Docs/ConfigExamples.md -------------------------------------------------------------------------------- /Docs/Configuration.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/Docs/Configuration.md -------------------------------------------------------------------------------- /Docs/ConfigurationCode.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/Docs/ConfigurationCode.md -------------------------------------------------------------------------------- /Docs/ConfigurationSettings.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/Docs/ConfigurationSettings.md -------------------------------------------------------------------------------- /Docs/ConvertToWatermark.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/Docs/ConvertToWatermark.md -------------------------------------------------------------------------------- /Docs/DataExpiration.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/Docs/DataExpiration.md -------------------------------------------------------------------------------- /Docs/DataMaintenance.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/Docs/DataMaintenance.md -------------------------------------------------------------------------------- /Docs/Destinations.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/Docs/Destinations.md -------------------------------------------------------------------------------- /Docs/Examples/Configuration_Code_Example.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/Docs/Examples/Configuration_Code_Example.ipynb -------------------------------------------------------------------------------- /Docs/Examples/Fabric-Sync-Installer-Billing-Export.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/Docs/Examples/Fabric-Sync-Installer-Billing-Export.ipynb -------------------------------------------------------------------------------- /Docs/Examples/Watermark_Upgrade.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/Docs/Examples/Watermark_Upgrade.ipynb -------------------------------------------------------------------------------- /Docs/Fabric.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/Docs/Fabric.md -------------------------------------------------------------------------------- /Docs/Installation.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/Docs/Installation.md -------------------------------------------------------------------------------- /Docs/LoadGroups.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/Docs/LoadGroups.md -------------------------------------------------------------------------------- /Docs/Logging.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/Docs/Logging.md -------------------------------------------------------------------------------- /Docs/ObjectTypes.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/Docs/ObjectTypes.md -------------------------------------------------------------------------------- /Docs/Optimizations.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/Docs/Optimizations.md -------------------------------------------------------------------------------- /Docs/ReleaseLog.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/Docs/ReleaseLog.md -------------------------------------------------------------------------------- /Docs/RotateCredentials.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/Docs/RotateCredentials.md -------------------------------------------------------------------------------- /Docs/Scheduling.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/Docs/Scheduling.md -------------------------------------------------------------------------------- /Docs/Synchronization.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/Docs/Synchronization.md -------------------------------------------------------------------------------- /Docs/TableConfiguration.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/Docs/TableConfiguration.md -------------------------------------------------------------------------------- /Docs/UpdateBQConnector.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/Docs/UpdateBQConnector.md -------------------------------------------------------------------------------- /Docs/images/load_groups.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/Docs/images/load_groups.png -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/LICENSE -------------------------------------------------------------------------------- /Notebooks/v2.0.0/Fabric-Sync-Environment.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/Notebooks/v2.0.0/Fabric-Sync-Environment.ipynb -------------------------------------------------------------------------------- /Notebooks/v2.0.0/Fabric-Sync-Installer.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/Notebooks/v2.0.0/Fabric-Sync-Installer.ipynb -------------------------------------------------------------------------------- /Notebooks/v2.0.0/Fabric-Sync-Maintenance-Environment.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/Notebooks/v2.0.0/Fabric-Sync-Maintenance-Environment.ipynb -------------------------------------------------------------------------------- /Notebooks/v2.0.0/Fabric-Sync-Maintenance.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/Notebooks/v2.0.0/Fabric-Sync-Maintenance.ipynb -------------------------------------------------------------------------------- /Notebooks/v2.0.0/Fabric-Sync.ipynb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/Notebooks/v2.0.0/Fabric-Sync.ipynb -------------------------------------------------------------------------------- /Packages/FabricSync/FabricSync/BQ/APIClient.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/Packages/FabricSync/FabricSync/BQ/APIClient.py -------------------------------------------------------------------------------- /Packages/FabricSync/FabricSync/BQ/Auth.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/Packages/FabricSync/FabricSync/BQ/Auth.py -------------------------------------------------------------------------------- /Packages/FabricSync/FabricSync/BQ/BigQueryAPI.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/Packages/FabricSync/FabricSync/BQ/BigQueryAPI.py -------------------------------------------------------------------------------- /Packages/FabricSync/FabricSync/BQ/Constants.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/Packages/FabricSync/FabricSync/BQ/Constants.py -------------------------------------------------------------------------------- /Packages/FabricSync/FabricSync/BQ/Core.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/Packages/FabricSync/FabricSync/BQ/Core.py -------------------------------------------------------------------------------- /Packages/FabricSync/FabricSync/BQ/Enum.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/Packages/FabricSync/FabricSync/BQ/Enum.py -------------------------------------------------------------------------------- /Packages/FabricSync/FabricSync/BQ/Exceptions.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/Packages/FabricSync/FabricSync/BQ/Exceptions.py -------------------------------------------------------------------------------- /Packages/FabricSync/FabricSync/BQ/Experimental.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/Packages/FabricSync/FabricSync/BQ/Experimental.py -------------------------------------------------------------------------------- /Packages/FabricSync/FabricSync/BQ/Expiration.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/Packages/FabricSync/FabricSync/BQ/Expiration.py -------------------------------------------------------------------------------- /Packages/FabricSync/FabricSync/BQ/FileSystem.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/Packages/FabricSync/FabricSync/BQ/FileSystem.py -------------------------------------------------------------------------------- /Packages/FabricSync/FabricSync/BQ/GoogleStorageAPI.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/Packages/FabricSync/FabricSync/BQ/GoogleStorageAPI.py -------------------------------------------------------------------------------- /Packages/FabricSync/FabricSync/BQ/Http.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/Packages/FabricSync/FabricSync/BQ/Http.py -------------------------------------------------------------------------------- /Packages/FabricSync/FabricSync/BQ/Lakehouse.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/Packages/FabricSync/FabricSync/BQ/Lakehouse.py -------------------------------------------------------------------------------- /Packages/FabricSync/FabricSync/BQ/Loader.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/Packages/FabricSync/FabricSync/BQ/Loader.py -------------------------------------------------------------------------------- /Packages/FabricSync/FabricSync/BQ/Logging.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/Packages/FabricSync/FabricSync/BQ/Logging.py -------------------------------------------------------------------------------- /Packages/FabricSync/FabricSync/BQ/Maintenance.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/Packages/FabricSync/FabricSync/BQ/Maintenance.py -------------------------------------------------------------------------------- /Packages/FabricSync/FabricSync/BQ/Metadata.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/Packages/FabricSync/FabricSync/BQ/Metadata.py -------------------------------------------------------------------------------- /Packages/FabricSync/FabricSync/BQ/Metastore.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/Packages/FabricSync/FabricSync/BQ/Metastore.py -------------------------------------------------------------------------------- /Packages/FabricSync/FabricSync/BQ/Mirror.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/Packages/FabricSync/FabricSync/BQ/Mirror.py -------------------------------------------------------------------------------- /Packages/FabricSync/FabricSync/BQ/Model/Config.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/Packages/FabricSync/FabricSync/BQ/Model/Config.py -------------------------------------------------------------------------------- /Packages/FabricSync/FabricSync/BQ/Model/Core.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/Packages/FabricSync/FabricSync/BQ/Model/Core.py -------------------------------------------------------------------------------- /Packages/FabricSync/FabricSync/BQ/Model/Maintenance.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/Packages/FabricSync/FabricSync/BQ/Model/Maintenance.py -------------------------------------------------------------------------------- /Packages/FabricSync/FabricSync/BQ/Model/Schedule.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/Packages/FabricSync/FabricSync/BQ/Model/Schedule.py -------------------------------------------------------------------------------- /Packages/FabricSync/FabricSync/BQ/Model/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /Packages/FabricSync/FabricSync/BQ/ModelValidation.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/Packages/FabricSync/FabricSync/BQ/ModelValidation.py -------------------------------------------------------------------------------- /Packages/FabricSync/FabricSync/BQ/Schedule.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/Packages/FabricSync/FabricSync/BQ/Schedule.py -------------------------------------------------------------------------------- /Packages/FabricSync/FabricSync/BQ/SessionManager.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/Packages/FabricSync/FabricSync/BQ/SessionManager.py -------------------------------------------------------------------------------- /Packages/FabricSync/FabricSync/BQ/Setup.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/Packages/FabricSync/FabricSync/BQ/Setup.py -------------------------------------------------------------------------------- /Packages/FabricSync/FabricSync/BQ/Sync.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/Packages/FabricSync/FabricSync/BQ/Sync.py -------------------------------------------------------------------------------- /Packages/FabricSync/FabricSync/BQ/SyncCore.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/Packages/FabricSync/FabricSync/BQ/SyncCore.py -------------------------------------------------------------------------------- /Packages/FabricSync/FabricSync/BQ/SyncUtils.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/Packages/FabricSync/FabricSync/BQ/SyncUtils.py -------------------------------------------------------------------------------- /Packages/FabricSync/FabricSync/BQ/Threading.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/Packages/FabricSync/FabricSync/BQ/Threading.py -------------------------------------------------------------------------------- /Packages/FabricSync/FabricSync/BQ/Utils.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/Packages/FabricSync/FabricSync/BQ/Utils.py -------------------------------------------------------------------------------- /Packages/FabricSync/FabricSync/BQ/Validation.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/Packages/FabricSync/FabricSync/BQ/Validation.py -------------------------------------------------------------------------------- /Packages/FabricSync/FabricSync/BQ/Warnings.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/Packages/FabricSync/FabricSync/BQ/Warnings.py -------------------------------------------------------------------------------- /Packages/FabricSync/FabricSync/BQ/__init__.py: -------------------------------------------------------------------------------- 1 | 2 | -------------------------------------------------------------------------------- /Packages/FabricSync/FabricSync/Meta.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/Packages/FabricSync/FabricSync/Meta.py -------------------------------------------------------------------------------- /Packages/FabricSync/FabricSync/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/README.md -------------------------------------------------------------------------------- /SECURITY.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/SECURITY.md -------------------------------------------------------------------------------- /SUPPORT.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/SUPPORT.md -------------------------------------------------------------------------------- /Setup/v2.0.0/Data/bq_data_types.csv: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/Setup/v2.0.0/Data/bq_data_types.csv -------------------------------------------------------------------------------- /fabricsync.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/fabricsync.png -------------------------------------------------------------------------------- /pyproject.toml: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/pyproject.toml -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/requirements.txt -------------------------------------------------------------------------------- /uv.lock: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/FabricBQSync/HEAD/uv.lock --------------------------------------------------------------------------------