├── LICENSE ├── README.md ├── files └── deployment_file.json ├── media ├── analytical_pathway_assets │ ├── ap_high_cu_sql_analytics_endpoint.png │ ├── ap_high_cu_sql_analytics_endpoint_situation_1.png │ ├── ap_sm_healthiness.png │ └── ap_sm_refresh_to_details.png ├── gateway_monitoring_assets │ ├── gwm_screenshot_1.png │ ├── gwm_screenshot_10.png │ ├── gwm_screenshot_2.png │ ├── gwm_screenshot_3.png │ ├── gwm_screenshot_4.png │ ├── gwm_screenshot_5.png │ ├── gwm_screenshot_6.png │ ├── gwm_screenshot_7.png │ ├── gwm_screenshot_8.png │ └── gwm_screenshot_9.png ├── general │ ├── FUAM_basic_architecture.png │ ├── fuam_basic_reporting_cover.png │ ├── fuam_basic_video_cover.png │ ├── fuam_cover.png │ ├── fuam_cover_main.png │ ├── fuam_logo.png │ ├── fuam_monitoring_map_cover.png │ └── fuam_text_logo.png ├── screenshots │ ├── report_screenshot_0.png │ ├── report_screenshot_1.png │ ├── report_screenshot_10.png │ ├── report_screenshot_11.png │ ├── report_screenshot_12.png │ ├── report_screenshot_13.png │ ├── report_screenshot_14.png │ ├── report_screenshot_15.png │ ├── report_screenshot_2.png │ ├── report_screenshot_3.png │ ├── report_screenshot_4.png │ ├── report_screenshot_5.png │ ├── report_screenshot_6.png │ ├── report_screenshot_7.png │ ├── report_screenshot_8.png │ └── report_screenshot_9.png ├── sqlea_assets │ ├── SQLEA_screenshot_0.png │ ├── SQLEA_screenshot_1.png │ ├── SQLEA_screenshot_2.png │ ├── SQLEA_screenshot_3.png │ ├── SQLEA_screenshot_4.png │ └── SQLEA_screenshot_5.png ├── tmmda_assets │ ├── TMMDA_cover.png │ ├── TMMDA_how_to_connect.png │ ├── TMMDA_screenshot_1.png │ ├── TMMDA_screenshot_10.png │ ├── TMMDA_screenshot_2.png │ ├── TMMDA_screenshot_3.png │ ├── TMMDA_screenshot_4.png │ ├── TMMDA_screenshot_5.png │ ├── TMMDA_screenshot_6.png │ ├── TMMDA_screenshot_7.png │ ├── TMMDA_screenshot_8.png │ └── TMMDA_screenshot_9.png └── wiki_deployment │ ├── FUAM_basic_deployment_process_3_1.png │ ├── FUAM_basic_deployment_process_3_2.png │ ├── FUAM_basic_deployment_process_4_1.png │ ├── FUAM_basic_deployment_process_5_1.png │ ├── FUAM_basic_deployment_process_6_1.png │ ├── FUAM_basic_deployment_process_7_1.png │ ├── FUAM_basic_deployment_process_cover.png │ ├── FUAM_basic_upgrade_process_cover.png │ └── icon_FUAM_workspace.png └── start └── Deploy_FUAM_Basic_Gen2.ipynb /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2024 GT Analytics 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Fabric Unified Admin Monitoring (FUAM) 2 | 3 | ```diff 4 | - FUAM is now part of the Fabric Toolbox and will be actively developed there. 5 | - As part of this step we also merged both version. 6 | - The complete FUAM solution available as an open-source solution. 7 | ``` 8 | **If you want to get more information or deploy the latest version, head over to Fabric toolbox:** 9 | 10 | ### [Explore FUAM in fabric-toolbox](https://github.com/microsoft/fabric-toolbox/tree/main/monitoring/fabric-unified-admin-monitoring) 11 | 12 | 13 | 14 | ----------------------- 15 | ----------------------- 16 | ----------------------- 17 | 18 | 19 | 20 | ![image](./media/general/fuam_cover_main.png) 21 | 22 | > [!CAUTION] 23 | > The FUAM solution accelerator is not an official Microsoft product! It is a solution accelerator, which can help you implement a monitoring solution within Fabric. As such there is no offical support available and there is a risk that things might break. 24 | 25 | ## Introduction 26 | 27 | Fabric Unfied Admin Monitoring (short: FUAM) is a solution to enable a holistic monitoring on top of Power BI and Fabric. 28 | Today monitoring for Fabric can be done through different reports, apps and tools. Here is a short overview about the available monitoring solutions which are shipped with Fabric: 29 | - Feature Usage & Adoption 30 | - Purview Hub 31 | - Capacity Metrics App 32 | - Workspace Monitoring 33 | - Usage Metrics Report 34 | 35 | 36 | FUAM has the goal to provide a more holistic view on top of the various information, which can be extracted from Fabric, allowing it's users to analyze at a very high level, but also to deep dive into specific artifacts for a more fine granular data analysis. 37 | 38 | [Watch FUAM Introduction video on Youtube](https://www.youtube.com/watch?v=Ai71Xzr_2Ds) 39 | 40 | ![image](./media/general/fuam_monitoring_map_cover.png) 41 | 42 | FUAM is completely build with Fabric capabilities with Pipelines and Notebooks as the main tool to extract and transform data. All of the data is stored in it's raw format, but also in Delta Parquet, enabling the user to directly use it through Power BI Direct Lake or the Lakehouse SQL Endpoint via SQL. 43 | FUAM comes with a set of standard report enabling a quick overview on the data, but it's intended to give the users all tool to customize or build the own reports on top of the data model. Through the openness of Fabric it's of course also possible to combine it with your own data enabling you to do the analysis you wish. 44 | 45 | 46 | ## FUAM Basic Content 47 | FUAM Basic extracts the following data from the tenant: 48 | - Tenant Settings 49 | - Delegated Tenant Settings 50 | - Activities 51 | - Workspaces 52 | - Capacities 53 | - Tenant meta data (Scanner API) 54 | - Capacity Refreshables 55 | - Git Connections 56 | 57 | 58 | ### Architecture 59 | 60 | The architecture of FUAM is built on **Fabric items** like Pipelines, Notebooks, Lakehouses, Semantic models and Power BI reports. 61 | We have built the component in a **modular structure**, which helps you to extend FUAM with your own modules. This architecture design helps to maintain the solution also with ease. 62 | 63 | The **data ingestion** logic is orchastrated and parametizable, which allows to use the main orchestration pipeline for initial and incremental data loads. 64 | **FUAM Lakehouse** is one of the core component in the architecture. All the data is transformed and persisted in a way, which open amazing capabilities analyzing the collected data in a semantic model with DirectLake mode. 65 | ![image](./media/general/FUAM_basic_architecture.png) 66 | 67 | Architectural documentation can be found on our Wiki page: [FUAM Architecture](https://github.com/GT-Analytics/fuam-basic/wiki/Architecture) 68 | 69 | ------------------------------------ 70 | 71 | # FUAM Basic Deployment 72 | 73 | Are you ready to try FUAM out? We have prepared two step-by-step documentations, which support you to deploy FUAM in your workspace on your tenant. 74 | 75 | [Click here to **deploy** FUAM Basic](https://github.com/GT-Analytics/fuam-basic/wiki/Lifecycle:-Initial-Deployment-via-Notebook) 76 | 77 | [Click here to **update** FUAM Basic](https://github.com/GT-Analytics/fuam-basic/wiki/Lifecycle:-Upgrading-via-Notebook) 78 | 79 | ------------------------------------ 80 | 81 | # Other Documentation 82 | 83 | We have collected all insights, documentation and helpful links in our wiki. 84 | 85 | [Go to FUAM wiki](https://github.com/GT-Analytics/fuam-basic/wiki) 86 | 87 | ## Remarks & Limitations 88 | 89 | There are couple of important information, which you should know. 90 | Please visit for more details the 91 | [Read Remarks on wiki](https://github.com/GT-Analytics/fuam-basic/wiki/Remarks-to-FUAM) 92 | 93 | 94 | ## Important 95 | 96 | - This is a community project that is **not** supported by Microsoft. 97 | - **The code in this repository and any related information are without warranty of any kind.** 98 | - The deployed solution will utilize CUs (Compute Units) on your Microsoft Fabric SKU on your PBI/Fabric tenant. 99 | - FUAM Basic has been tested on some large tenants, however the CU utilization depends on multiple factors like: count of Fabric SKUs, count of workspaces, count of users, count of changes within the tenant, etc. 100 | - Please test the solution on a non-production Fabric SKU first without impacting other workloads on your tenant. 101 | 102 | 103 | ------------------------------------ 104 | 105 | # FUAM Reporting | Screenshots 106 | 107 | FUAM comes with a set of standard report enabling a quick overview on the data, but it's intended to give the users all tool to customize or build the own reports on top of the data model. 108 | 109 | ![image](./media/general/fuam_basic_reporting_cover.png) 110 | 111 | #### FUAM Reporting | Tenant Overview | Screenshots 112 | Report name in Workspace: **FUAM_Basic_Overview_Report** 113 | 114 | ![image](./media/screenshots/report_screenshot_1.png) 115 | 116 | ![image](./media/screenshots/report_screenshot_2.png) 117 | 118 | ![image](./media/screenshots/report_screenshot_3.png) 119 | 120 | ![image](./media/screenshots/report_screenshot_4.png) 121 | 122 | ![image](./media/screenshots/report_screenshot_5.png) 123 | 124 | ![image](./media/screenshots/report_screenshot_6.png) 125 | 126 | ![image](./media/screenshots/report_screenshot_7.png) 127 | 128 | ![image](./media/screenshots/report_screenshot_8.png) 129 | 130 | ![image](./media/screenshots/report_screenshot_9.png) 131 | 132 | ![image](./media/screenshots/report_screenshot_10.png) 133 | 134 | ![image](./media/screenshots/report_screenshot_11.png) 135 | 136 | ![image](./media/screenshots/report_screenshot_12.png) 137 | 138 | ![image](./media/screenshots/report_screenshot_13.png) 139 | 140 | ![image](./media/screenshots/report_screenshot_14.png) 141 | 142 | ![image](./media/screenshots/report_screenshot_15.png) 143 | 144 | -------------------------- 145 | 146 | 147 | ### Logo 148 |

149 | 150 |

151 | -------------------------------------------------------------------------------- /media/analytical_pathway_assets/ap_high_cu_sql_analytics_endpoint.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GT-Analytics/fuam-basic/cbfd190f41c29c209caf1a4d96458761d71e42f2/media/analytical_pathway_assets/ap_high_cu_sql_analytics_endpoint.png -------------------------------------------------------------------------------- /media/analytical_pathway_assets/ap_high_cu_sql_analytics_endpoint_situation_1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GT-Analytics/fuam-basic/cbfd190f41c29c209caf1a4d96458761d71e42f2/media/analytical_pathway_assets/ap_high_cu_sql_analytics_endpoint_situation_1.png -------------------------------------------------------------------------------- /media/analytical_pathway_assets/ap_sm_healthiness.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GT-Analytics/fuam-basic/cbfd190f41c29c209caf1a4d96458761d71e42f2/media/analytical_pathway_assets/ap_sm_healthiness.png -------------------------------------------------------------------------------- /media/analytical_pathway_assets/ap_sm_refresh_to_details.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GT-Analytics/fuam-basic/cbfd190f41c29c209caf1a4d96458761d71e42f2/media/analytical_pathway_assets/ap_sm_refresh_to_details.png -------------------------------------------------------------------------------- /media/gateway_monitoring_assets/gwm_screenshot_1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GT-Analytics/fuam-basic/cbfd190f41c29c209caf1a4d96458761d71e42f2/media/gateway_monitoring_assets/gwm_screenshot_1.png -------------------------------------------------------------------------------- /media/gateway_monitoring_assets/gwm_screenshot_10.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GT-Analytics/fuam-basic/cbfd190f41c29c209caf1a4d96458761d71e42f2/media/gateway_monitoring_assets/gwm_screenshot_10.png -------------------------------------------------------------------------------- /media/gateway_monitoring_assets/gwm_screenshot_2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GT-Analytics/fuam-basic/cbfd190f41c29c209caf1a4d96458761d71e42f2/media/gateway_monitoring_assets/gwm_screenshot_2.png -------------------------------------------------------------------------------- /media/gateway_monitoring_assets/gwm_screenshot_3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GT-Analytics/fuam-basic/cbfd190f41c29c209caf1a4d96458761d71e42f2/media/gateway_monitoring_assets/gwm_screenshot_3.png -------------------------------------------------------------------------------- /media/gateway_monitoring_assets/gwm_screenshot_4.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GT-Analytics/fuam-basic/cbfd190f41c29c209caf1a4d96458761d71e42f2/media/gateway_monitoring_assets/gwm_screenshot_4.png -------------------------------------------------------------------------------- /media/gateway_monitoring_assets/gwm_screenshot_5.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GT-Analytics/fuam-basic/cbfd190f41c29c209caf1a4d96458761d71e42f2/media/gateway_monitoring_assets/gwm_screenshot_5.png -------------------------------------------------------------------------------- /media/gateway_monitoring_assets/gwm_screenshot_6.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GT-Analytics/fuam-basic/cbfd190f41c29c209caf1a4d96458761d71e42f2/media/gateway_monitoring_assets/gwm_screenshot_6.png -------------------------------------------------------------------------------- /media/gateway_monitoring_assets/gwm_screenshot_7.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GT-Analytics/fuam-basic/cbfd190f41c29c209caf1a4d96458761d71e42f2/media/gateway_monitoring_assets/gwm_screenshot_7.png -------------------------------------------------------------------------------- /media/gateway_monitoring_assets/gwm_screenshot_8.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GT-Analytics/fuam-basic/cbfd190f41c29c209caf1a4d96458761d71e42f2/media/gateway_monitoring_assets/gwm_screenshot_8.png -------------------------------------------------------------------------------- /media/gateway_monitoring_assets/gwm_screenshot_9.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GT-Analytics/fuam-basic/cbfd190f41c29c209caf1a4d96458761d71e42f2/media/gateway_monitoring_assets/gwm_screenshot_9.png -------------------------------------------------------------------------------- /media/general/FUAM_basic_architecture.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GT-Analytics/fuam-basic/cbfd190f41c29c209caf1a4d96458761d71e42f2/media/general/FUAM_basic_architecture.png -------------------------------------------------------------------------------- /media/general/fuam_basic_reporting_cover.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GT-Analytics/fuam-basic/cbfd190f41c29c209caf1a4d96458761d71e42f2/media/general/fuam_basic_reporting_cover.png -------------------------------------------------------------------------------- /media/general/fuam_basic_video_cover.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GT-Analytics/fuam-basic/cbfd190f41c29c209caf1a4d96458761d71e42f2/media/general/fuam_basic_video_cover.png -------------------------------------------------------------------------------- /media/general/fuam_cover.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GT-Analytics/fuam-basic/cbfd190f41c29c209caf1a4d96458761d71e42f2/media/general/fuam_cover.png -------------------------------------------------------------------------------- /media/general/fuam_cover_main.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GT-Analytics/fuam-basic/cbfd190f41c29c209caf1a4d96458761d71e42f2/media/general/fuam_cover_main.png -------------------------------------------------------------------------------- /media/general/fuam_logo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GT-Analytics/fuam-basic/cbfd190f41c29c209caf1a4d96458761d71e42f2/media/general/fuam_logo.png -------------------------------------------------------------------------------- /media/general/fuam_monitoring_map_cover.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GT-Analytics/fuam-basic/cbfd190f41c29c209caf1a4d96458761d71e42f2/media/general/fuam_monitoring_map_cover.png -------------------------------------------------------------------------------- /media/general/fuam_text_logo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GT-Analytics/fuam-basic/cbfd190f41c29c209caf1a4d96458761d71e42f2/media/general/fuam_text_logo.png -------------------------------------------------------------------------------- /media/screenshots/report_screenshot_0.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GT-Analytics/fuam-basic/cbfd190f41c29c209caf1a4d96458761d71e42f2/media/screenshots/report_screenshot_0.png -------------------------------------------------------------------------------- /media/screenshots/report_screenshot_1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GT-Analytics/fuam-basic/cbfd190f41c29c209caf1a4d96458761d71e42f2/media/screenshots/report_screenshot_1.png -------------------------------------------------------------------------------- /media/screenshots/report_screenshot_10.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GT-Analytics/fuam-basic/cbfd190f41c29c209caf1a4d96458761d71e42f2/media/screenshots/report_screenshot_10.png -------------------------------------------------------------------------------- /media/screenshots/report_screenshot_11.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GT-Analytics/fuam-basic/cbfd190f41c29c209caf1a4d96458761d71e42f2/media/screenshots/report_screenshot_11.png -------------------------------------------------------------------------------- /media/screenshots/report_screenshot_12.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GT-Analytics/fuam-basic/cbfd190f41c29c209caf1a4d96458761d71e42f2/media/screenshots/report_screenshot_12.png -------------------------------------------------------------------------------- /media/screenshots/report_screenshot_13.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GT-Analytics/fuam-basic/cbfd190f41c29c209caf1a4d96458761d71e42f2/media/screenshots/report_screenshot_13.png -------------------------------------------------------------------------------- /media/screenshots/report_screenshot_14.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GT-Analytics/fuam-basic/cbfd190f41c29c209caf1a4d96458761d71e42f2/media/screenshots/report_screenshot_14.png -------------------------------------------------------------------------------- /media/screenshots/report_screenshot_15.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GT-Analytics/fuam-basic/cbfd190f41c29c209caf1a4d96458761d71e42f2/media/screenshots/report_screenshot_15.png -------------------------------------------------------------------------------- /media/screenshots/report_screenshot_2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GT-Analytics/fuam-basic/cbfd190f41c29c209caf1a4d96458761d71e42f2/media/screenshots/report_screenshot_2.png -------------------------------------------------------------------------------- /media/screenshots/report_screenshot_3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GT-Analytics/fuam-basic/cbfd190f41c29c209caf1a4d96458761d71e42f2/media/screenshots/report_screenshot_3.png -------------------------------------------------------------------------------- /media/screenshots/report_screenshot_4.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GT-Analytics/fuam-basic/cbfd190f41c29c209caf1a4d96458761d71e42f2/media/screenshots/report_screenshot_4.png -------------------------------------------------------------------------------- /media/screenshots/report_screenshot_5.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GT-Analytics/fuam-basic/cbfd190f41c29c209caf1a4d96458761d71e42f2/media/screenshots/report_screenshot_5.png -------------------------------------------------------------------------------- /media/screenshots/report_screenshot_6.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GT-Analytics/fuam-basic/cbfd190f41c29c209caf1a4d96458761d71e42f2/media/screenshots/report_screenshot_6.png -------------------------------------------------------------------------------- /media/screenshots/report_screenshot_7.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GT-Analytics/fuam-basic/cbfd190f41c29c209caf1a4d96458761d71e42f2/media/screenshots/report_screenshot_7.png -------------------------------------------------------------------------------- /media/screenshots/report_screenshot_8.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GT-Analytics/fuam-basic/cbfd190f41c29c209caf1a4d96458761d71e42f2/media/screenshots/report_screenshot_8.png -------------------------------------------------------------------------------- /media/screenshots/report_screenshot_9.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GT-Analytics/fuam-basic/cbfd190f41c29c209caf1a4d96458761d71e42f2/media/screenshots/report_screenshot_9.png -------------------------------------------------------------------------------- /media/sqlea_assets/SQLEA_screenshot_0.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GT-Analytics/fuam-basic/cbfd190f41c29c209caf1a4d96458761d71e42f2/media/sqlea_assets/SQLEA_screenshot_0.png -------------------------------------------------------------------------------- /media/sqlea_assets/SQLEA_screenshot_1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GT-Analytics/fuam-basic/cbfd190f41c29c209caf1a4d96458761d71e42f2/media/sqlea_assets/SQLEA_screenshot_1.png -------------------------------------------------------------------------------- /media/sqlea_assets/SQLEA_screenshot_2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GT-Analytics/fuam-basic/cbfd190f41c29c209caf1a4d96458761d71e42f2/media/sqlea_assets/SQLEA_screenshot_2.png -------------------------------------------------------------------------------- /media/sqlea_assets/SQLEA_screenshot_3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GT-Analytics/fuam-basic/cbfd190f41c29c209caf1a4d96458761d71e42f2/media/sqlea_assets/SQLEA_screenshot_3.png -------------------------------------------------------------------------------- /media/sqlea_assets/SQLEA_screenshot_4.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GT-Analytics/fuam-basic/cbfd190f41c29c209caf1a4d96458761d71e42f2/media/sqlea_assets/SQLEA_screenshot_4.png -------------------------------------------------------------------------------- /media/sqlea_assets/SQLEA_screenshot_5.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GT-Analytics/fuam-basic/cbfd190f41c29c209caf1a4d96458761d71e42f2/media/sqlea_assets/SQLEA_screenshot_5.png -------------------------------------------------------------------------------- /media/tmmda_assets/TMMDA_cover.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GT-Analytics/fuam-basic/cbfd190f41c29c209caf1a4d96458761d71e42f2/media/tmmda_assets/TMMDA_cover.png -------------------------------------------------------------------------------- /media/tmmda_assets/TMMDA_how_to_connect.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GT-Analytics/fuam-basic/cbfd190f41c29c209caf1a4d96458761d71e42f2/media/tmmda_assets/TMMDA_how_to_connect.png -------------------------------------------------------------------------------- /media/tmmda_assets/TMMDA_screenshot_1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GT-Analytics/fuam-basic/cbfd190f41c29c209caf1a4d96458761d71e42f2/media/tmmda_assets/TMMDA_screenshot_1.png -------------------------------------------------------------------------------- /media/tmmda_assets/TMMDA_screenshot_10.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GT-Analytics/fuam-basic/cbfd190f41c29c209caf1a4d96458761d71e42f2/media/tmmda_assets/TMMDA_screenshot_10.png -------------------------------------------------------------------------------- /media/tmmda_assets/TMMDA_screenshot_2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GT-Analytics/fuam-basic/cbfd190f41c29c209caf1a4d96458761d71e42f2/media/tmmda_assets/TMMDA_screenshot_2.png -------------------------------------------------------------------------------- /media/tmmda_assets/TMMDA_screenshot_3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GT-Analytics/fuam-basic/cbfd190f41c29c209caf1a4d96458761d71e42f2/media/tmmda_assets/TMMDA_screenshot_3.png -------------------------------------------------------------------------------- /media/tmmda_assets/TMMDA_screenshot_4.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GT-Analytics/fuam-basic/cbfd190f41c29c209caf1a4d96458761d71e42f2/media/tmmda_assets/TMMDA_screenshot_4.png -------------------------------------------------------------------------------- /media/tmmda_assets/TMMDA_screenshot_5.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GT-Analytics/fuam-basic/cbfd190f41c29c209caf1a4d96458761d71e42f2/media/tmmda_assets/TMMDA_screenshot_5.png -------------------------------------------------------------------------------- /media/tmmda_assets/TMMDA_screenshot_6.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GT-Analytics/fuam-basic/cbfd190f41c29c209caf1a4d96458761d71e42f2/media/tmmda_assets/TMMDA_screenshot_6.png -------------------------------------------------------------------------------- /media/tmmda_assets/TMMDA_screenshot_7.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GT-Analytics/fuam-basic/cbfd190f41c29c209caf1a4d96458761d71e42f2/media/tmmda_assets/TMMDA_screenshot_7.png -------------------------------------------------------------------------------- /media/tmmda_assets/TMMDA_screenshot_8.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GT-Analytics/fuam-basic/cbfd190f41c29c209caf1a4d96458761d71e42f2/media/tmmda_assets/TMMDA_screenshot_8.png -------------------------------------------------------------------------------- /media/tmmda_assets/TMMDA_screenshot_9.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GT-Analytics/fuam-basic/cbfd190f41c29c209caf1a4d96458761d71e42f2/media/tmmda_assets/TMMDA_screenshot_9.png -------------------------------------------------------------------------------- /media/wiki_deployment/FUAM_basic_deployment_process_3_1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GT-Analytics/fuam-basic/cbfd190f41c29c209caf1a4d96458761d71e42f2/media/wiki_deployment/FUAM_basic_deployment_process_3_1.png -------------------------------------------------------------------------------- /media/wiki_deployment/FUAM_basic_deployment_process_3_2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GT-Analytics/fuam-basic/cbfd190f41c29c209caf1a4d96458761d71e42f2/media/wiki_deployment/FUAM_basic_deployment_process_3_2.png -------------------------------------------------------------------------------- /media/wiki_deployment/FUAM_basic_deployment_process_4_1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GT-Analytics/fuam-basic/cbfd190f41c29c209caf1a4d96458761d71e42f2/media/wiki_deployment/FUAM_basic_deployment_process_4_1.png -------------------------------------------------------------------------------- /media/wiki_deployment/FUAM_basic_deployment_process_5_1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GT-Analytics/fuam-basic/cbfd190f41c29c209caf1a4d96458761d71e42f2/media/wiki_deployment/FUAM_basic_deployment_process_5_1.png -------------------------------------------------------------------------------- /media/wiki_deployment/FUAM_basic_deployment_process_6_1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GT-Analytics/fuam-basic/cbfd190f41c29c209caf1a4d96458761d71e42f2/media/wiki_deployment/FUAM_basic_deployment_process_6_1.png -------------------------------------------------------------------------------- /media/wiki_deployment/FUAM_basic_deployment_process_7_1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GT-Analytics/fuam-basic/cbfd190f41c29c209caf1a4d96458761d71e42f2/media/wiki_deployment/FUAM_basic_deployment_process_7_1.png -------------------------------------------------------------------------------- /media/wiki_deployment/FUAM_basic_deployment_process_cover.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GT-Analytics/fuam-basic/cbfd190f41c29c209caf1a4d96458761d71e42f2/media/wiki_deployment/FUAM_basic_deployment_process_cover.png -------------------------------------------------------------------------------- /media/wiki_deployment/FUAM_basic_upgrade_process_cover.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GT-Analytics/fuam-basic/cbfd190f41c29c209caf1a4d96458761d71e42f2/media/wiki_deployment/FUAM_basic_upgrade_process_cover.png -------------------------------------------------------------------------------- /media/wiki_deployment/icon_FUAM_workspace.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GT-Analytics/fuam-basic/cbfd190f41c29c209caf1a4d96458761d71e42f2/media/wiki_deployment/icon_FUAM_workspace.png -------------------------------------------------------------------------------- /start/Deploy_FUAM_Basic_Gen2.ipynb: -------------------------------------------------------------------------------- 1 | {"cells":[{"cell_type":"markdown","source":["#### Welcome to FUAM Basic\n","\n","This notebook deployes the latest FUAM Basic in the current workspace. It works for initial deployment and for the upgrade process of FUAM.\n","\n","**Please visit our wiki sites depends on your scenario:**\n","\n","[Visit wiki - How to deploy FUAM](https://github.com/GT-Analytics/fuam-basic/wiki/Lifecycle:-Initial-Deployment-via-Notebook)\n","\n","[Visit wiki - How to upgrade FUAM](https://github.com/GT-Analytics/fuam-basic/wiki/Lifecycle:-Upgrading-via-Notebook)\n","\n","\n","**What is happening in this notebook?**\n"," - The notebook checks the two cloud connections for FUAM (if initial deployment, connections will be created, otherwise check only)\n"," - It downloads the latest FUAM Basic deployment file from Github\n"," - It deploys/updates the Fabric items in your workspace\n"," - It creates/overwrites a static table in your lakehouse (capacity_regions)\n","\n","**Next steps**\n","- Run this notebook\n","\n","If you **deploy** FUAM in this workspace at the **first time**:\n","- Navigate to the cloud connections\n","- Search under cloud connection for **fabric-service-api admin** and for **pbi-service-api admin**\n","- Add the credentials of your service principal to these connections\n","\n","If you **update** your existing FUAM workspace:\n","- After the notebooks has been executed, you are **done**\n"],"metadata":{"microsoft":{"language":"python","language_group":"synapse_pyspark"},"nteract":{"transient":{"deleting":false}}},"id":"b4701d36-0f3b-4b4a-a82a-5a435efbe958"},{"cell_type":"markdown","source":["##### Attach Lakehouse dynamically"],"metadata":{"jp-MarkdownHeadingCollapsed":true,"microsoft":{"language":"python","language_group":"synapse_pyspark"},"nteract":{"transient":{"deleting":false}}},"id":"dcf5d940-fcbc-467d-969b-4007c8477609"},{"cell_type":"code","source":["try:\n"," notebookutils.lakehouse.create(name = \"FUAM_Config_Lakehouse\")\n","except Exception as ex:\n"," print('Lakehouse already exists')"],"outputs":[],"execution_count":null,"metadata":{"microsoft":{"language":"python","language_group":"synapse_pyspark"}},"id":"dfdabf9c-4704-4d3f-a14d-50104959d3cc"},{"cell_type":"code","source":["%%configure -f\n","\n","{ \n"," \"defaultLakehouse\": { \n"," \"name\": \"FUAM_Config_Lakehouse\"\n"," }\n"," }"],"outputs":[],"execution_count":null,"metadata":{"jupyter":{"outputs_hidden":false,"source_hidden":false},"microsoft":{"language":"python","language_group":"synapse_pyspark"},"nteract":{"transient":{"deleting":false}}},"id":"6a3d6cbd-ad6e-4fc4-846a-05282c941f34"},{"cell_type":"markdown","source":["##### Connection names"],"metadata":{"nteract":{"transient":{"deleting":false}},"microsoft":{"language":"python","language_group":"synapse_pyspark"},"jp-MarkdownHeadingCollapsed":true},"id":"cebd72e9-02b4-4731-bf14-56e3ae7c2a3d"},{"cell_type":"code","source":["pbi_connection_name = 'pbi-service-api admin'\n","fabric_connection_name = 'fabric-service-api admin'"],"outputs":[],"execution_count":null,"metadata":{"microsoft":{"language":"python","language_group":"synapse_pyspark"}},"id":"e56abde2-18ba-44cc-8375-34c6959e0698"},{"cell_type":"markdown","source":["##### Pre-Deployment logic"],"metadata":{"jp-MarkdownHeadingCollapsed":true,"nteract":{"transient":{"deleting":false}}},"id":"e9c2e6b6-aa02-41e5-b161-35cf1773e8a4"},{"cell_type":"code","source":["import json\n","import requests\n","import base64\n","import time\n","# Target workspaceId\n","workspace = spark.conf.get(\"trident.workspace.id\")\n","# Get Access Token\n","pbi_access_token = mssparkutils.credentials.getToken(\"https://analysis.windows.net/powerbi/api\")\n","header = {'Content-Type':'application/json','Authorization': f'Bearer {pbi_access_token}'}"],"outputs":[],"execution_count":null,"metadata":{"microsoft":{"language":"python","language_group":"synapse_pyspark"}},"id":"a813d3d3-e8da-4048-b453-0440e10b5260"},{"cell_type":"code","source":["def create_or_get_connection(name, baseUrl, audience):\n"," # Check if connection has already been created\n"," url = 'https://api.fabric.microsoft.com/v1/connections/'\n"," continuation_token = None\n"," id = None\n"," connection_exists = False\n","\n"," while True:\n"," if continuation_token:\n"," url_with_token = url + f\"&continuationToken={continuation_token}\"\n"," else:\n"," url_with_token = url\n","\n"," response = requests.get(url=url, headers=header).json()\n"," for row in response['value']:\n"," if row['displayName'] == name:\n"," id = row[\"id\"]\n"," print(\"Connection already exists. Id:\" + id)\n"," connection_exists = True\n"," return(id)\n"," continuationToken = response.get(\"continuationToken\")\n"," if not continuation_token:\n"," print(f\"I am done for {url}\")\n"," break\n","\n"," # In case there is no connection available yet. Create a new one automatically\n"," if connection_exists == False:\n"," conn_json = {\"connectivityType\": \"ShareableCloud\",\n"," \"displayName\": name,\n"," \"connectionDetails\": {\n"," \"type\": \"WebForPipeline\",\n"," \"creationMethod\": \"WebForPipeline.Contents\",\n"," \"parameters\": [{\n"," \"dataType\": \"Text\",\n"," \"name\": \"baseUrl\",\n"," \"value\": baseUrl\n"," },\n"," {\n"," \"dataType\": \"Text\",\n"," \"name\": \"audience\",\n"," \"value\": audience\n"," }\n"," ]\n"," },\n"," \"privacyLevel\": \"Organizational\",\n"," \"credentialDetails\": {\n"," \"singleSignOnType\": \"None\",\n"," \"connectionEncryption\": \"NotEncrypted\",\n"," \"skipTestConnection\": False,\n"," \"credentials\": {\"credentialType\": \"Anonymous\"}\n"," } \n"," }\n"," url = 'https://api.fabric.microsoft.com/v1/connections/'\n"," response = requests.post(url=url, headers=header, json = conn_json)\n"," print(response.json())\n"," conn_id = response.json()['id']\n"," print(\"Connection created: \" + conn_id + \" . Enter the service principal credentials\")\n"," return(conn_id)\n"],"outputs":[],"execution_count":null,"metadata":{"microsoft":{"language":"python","language_group":"synapse_pyspark"}},"id":"62c90a6f-bb98-4765-af72-6d1b9bceb2e3"},{"cell_type":"code","source":["conn_pbi_service_api_admin = create_or_get_connection(pbi_connection_name, \"https://api.powerbi.com/v1.0/myorg/admin\", \"https://analysis.windows.net/powerbi/api\" )\n","conn_fabric_service_api_admin = create_or_get_connection(fabric_connection_name, \"https://api.fabric.microsoft.com/v1/admin\", \"https://api.fabric.microsoft.com\" )"],"outputs":[],"execution_count":null,"metadata":{"microsoft":{"language":"python","language_group":"synapse_pyspark"}},"id":"57603a50-ff92-485d-8d5d-d1025b8289cd"},{"cell_type":"markdown","source":["##### Deployment logic"],"metadata":{"jp-MarkdownHeadingCollapsed":true,"microsoft":{"language":"python","language_group":"synapse_pyspark"},"nteract":{"transient":{"deleting":false}}},"id":"36b6bc03-7fa5-4dae-9bc8-0dd7b1441b4a"},{"cell_type":"code","source":["# Helper variables\n","fuam_lakehouse_datasets = ['FUAM_Basic_PBI_Overview_SM']"],"outputs":[],"execution_count":null,"metadata":{"jupyter":{"outputs_hidden":false,"source_hidden":false},"microsoft":{"language":"python","language_group":"synapse_pyspark"},"nteract":{"transient":{"deleting":false}}},"id":"eea473a1-7a57-4a22-a194-2c04fadef7a3"},{"cell_type":"code","source":["# Open deployment json file\n","deployment = {}\n","print(\"Downloading from Github to FUAM_Config_Lakehouse\")\n","url = 'https://raw.githubusercontent.com/GT-Analytics/fuam-basic/refs/heads/main/files/deployment_file.json'\n","github_download = requests.get(url)\n","folder_path = mssparkutils.fs.getMountPath('/default') + \"/Files/deployment/\"\n","mssparkutils.fs.mkdirs(f\"file://\" +folder_path)\n","\n","with open(folder_path + \"deployment_file.json\", \"w\") as f:\n"," f.write(json.dumps(github_download.json()))\n","\n","print(\"Read from FUAM_Config_Lakehouse\")\n","\n","with open(mssparkutils.fs.getMountPath('/default') + \"/Files/deployment/deployment_file.json\") as f:\n"," deployment = json.load(f)"],"outputs":[],"execution_count":null,"metadata":{"jupyter":{"outputs_hidden":false,"source_hidden":false},"microsoft":{"language":"python","language_group":"synapse_pyspark"},"nteract":{"transient":{"deleting":false}}},"id":"12761406-47f5-4f17-87ff-a7072a497dd6"},{"cell_type":"code","source":["# Prepare JSON for deployment\n","guids_to_replace = [{ \"old_id\" : deployment[\"old_workspace\"] , \"new_id\" : workspace}]\n","guids_to_replace.append({ \"old_id\" : deployment[\"connections\"][\"conn_pbi_service_api_admin_old\"] , \"new_id\" : conn_pbi_service_api_admin})\n","guids_to_replace.append({ \"old_id\" : deployment[\"connections\"][\"conn_fabric_service_api_admin_old\"] , \"new_id\" : conn_fabric_service_api_admin})"],"outputs":[],"execution_count":null,"metadata":{"collapsed":false,"jupyter":{"outputs_hidden":false,"source_hidden":false},"microsoft":{"language":"python","language_group":"synapse_pyspark"},"nteract":{"transient":{"deleting":false}}},"id":"a984f68b-e3f2-4bc5-88eb-3f0ae72c2126"},{"cell_type":"code","source":["# Get existing items\n","# (relevant for FUAM release update)\n","url = 'https://api.fabric.microsoft.com/v1/workspaces/'+ workspace +'/items/'\n","existing_items = requests.get(url=url, headers=header).json()[\"value\"]"],"outputs":[],"execution_count":null,"metadata":{"jupyter":{"outputs_hidden":false,"source_hidden":false},"microsoft":{"language":"python","language_group":"synapse_pyspark"},"nteract":{"transient":{"deleting":false}}},"id":"5e5e1f3c-2169-4620-91b0-9d1216d7baa0"},{"cell_type":"code","source":["# Function to get ids from existing items\n","# (relevant for FUAM release update)\n","def id_for_existing_items ( name , type):\n"," for it in existing_items:\n"," if name == it[\"displayName\"] and type == it[\"type\"]:\n"," return it[\"id\"]\n"," return \"New Item\""],"outputs":[],"execution_count":null,"metadata":{"jupyter":{"outputs_hidden":false,"source_hidden":false},"microsoft":{"language":"python","language_group":"synapse_pyspark"},"nteract":{"transient":{"deleting":false}}},"id":"9d2494b7-b9dc-4b3a-8671-527b618bf333"},{"cell_type":"code","source":["guids_to_replace"],"outputs":[],"execution_count":null,"metadata":{"jupyter":{"outputs_hidden":false,"source_hidden":false},"microsoft":{"language":"python","language_group":"synapse_pyspark"},"nteract":{"transient":{"deleting":false}}},"id":"829026fd-13a5-4684-b926-35878b0b5d3a"},{"cell_type":"code","source":["items_to_deploy = deployment[\"items\"]"],"outputs":[],"execution_count":null,"metadata":{"jupyter":{"outputs_hidden":false,"source_hidden":false},"microsoft":{"language":"python","language_group":"synapse_pyspark"},"nteract":{"transient":{"deleting":false}}},"id":"bc36800b-20a9-48d0-9203-f4a38742968a"},{"cell_type":"code","source":["# Function to check if existing items\n","# (relevant for FUAM release update)\n","def check_if_item_exists(old_id):\n"," for row in guids_to_replace:\n"," if old_id == row['old_id']:\n"," return True\n"," return False"],"outputs":[],"execution_count":null,"metadata":{"jupyter":{"outputs_hidden":false,"source_hidden":false},"microsoft":{"language":"python","language_group":"synapse_pyspark"},"nteract":{"transient":{"deleting":false}}},"id":"b627f710-46de-436e-951f-58d0335bc946"},{"cell_type":"code","source":["# Deploy items one by one to workspace\n","# if item new, then create it\n","# if exists already, then update it\n","for item in items_to_deploy:\n"," rename_item = {}\n"," rename_item[\"old_id\"] = item[\"org_id\"]\n","\n"," print('Deploy ' + item['displayName'] ) \n","\n"," if 'definition' in item.keys():\n"," b = item['definition']['parts'][0]['payload']\n"," decoded = base64.b64decode(b).decode('utf-8')\n","\n"," for repl in guids_to_replace:\n"," decoded = decoded.replace(repl[\"old_id\"], repl[\"new_id\"])\n"," encoded = base64.b64encode(decoded.encode('utf-8'))\n"," item['definition']['parts'][0]['payload'] = encoded\n","\n"," it = item\n"," header = {'Content-Type':'application/json','Authorization': f'Bearer {pbi_access_token}'}\n","\n"," existing_id = id_for_existing_items(item['displayName'], item['type'])\n"," if existing_id == \"New Item\":\n"," print( \"Create \")\n"," url = 'https://api.fabric.microsoft.com/v1/workspaces/'+ workspace + '/items/'\n"," response = requests.post(url=url, headers=header, json = item)\n"," else:\n"," print( \"Update \")\n"," url = 'https://api.fabric.microsoft.com/v1/workspaces/'+ workspace + '/items/' + existing_id + \"/updateDefinition\"\n"," response = requests.post(url=url, headers=header, json = item) \n","\n"," if response.status_code == 202:\n"," get_op = 'Running'\n"," while get_op != 'Succeeded' and get_op != 'Failed':\n"," time.sleep(1.5)\n"," \n"," header = {'Content-Type':'application/json','Authorization': f'Bearer {pbi_access_token}'}\n"," response2 = requests.get(url=response.headers[\"location\"], headers=header)\n"," get_op = response2.json()['status']\n"," print(get_op)\n","\n"," header = {'Content-Type':'application/json','Authorization': f'Bearer {pbi_access_token}'}\n","\n"," response3 = requests.get(url=response.headers[\"location\"]+ \"/result\", headers=header)\n"," response3 = response3.json()\n"," else:\n"," if existing_id == \"New Item\":\n"," response3 = response.json()\n"," if existing_id == \"New Item\":\n"," rename_item[\"new_id\"] = response3[\"id\"]\n"," else:\n"," rename_item[\"new_id\"] = existing_id\n"," guids_to_replace.append(rename_item)"],"outputs":[],"execution_count":null,"metadata":{"jupyter":{"outputs_hidden":false,"source_hidden":false},"microsoft":{"language":"python","language_group":"synapse_pyspark"},"nteract":{"transient":{"deleting":false}}},"id":"fba584a7-5e8f-49c5-8ef4-e865a988236e"},{"cell_type":"code","source":["# Get existing items after deployment\n","header = {'Content-Type':'application/json','Authorization': f'Bearer {pbi_access_token}'}\n","url = 'https://api.fabric.microsoft.com/v1/workspaces/'+ workspace +'/items/'\n","existing_items = requests.get(url=url, headers=header).json()[\"value\"]"],"outputs":[],"execution_count":null,"metadata":{"jupyter":{"outputs_hidden":false,"source_hidden":false},"microsoft":{"language":"python","language_group":"synapse_pyspark"},"nteract":{"transient":{"deleting":false}}},"id":"1b2e5e9b-a2ef-4755-abf7-396196a12c93"},{"cell_type":"code","source":["# Get SQL Endpoint properties for main Lakehouse\n","header = {'Content-Type':'application/json','Authorization': f'Bearer {pbi_access_token}'}\n","url = 'https://api.fabric.microsoft.com/v1/workspaces/'+ workspace +'/lakehouses/' + id_for_existing_items('FUAM_Lakehouse', 'Lakehouse')\n","response = requests.get(url=url, headers=header)\n","new_sqlEndPointProperties = response.json()['properties']['sqlEndpointProperties']"],"outputs":[],"execution_count":null,"metadata":{"jupyter":{"outputs_hidden":false,"source_hidden":false},"microsoft":{"language":"python","language_group":"synapse_pyspark"},"nteract":{"transient":{"deleting":false}}},"id":"c80d0c10-6db0-431b-9bbe-dbfb2bb47146"},{"cell_type":"code","source":["new_sqlEndPointProperties"],"outputs":[],"execution_count":null,"metadata":{"jupyter":{"outputs_hidden":false,"source_hidden":false},"microsoft":{"language":"python","language_group":"synapse_pyspark"},"nteract":{"transient":{"deleting":false}}},"id":"092b5f5d-2d3b-4f39-97e0-ad76d9b83b1b"},{"cell_type":"code","source":["# Set SQL Endpoint\n","old_sql_EndPointProperties = deployment['sqlEndPointProperties']\n","old_sql_EndPointProperties"],"outputs":[],"execution_count":null,"metadata":{"jupyter":{"outputs_hidden":false,"source_hidden":false},"microsoft":{"language":"python","language_group":"synapse_pyspark"},"nteract":{"transient":{"deleting":false}}},"id":"295c7ba6-40ee-4c75-9664-05a69669fcec"},{"cell_type":"code","source":["update_datasource_json = {}\n","updateDetails = []\n","single_updateDetails = {}\n","single_updateDetails['datasourceSelector'] = {}\n","single_updateDetails['datasourceSelector']['datasourceType'] = \"Sql\"\n","single_updateDetails['datasourceSelector'][\"connectionDetails\"] = {}\n","single_updateDetails['datasourceSelector'][\"connectionDetails\"][\"server\"] = old_sql_EndPointProperties['connectionString']\n","single_updateDetails['datasourceSelector'][\"connectionDetails\"][\"database\"] = old_sql_EndPointProperties['id']\n","\n","single_updateDetails['connectionDetails'] = {}\n","single_updateDetails['connectionDetails'][\"server\"] = new_sqlEndPointProperties['connectionString']\n","single_updateDetails['connectionDetails'][\"database\"] = new_sqlEndPointProperties['id']\n","\n","updateDetails.append(single_updateDetails)\n","update_datasource_json['updateDetails'] = updateDetails"],"outputs":[],"execution_count":null,"metadata":{"jupyter":{"outputs_hidden":false,"source_hidden":false},"microsoft":{"language":"python","language_group":"synapse_pyspark"},"nteract":{"transient":{"deleting":false}}},"id":"68d8c43a-6e75-45ed-a423-1b0a4b443481"},{"cell_type":"code","source":["update_datasource_json"],"outputs":[],"execution_count":null,"metadata":{"jupyter":{"outputs_hidden":false,"source_hidden":false},"microsoft":{"language":"python","language_group":"synapse_pyspark"},"nteract":{"transient":{"deleting":false}}},"id":"5f5cfafd-242b-4a12-a4c6-906c495d6d0f"},{"cell_type":"code","source":["# Update connection between semantic model and lakehouse\n","for sm in fuam_lakehouse_datasets:\n"," print(sm)\n","\n"," max_tries = 0\n"," status_code = 0\n"," while (status_code != 200) & (max_tries < 3):\n"," \n"," header = {'Content-Type':'application/json','Authorization': f'Bearer {pbi_access_token}'}\n"," url = 'https://api.powerbi.com/v1.0/myorg/datasets/'+ id_for_existing_items(sm, 'SemanticModel') + '/Default.UpdateDatasources' \n"," response = requests.post(url=url, headers=header, json = update_datasource_json)\n"," print(response.status_code)\n"," \n"," print(f\"Status code for semantic model {sm}:\" + str(response.status_code))\n"," max_tries = max_tries + 1 \n"," status_code = response.status_code\n"],"outputs":[],"execution_count":null,"metadata":{"jupyter":{"outputs_hidden":false,"source_hidden":false},"microsoft":{"language":"python","language_group":"synapse_pyspark"},"nteract":{"transient":{"deleting":false}}},"id":"4ce3483c-a796-4f40-849a-903bcb3c208e"},{"cell_type":"code","source":["# Import table definitions and create tables in FUAM_Lakehouse\n","existing_tables = [table['name'] for table in notebookutils.lakehouse.listTables(\"FUAM_Lakehouse\")]\n","for table_definition in deployment[\"table_definitions\"]:\n"," if not(table_definition['table'] in existing_tables):\n"," print(\"Create table \" + table_definition['table'])\n"," spark.sql(table_definition['create_sql'])"],"outputs":[],"execution_count":null,"metadata":{"microsoft":{"language":"python","language_group":"synapse_pyspark"}},"id":"895a363f-9983-4148-a412-858ed0724517"},{"cell_type":"code","source":["# Refresh datasets\n","for sm in fuam_lakehouse_datasets:\n"," print(sm)\n"," refresh_json = {}\n"," refresh_json[\"notifyOption\"] = \"NoNotification\"\n"," refresh_json[\"retryCount\"] = \"3\"\n"," header = {'Content-Type':'application/json','Authorization': f'Bearer {pbi_access_token}'}\n"," url = 'https://api.powerbi.com/v1.0/myorg/datasets/'+ id_for_existing_items(sm, 'SemanticModel') + '/refreshes' \n"," response = requests.post(url=url, headers=header, json = refresh_json)\n"," print(response.status_code)"],"outputs":[],"execution_count":null,"metadata":{"microsoft":{"language":"python","language_group":"synapse_pyspark"}},"id":"bc9cfada-3673-4112-8f83-957d310e4f37"},{"cell_type":"markdown","source":["##### Post-Deployment logic"],"metadata":{"jp-MarkdownHeadingCollapsed":true,"microsoft":{"language":"python","language_group":"synapse_pyspark"},"nteract":{"transient":{"deleting":false}}},"id":"3d62f436-cdaa-4314-82ea-045f6f44a9e7"},{"cell_type":"code","source":["%%configure -f\n","\n","{ \n"," \"defaultLakehouse\": { \n"," \"name\": \"FUAM_Lakehouse\"\n"," }\n","}"],"outputs":[],"execution_count":null,"metadata":{"microsoft":{"language":"python","language_group":"synapse_pyspark"}},"id":"50bf7b03-dea6-4910-b960-5f7acfcc442e"},{"cell_type":"code","source":["from pyspark.sql.functions import explode, sequence\n","from pyspark.sql.types import StructType,StructField, StringType, IntegerType\n","import pandas as pd"],"outputs":[],"execution_count":null,"metadata":{"microsoft":{"language":"python","language_group":"synapse_pyspark"}},"id":"385f270a-d2ad-4b42-8621-825b852e32de"},{"cell_type":"code","source":["# Define the data as a dictionary\n","capacity_regions = pd.DataFrame(\n"," [\n"," (\"Asia Pacific\", \"Australia East\", 31.2532, 146.9211, \"New South Wales\"),\n"," (\"Asia Pacific\", \"Australia Southeast\", 36.9848, 143.3906, \"Victoria\"),\n"," (\"Asia Pacific\", \"Central India\", 18.5204, 73.8567, \"Pune\"),\n"," (\"Asia Pacific\", \"East Asia\", 22.3193, 114.1694, \"Hong Kong\"),\n"," (\"Asia Pacific\", \"Japan East\", 35.6764, 139.65, \"Tokyo\"),\n"," (\"Asia Pacific\", \"Korea Central\", 37.5519, 126.9918, \"Seoul\"),\n"," (\"Asia Pacific\", \"Southeast Asia\", 1.3521, 103.8198, \"Singapore\"),\n"," (\"Asia Pacific\", \"South India\", 13.0827, 80.2707, \"Chennai\"),\n"," (\"Europe\", \"North Europe\", 53.7798, 7.3055, \"Ireland\"),\n"," (\"Europe\", \"West Europe\", 52.1326, 5.2913, \"Netherlands\"),\n"," (\"Europe\", \"France Central\", 48.8566, 2.3522, \"Paris\"),\n"," (\"Europe\", \"Germany West Central\", 50.1109, 8.6821, \"Frankfurt am Main\"),\n"," (\"Europe\", \"Norway East\", 59.9139, 10.7522, \"Oslo\"),\n"," (\"Europe\", \"Sweden Central\", 60.6749, 17.1413, \"Gävle\"),\n"," (\"Europe\", \"Switzerland North\", 47.3769, 8.5417, \"Zürich\"),\n"," (\"Europe\", \"Switzerland West\", 46.2044, 6.1432, \"Geneva\"),\n"," (\"Europe\", \"UK South\", 51.5072, -0.1276, \"London\"),\n"," (\"Europe\", \"UK West\", 51.4837, -3.1681, \"Cardiff\"),\n"," (\"Americas\", \"Brazil South\", -23.5558, -46.6396, \"São Paulo State\"),\n"," (\"Americas\", \"Canada Central\", 43.6532, -79.3832, \"Toronto\"),\n"," (\"Americas\", \"Canada East\", 46.8131, -71.2075, \"Quebec City\"),\n"," (\"Americas\", \"East US\", 37.4316, -78.6569, \"Virginia\"),\n"," (\"Americas\", \"East US 2\", 37.4316, -78.6569, \"Virginia\"),\n"," (\"Americas\", \"North Central US\", 40.6331, -89.3985, \"Illinois\"),\n"," (\"Americas\", \"South Central US\", 31.9686, -99.9018, \"Texas\"),\n"," (\"Americas\", \"West US\", 36.7783, -119.4179, \"California\"),\n"," (\"Americas\", \"West US 2\", 47.7511, -120.7401, \"Washington\"),\n"," (\"Americas\", \"West US 3\", 34.0489, -111.0937, \"Arizona\"),\n"," (\"Middle East and Africa\", \"South Africa North\", -26.2056, 28.0337, \"Johannesburg\"),\n"," (\"Middle East and Africa\", \"UAE North\", 25.2048, 55.2708, \"Dubai\")\n","],\n"," columns=[\n"," \"Continent\",\n"," \"FabricRegion\",\n"," \"Latitude\",\n"," \"Longitude\",\t\n"," \"Location\"\n"," ]\n",")\n","\n","# Create a DataFrame\n","capacity_regions_df = pd.DataFrame(capacity_regions)\n","\n","# Write Capacity regions to Lakehouse table\n","fc_convert_dict = {'Continent': str, 'FabricRegion': str, 'Latitude': str, 'Longitude': str, 'Location': str}\n","rules_catalog_df = capacity_regions_df.astype(fc_convert_dict)\n","fc_spark_df = spark.createDataFrame(capacity_regions_df)\n","\n","fc_spark_df.write.mode(\"overwrite\").option(\"mergeSchema\", \"true\").format(\"delta\").saveAsTable(\"capacity_regions\")\n","print(\"Congratulations. FUAM Basic has been successfully deployed.\")\n","print(\"Please navigate back to the top of the page to continue with the next step of the configuration.\")"],"outputs":[],"execution_count":null,"metadata":{"microsoft":{"language":"python","language_group":"synapse_pyspark"}},"id":"f5cb762b-ff9d-4666-8b93-2b5882e896d0"}],"metadata":{"kernel_info":{"name":"synapse_pyspark"},"kernelspec":{"name":"synapse_pyspark","language":"Python","display_name":"Synapse PySpark"},"language_info":{"name":"python"},"microsoft":{"language":"python","language_group":"synapse_pyspark","ms_spell_check":{"ms_spell_check_language":"en"}},"nteract":{"version":"nteract-front-end@1.0.0"},"widgets":{},"synapse_widget":{"state":{},"version":"0.1"},"spark_compute":{"compute_id":"/trident/default","session_options":{"conf":{"spark.synapse.nbs.session.timeout":"1200000"}}},"dependencies":{"lakehouse":{}}},"nbformat":4,"nbformat_minor":5} --------------------------------------------------------------------------------