├── .gitignore ├── CODE_OF_CONDUCT.md ├── LICENSE ├── README.md ├── SECURITY.md ├── SUPPORT.md ├── assets ├── azure-ai-foundry-deploy-gpt4o-confirm.png ├── azure-ai-foundry-deploy-gpt4o.png ├── azure-ai-foundry-deploy.png ├── azure-ai-foundry-deployed-gpt4o.png ├── azure-blob-storage-copy-qas.png ├── azure-blob-storage-copy-topics.png ├── azure-blob-storage-sas.png ├── azure-blob-storage-topics-and-qas.png ├── azure-blob-storage-topics.png ├── azure-change-access-level-anonymous.png ├── azure-change-access-level.png ├── azure-create-a-resource.png ├── azure-create-azure-openai-details.png ├── azure-create-azure-openai.png ├── azure-create-storage-account-details.png ├── azure-create-storage-account.png ├── azure-enable-anonymous-access.png ├── azure-enable-resource-sharing.png ├── azure-enable-static-website.png ├── azure-explore-ai-foundry.png ├── banner.jpg ├── banner.png ├── data-pipeline-clone-source.png ├── data-pipeline-clone.png ├── data-pipeline-completed.png ├── data-pipeline-connect-to-destination1.png ├── data-pipeline-connect-to-destination2.png ├── data-pipeline-connection-azure-blob.png ├── data-pipeline-connection.png ├── data-pipeline-copy-confirm.png ├── data-pipeline-destination-qrcodes.png ├── data-pipeline-running.png ├── data-pipeline-source-connection-lakehouse-topics.png ├── data-pipeline-source-qrcodes.png ├── fabric-create-lakehouse.png ├── fabric-create-pipeline.png ├── fabric-create-workspace.png ├── fabric-dumpcsv-definition.png ├── fabric-generated-qas.png ├── fabric-module-metadata-mds.png ├── fabric-module-metadata1.png ├── fabric-module-metadata2.png ├── fabric-name-lakehouse.png ├── fabric-new-data-pipeline.png ├── fabric-new-notebook.png ├── fabric-notebook-imports.png ├── fabric-notebook-pip-install.png ├── fabric-openai-test.png ├── fabric-qna-curated.png ├── fabric-qna-uncurated.png ├── fabric-read-topics.png ├── fabric-saveas-notebook.png ├── fabric-topics-generated.png ├── fabric-upload-csv.png ├── fabric-upload-full-topic-and-qas.png ├── flashcards-pdf-preview.png ├── flashcards-site.png ├── github-actions-config.png ├── github-actions-confirm.png ├── github-actions-run.png ├── github-actions-running.png ├── github-actions-select.png ├── github-pages-live-inspect.png ├── github-pages-live.png ├── lakehouse-explorer-includes.png ├── lakehouse-explorer-markdown.png ├── lakehouse-explorer-qrcode.png ├── lakehouse-explorer-qrcodes.png ├── microsoft-learn-github-includes.png ├── microsoft-learn-github-kusto.png ├── microsoft-learn.png ├── moaw.png ├── printable-flashcards-generator-print.png ├── printable-flashcards-generator.png ├── workshop-website-repo-fork-actions.png ├── workshop-website-repo-fork-commit.png ├── workshop-website-repo-fork-edit.png ├── workshop-website-repo-fork.png └── workshop-website-repo.png └── workshop.md /.gitignore: -------------------------------------------------------------------------------- 1 | ## Ignore Visual Studio temporary files, build results, and 2 | ## files generated by popular Visual Studio add-ons. 3 | ## 4 | ## Get latest from https://github.com/github/gitignore/blob/main/VisualStudio.gitignore 5 | 6 | # User-specific files 7 | *.rsuser 8 | *.suo 9 | *.user 10 | *.userosscache 11 | *.sln.docstates 12 | 13 | # User-specific files (MonoDevelop/Xamarin Studio) 14 | *.userprefs 15 | 16 | # Mono auto generated files 17 | mono_crash.* 18 | 19 | # Build results 20 | [Dd]ebug/ 21 | [Dd]ebugPublic/ 22 | [Rr]elease/ 23 | [Rr]eleases/ 24 | x64/ 25 | x86/ 26 | [Ww][Ii][Nn]32/ 27 | [Aa][Rr][Mm]/ 28 | [Aa][Rr][Mm]64/ 29 | bld/ 30 | [Bb]in/ 31 | [Oo]bj/ 32 | [Ll]og/ 33 | [Ll]ogs/ 34 | 35 | # Visual Studio 2015/2017 cache/options directory 36 | .vs/ 37 | # Uncomment if you have tasks that create the project's static files in wwwroot 38 | #wwwroot/ 39 | 40 | # Visual Studio 2017 auto generated files 41 | Generated\ Files/ 42 | 43 | # MSTest test Results 44 | [Tt]est[Rr]esult*/ 45 | [Bb]uild[Ll]og.* 46 | 47 | # NUnit 48 | *.VisualState.xml 49 | TestResult.xml 50 | nunit-*.xml 51 | 52 | # Build Results of an ATL Project 53 | [Dd]ebugPS/ 54 | [Rr]eleasePS/ 55 | dlldata.c 56 | 57 | # Benchmark Results 58 | BenchmarkDotNet.Artifacts/ 59 | 60 | # .NET Core 61 | project.lock.json 62 | project.fragment.lock.json 63 | artifacts/ 64 | 65 | # ASP.NET Scaffolding 66 | ScaffoldingReadMe.txt 67 | 68 | # StyleCop 69 | StyleCopReport.xml 70 | 71 | # Files built by Visual Studio 72 | *_i.c 73 | *_p.c 74 | *_h.h 75 | *.ilk 76 | *.meta 77 | *.obj 78 | *.iobj 79 | *.pch 80 | *.pdb 81 | *.ipdb 82 | *.pgc 83 | *.pgd 84 | *.rsp 85 | *.sbr 86 | *.tlb 87 | *.tli 88 | *.tlh 89 | *.tmp 90 | *.tmp_proj 91 | *_wpftmp.csproj 92 | *.log 93 | *.tlog 94 | *.vspscc 95 | *.vssscc 96 | .builds 97 | *.pidb 98 | *.svclog 99 | *.scc 100 | 101 | # Chutzpah Test files 102 | _Chutzpah* 103 | 104 | # Visual C++ cache files 105 | ipch/ 106 | *.aps 107 | *.ncb 108 | *.opendb 109 | *.opensdf 110 | *.sdf 111 | *.cachefile 112 | *.VC.db 113 | *.VC.VC.opendb 114 | 115 | # Visual Studio profiler 116 | *.psess 117 | *.vsp 118 | *.vspx 119 | *.sap 120 | 121 | # Visual Studio Trace Files 122 | *.e2e 123 | 124 | # TFS 2012 Local Workspace 125 | $tf/ 126 | 127 | # Guidance Automation Toolkit 128 | *.gpState 129 | 130 | # ReSharper is a .NET coding add-in 131 | _ReSharper*/ 132 | *.[Rr]e[Ss]harper 133 | *.DotSettings.user 134 | 135 | # TeamCity is a build add-in 136 | _TeamCity* 137 | 138 | # DotCover is a Code Coverage Tool 139 | *.dotCover 140 | 141 | # AxoCover is a Code Coverage Tool 142 | .axoCover/* 143 | !.axoCover/settings.json 144 | 145 | # Coverlet is a free, cross platform Code Coverage Tool 146 | coverage*.json 147 | coverage*.xml 148 | coverage*.info 149 | 150 | # Visual Studio code coverage results 151 | *.coverage 152 | *.coveragexml 153 | 154 | # NCrunch 155 | _NCrunch_* 156 | .*crunch*.local.xml 157 | nCrunchTemp_* 158 | 159 | # MightyMoose 160 | *.mm.* 161 | AutoTest.Net/ 162 | 163 | # Web workbench (sass) 164 | .sass-cache/ 165 | 166 | # Installshield output folder 167 | [Ee]xpress/ 168 | 169 | # DocProject is a documentation generator add-in 170 | DocProject/buildhelp/ 171 | DocProject/Help/*.HxT 172 | DocProject/Help/*.HxC 173 | DocProject/Help/*.hhc 174 | DocProject/Help/*.hhk 175 | DocProject/Help/*.hhp 176 | DocProject/Help/Html2 177 | DocProject/Help/html 178 | 179 | # Click-Once directory 180 | publish/ 181 | 182 | # Publish Web Output 183 | *.[Pp]ublish.xml 184 | *.azurePubxml 185 | # Note: Comment the next line if you want to checkin your web deploy settings, 186 | # but database connection strings (with potential passwords) will be unencrypted 187 | *.pubxml 188 | *.publishproj 189 | 190 | # Microsoft Azure Web App publish settings. Comment the next line if you want to 191 | # checkin your Azure Web App publish settings, but sensitive information contained 192 | # in these scripts will be unencrypted 193 | PublishScripts/ 194 | 195 | # NuGet Packages 196 | *.nupkg 197 | # NuGet Symbol Packages 198 | *.snupkg 199 | # The packages folder can be ignored because of Package Restore 200 | **/[Pp]ackages/* 201 | # except build/, which is used as an MSBuild target. 202 | !**/[Pp]ackages/build/ 203 | # Uncomment if necessary however generally it will be regenerated when needed 204 | #!**/[Pp]ackages/repositories.config 205 | # NuGet v3's project.json files produces more ignorable files 206 | *.nuget.props 207 | *.nuget.targets 208 | 209 | # Microsoft Azure Build Output 210 | csx/ 211 | *.build.csdef 212 | 213 | # Microsoft Azure Emulator 214 | ecf/ 215 | rcf/ 216 | 217 | # Windows Store app package directories and files 218 | AppPackages/ 219 | BundleArtifacts/ 220 | Package.StoreAssociation.xml 221 | _pkginfo.txt 222 | *.appx 223 | *.appxbundle 224 | *.appxupload 225 | 226 | # Visual Studio cache files 227 | # files ending in .cache can be ignored 228 | *.[Cc]ache 229 | # but keep track of directories ending in .cache 230 | !?*.[Cc]ache/ 231 | 232 | # Others 233 | ClientBin/ 234 | ~$* 235 | *~ 236 | *.dbmdl 237 | *.dbproj.schemaview 238 | *.jfm 239 | *.pfx 240 | *.publishsettings 241 | orleans.codegen.cs 242 | 243 | # Including strong name files can present a security risk 244 | # (https://github.com/github/gitignore/pull/2483#issue-259490424) 245 | #*.snk 246 | 247 | # Since there are multiple workflows, uncomment next line to ignore bower_components 248 | # (https://github.com/github/gitignore/pull/1529#issuecomment-104372622) 249 | #bower_components/ 250 | 251 | # RIA/Silverlight projects 252 | Generated_Code/ 253 | 254 | # Backup & report files from converting an old project file 255 | # to a newer Visual Studio version. Backup files are not needed, 256 | # because we have git ;-) 257 | _UpgradeReport_Files/ 258 | Backup*/ 259 | UpgradeLog*.XML 260 | UpgradeLog*.htm 261 | ServiceFabricBackup/ 262 | *.rptproj.bak 263 | 264 | # SQL Server files 265 | *.mdf 266 | *.ldf 267 | *.ndf 268 | 269 | # Business Intelligence projects 270 | *.rdl.data 271 | *.bim.layout 272 | *.bim_*.settings 273 | *.rptproj.rsuser 274 | *- [Bb]ackup.rdl 275 | *- [Bb]ackup ([0-9]).rdl 276 | *- [Bb]ackup ([0-9][0-9]).rdl 277 | 278 | # Microsoft Fakes 279 | FakesAssemblies/ 280 | 281 | # GhostDoc plugin setting file 282 | *.GhostDoc.xml 283 | 284 | # Node.js Tools for Visual Studio 285 | .ntvs_analysis.dat 286 | node_modules/ 287 | 288 | # Visual Studio 6 build log 289 | *.plg 290 | 291 | # Visual Studio 6 workspace options file 292 | *.opt 293 | 294 | # Visual Studio 6 auto-generated workspace file (contains which files were open etc.) 295 | *.vbw 296 | 297 | # Visual Studio 6 auto-generated project file (contains which files were open etc.) 298 | *.vbp 299 | 300 | # Visual Studio 6 workspace and project file (working project files containing files to include in project) 301 | *.dsw 302 | *.dsp 303 | 304 | # Visual Studio 6 technical files 305 | *.ncb 306 | *.aps 307 | 308 | # Visual Studio LightSwitch build output 309 | **/*.HTMLClient/GeneratedArtifacts 310 | **/*.DesktopClient/GeneratedArtifacts 311 | **/*.DesktopClient/ModelManifest.xml 312 | **/*.Server/GeneratedArtifacts 313 | **/*.Server/ModelManifest.xml 314 | _Pvt_Extensions 315 | 316 | # Paket dependency manager 317 | .paket/paket.exe 318 | paket-files/ 319 | 320 | # FAKE - F# Make 321 | .fake/ 322 | 323 | # CodeRush personal settings 324 | .cr/personal 325 | 326 | # Python Tools for Visual Studio (PTVS) 327 | __pycache__/ 328 | *.pyc 329 | 330 | # Cake - Uncomment if you are using it 331 | # tools/** 332 | # !tools/packages.config 333 | 334 | # Tabs Studio 335 | *.tss 336 | 337 | # Telerik's JustMock configuration file 338 | *.jmconfig 339 | 340 | # BizTalk build output 341 | *.btp.cs 342 | *.btm.cs 343 | *.odx.cs 344 | *.xsd.cs 345 | 346 | # OpenCover UI analysis results 347 | OpenCover/ 348 | 349 | # Azure Stream Analytics local run output 350 | ASALocalRun/ 351 | 352 | # MSBuild Binary and Structured Log 353 | *.binlog 354 | 355 | # NVidia Nsight GPU debugger configuration file 356 | *.nvuser 357 | 358 | # MFractors (Xamarin productivity tool) working folder 359 | .mfractor/ 360 | 361 | # Local History for Visual Studio 362 | .localhistory/ 363 | 364 | # Visual Studio History (VSHistory) files 365 | .vshistory/ 366 | 367 | # BeatPulse healthcheck temp database 368 | healthchecksdb 369 | 370 | # Backup folder for Package Reference Convert tool in Visual Studio 2017 371 | MigrationBackup/ 372 | 373 | # Ionide (cross platform F# VS Code tools) working folder 374 | .ionide/ 375 | 376 | # Fody - auto-generated XML schema 377 | FodyWeavers.xsd 378 | 379 | # VS Code files for those working on multiple tools 380 | .vscode/* 381 | !.vscode/settings.json 382 | !.vscode/tasks.json 383 | !.vscode/launch.json 384 | !.vscode/extensions.json 385 | *.code-workspace 386 | 387 | # Local History for Visual Studio Code 388 | .history/ 389 | 390 | # Windows Installer files from build outputs 391 | *.cab 392 | *.msi 393 | *.msix 394 | *.msm 395 | *.msp 396 | 397 | # JetBrains Rider 398 | *.sln.iml 399 | 400 | .DS_Store -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | # Microsoft Open Source Code of Conduct 2 | 3 | This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/). 4 | 5 | Resources: 6 | 7 | - [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/) 8 | - [Microsoft Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) 9 | - Contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with questions or concerns 10 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) Microsoft Corporation. 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Welcome 2 | 3 | In this workshop, you will learn how to use [Microsoft Fabric](https://www.microsoft.com/microsoft-fabric) and with [Azure OpenAI](https://azure.microsoft.com/products/ai-services/openai-service) to generate a set of study flashcards, so you can learn new concepts in a fun and playful way. 4 | 5 | You can follow the full workshop at [MOAW](https://moaw.dev/), by clicking the following link [Fabric Flashcards Workshop](https://aka.ms/FFWorkshop), or the image below: 6 | 7 | [![Screenshot of MOAW workshop](./assets/moaw.png)](https://aka.ms/FFWorkshop) 8 | 9 | Or browse directly to the workshop source code here [workshop.md](./workshop.md) 10 | 11 | ## Contributing 12 | 13 | This project welcomes contributions and suggestions. Most contributions require you to agree to a 14 | Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us 15 | the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com. 16 | 17 | When you submit a pull request, a CLA bot will automatically determine whether you need to provide 18 | a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions 19 | provided by the bot. You will only need to do this once across all repos using our CLA. 20 | 21 | This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/). 22 | For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or 23 | contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments. 24 | 25 | ## Trademarks 26 | 27 | This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft 28 | trademarks or logos is subject to and must follow 29 | [Microsoft's Trademark & Brand Guidelines](https://www.microsoft.com/en-us/legal/intellectualproperty/trademarks/usage/general). 30 | Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. 31 | Any use of third-party trademarks or logos are subject to those third-party's policies. 32 | -------------------------------------------------------------------------------- /SECURITY.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | ## Security 4 | 5 | Microsoft takes the security of our software products and services seriously, which includes all source code repositories managed through our GitHub organizations, which include [Microsoft](https://github.com/Microsoft), [Azure](https://github.com/Azure), [DotNet](https://github.com/dotnet), [AspNet](https://github.com/aspnet) and [Xamarin](https://github.com/xamarin). 6 | 7 | If you believe you have found a security vulnerability in any Microsoft-owned repository that meets [Microsoft's definition of a security vulnerability](https://aka.ms/security.md/definition), please report it to us as described below. 8 | 9 | ## Reporting Security Issues 10 | 11 | **Please do not report security vulnerabilities through public GitHub issues.** 12 | 13 | Instead, please report them to the Microsoft Security Response Center (MSRC) at [https://msrc.microsoft.com/create-report](https://aka.ms/security.md/msrc/create-report). 14 | 15 | If you prefer to submit without logging in, send email to [secure@microsoft.com](mailto:secure@microsoft.com). If possible, encrypt your message with our PGP key; please download it from the [Microsoft Security Response Center PGP Key page](https://aka.ms/security.md/msrc/pgp). 16 | 17 | You should receive a response within 24 hours. If for some reason you do not, please follow up via email to ensure we received your original message. Additional information can be found at [microsoft.com/msrc](https://www.microsoft.com/msrc). 18 | 19 | Please include the requested information listed below (as much as you can provide) to help us better understand the nature and scope of the possible issue: 20 | 21 | * Type of issue (e.g. buffer overflow, SQL injection, cross-site scripting, etc.) 22 | * Full paths of source file(s) related to the manifestation of the issue 23 | * The location of the affected source code (tag/branch/commit or direct URL) 24 | * Any special configuration required to reproduce the issue 25 | * Step-by-step instructions to reproduce the issue 26 | * Proof-of-concept or exploit code (if possible) 27 | * Impact of the issue, including how an attacker might exploit the issue 28 | 29 | This information will help us triage your report more quickly. 30 | 31 | If you are reporting for a bug bounty, more complete reports can contribute to a higher bounty award. Please visit our [Microsoft Bug Bounty Program](https://aka.ms/security.md/msrc/bounty) page for more details about our active programs. 32 | 33 | ## Preferred Languages 34 | 35 | We prefer all communications to be in English. 36 | 37 | ## Policy 38 | 39 | Microsoft follows the principle of [Coordinated Vulnerability Disclosure](https://aka.ms/security.md/cvd). 40 | 41 | 42 | -------------------------------------------------------------------------------- /SUPPORT.md: -------------------------------------------------------------------------------- 1 | # TODO: The maintainer of this repo has not yet edited this file 2 | 3 | **REPO OWNER**: Do you want Customer Service & Support (CSS) support for this product/project? 4 | 5 | - **No CSS support:** Fill out this template with information about how to file issues and get help. 6 | - **Yes CSS support:** Fill out an intake form at [aka.ms/onboardsupport](https://aka.ms/onboardsupport). CSS will work with/help you to determine next steps. 7 | - **Not sure?** Fill out an intake as though the answer were "Yes". CSS will help you decide. 8 | 9 | *Then remove this first heading from this SUPPORT.MD file before publishing your repo.* 10 | 11 | # Support 12 | 13 | ## How to file issues and get help 14 | 15 | This project uses GitHub Issues to track bugs and feature requests. Please search the existing 16 | issues before filing new issues to avoid duplicates. For new issues, file your bug or 17 | feature request as a new Issue. 18 | 19 | For help and questions about using this project, please **REPO MAINTAINER: INSERT INSTRUCTIONS HERE 20 | FOR HOW TO ENGAGE REPO OWNERS OR COMMUNITY FOR HELP. COULD BE A STACK OVERFLOW TAG OR OTHER 21 | CHANNEL. WHERE WILL YOU HELP PEOPLE?**. 22 | 23 | ## Microsoft Support Policy 24 | 25 | Support for this **PROJECT or PRODUCT** is limited to the resources listed above. 26 | -------------------------------------------------------------------------------- /assets/azure-ai-foundry-deploy-gpt4o-confirm.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/azure-ai-foundry-deploy-gpt4o-confirm.png -------------------------------------------------------------------------------- /assets/azure-ai-foundry-deploy-gpt4o.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/azure-ai-foundry-deploy-gpt4o.png -------------------------------------------------------------------------------- /assets/azure-ai-foundry-deploy.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/azure-ai-foundry-deploy.png -------------------------------------------------------------------------------- /assets/azure-ai-foundry-deployed-gpt4o.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/azure-ai-foundry-deployed-gpt4o.png -------------------------------------------------------------------------------- /assets/azure-blob-storage-copy-qas.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/azure-blob-storage-copy-qas.png -------------------------------------------------------------------------------- /assets/azure-blob-storage-copy-topics.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/azure-blob-storage-copy-topics.png -------------------------------------------------------------------------------- /assets/azure-blob-storage-sas.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/azure-blob-storage-sas.png -------------------------------------------------------------------------------- /assets/azure-blob-storage-topics-and-qas.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/azure-blob-storage-topics-and-qas.png -------------------------------------------------------------------------------- /assets/azure-blob-storage-topics.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/azure-blob-storage-topics.png -------------------------------------------------------------------------------- /assets/azure-change-access-level-anonymous.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/azure-change-access-level-anonymous.png -------------------------------------------------------------------------------- /assets/azure-change-access-level.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/azure-change-access-level.png -------------------------------------------------------------------------------- /assets/azure-create-a-resource.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/azure-create-a-resource.png -------------------------------------------------------------------------------- /assets/azure-create-azure-openai-details.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/azure-create-azure-openai-details.png -------------------------------------------------------------------------------- /assets/azure-create-azure-openai.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/azure-create-azure-openai.png -------------------------------------------------------------------------------- /assets/azure-create-storage-account-details.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/azure-create-storage-account-details.png -------------------------------------------------------------------------------- /assets/azure-create-storage-account.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/azure-create-storage-account.png -------------------------------------------------------------------------------- /assets/azure-enable-anonymous-access.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/azure-enable-anonymous-access.png -------------------------------------------------------------------------------- /assets/azure-enable-resource-sharing.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/azure-enable-resource-sharing.png -------------------------------------------------------------------------------- /assets/azure-enable-static-website.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/azure-enable-static-website.png -------------------------------------------------------------------------------- /assets/azure-explore-ai-foundry.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/azure-explore-ai-foundry.png -------------------------------------------------------------------------------- /assets/banner.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/banner.jpg -------------------------------------------------------------------------------- /assets/banner.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/banner.png -------------------------------------------------------------------------------- /assets/data-pipeline-clone-source.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/data-pipeline-clone-source.png -------------------------------------------------------------------------------- /assets/data-pipeline-clone.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/data-pipeline-clone.png -------------------------------------------------------------------------------- /assets/data-pipeline-completed.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/data-pipeline-completed.png -------------------------------------------------------------------------------- /assets/data-pipeline-connect-to-destination1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/data-pipeline-connect-to-destination1.png -------------------------------------------------------------------------------- /assets/data-pipeline-connect-to-destination2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/data-pipeline-connect-to-destination2.png -------------------------------------------------------------------------------- /assets/data-pipeline-connection-azure-blob.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/data-pipeline-connection-azure-blob.png -------------------------------------------------------------------------------- /assets/data-pipeline-connection.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/data-pipeline-connection.png -------------------------------------------------------------------------------- /assets/data-pipeline-copy-confirm.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/data-pipeline-copy-confirm.png -------------------------------------------------------------------------------- /assets/data-pipeline-destination-qrcodes.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/data-pipeline-destination-qrcodes.png -------------------------------------------------------------------------------- /assets/data-pipeline-running.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/data-pipeline-running.png -------------------------------------------------------------------------------- /assets/data-pipeline-source-connection-lakehouse-topics.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/data-pipeline-source-connection-lakehouse-topics.png -------------------------------------------------------------------------------- /assets/data-pipeline-source-qrcodes.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/data-pipeline-source-qrcodes.png -------------------------------------------------------------------------------- /assets/fabric-create-lakehouse.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/fabric-create-lakehouse.png -------------------------------------------------------------------------------- /assets/fabric-create-pipeline.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/fabric-create-pipeline.png -------------------------------------------------------------------------------- /assets/fabric-create-workspace.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/fabric-create-workspace.png -------------------------------------------------------------------------------- /assets/fabric-dumpcsv-definition.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/fabric-dumpcsv-definition.png -------------------------------------------------------------------------------- /assets/fabric-generated-qas.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/fabric-generated-qas.png -------------------------------------------------------------------------------- /assets/fabric-module-metadata-mds.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/fabric-module-metadata-mds.png -------------------------------------------------------------------------------- /assets/fabric-module-metadata1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/fabric-module-metadata1.png -------------------------------------------------------------------------------- /assets/fabric-module-metadata2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/fabric-module-metadata2.png -------------------------------------------------------------------------------- /assets/fabric-name-lakehouse.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/fabric-name-lakehouse.png -------------------------------------------------------------------------------- /assets/fabric-new-data-pipeline.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/fabric-new-data-pipeline.png -------------------------------------------------------------------------------- /assets/fabric-new-notebook.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/fabric-new-notebook.png -------------------------------------------------------------------------------- /assets/fabric-notebook-imports.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/fabric-notebook-imports.png -------------------------------------------------------------------------------- /assets/fabric-notebook-pip-install.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/fabric-notebook-pip-install.png -------------------------------------------------------------------------------- /assets/fabric-openai-test.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/fabric-openai-test.png -------------------------------------------------------------------------------- /assets/fabric-qna-curated.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/fabric-qna-curated.png -------------------------------------------------------------------------------- /assets/fabric-qna-uncurated.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/fabric-qna-uncurated.png -------------------------------------------------------------------------------- /assets/fabric-read-topics.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/fabric-read-topics.png -------------------------------------------------------------------------------- /assets/fabric-saveas-notebook.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/fabric-saveas-notebook.png -------------------------------------------------------------------------------- /assets/fabric-topics-generated.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/fabric-topics-generated.png -------------------------------------------------------------------------------- /assets/fabric-upload-csv.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/fabric-upload-csv.png -------------------------------------------------------------------------------- /assets/fabric-upload-full-topic-and-qas.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/fabric-upload-full-topic-and-qas.png -------------------------------------------------------------------------------- /assets/flashcards-pdf-preview.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/flashcards-pdf-preview.png -------------------------------------------------------------------------------- /assets/flashcards-site.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/flashcards-site.png -------------------------------------------------------------------------------- /assets/github-actions-config.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/github-actions-config.png -------------------------------------------------------------------------------- /assets/github-actions-confirm.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/github-actions-confirm.png -------------------------------------------------------------------------------- /assets/github-actions-run.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/github-actions-run.png -------------------------------------------------------------------------------- /assets/github-actions-running.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/github-actions-running.png -------------------------------------------------------------------------------- /assets/github-actions-select.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/github-actions-select.png -------------------------------------------------------------------------------- /assets/github-pages-live-inspect.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/github-pages-live-inspect.png -------------------------------------------------------------------------------- /assets/github-pages-live.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/github-pages-live.png -------------------------------------------------------------------------------- /assets/lakehouse-explorer-includes.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/lakehouse-explorer-includes.png -------------------------------------------------------------------------------- /assets/lakehouse-explorer-markdown.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/lakehouse-explorer-markdown.png -------------------------------------------------------------------------------- /assets/lakehouse-explorer-qrcode.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/lakehouse-explorer-qrcode.png -------------------------------------------------------------------------------- /assets/lakehouse-explorer-qrcodes.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/lakehouse-explorer-qrcodes.png -------------------------------------------------------------------------------- /assets/microsoft-learn-github-includes.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/microsoft-learn-github-includes.png -------------------------------------------------------------------------------- /assets/microsoft-learn-github-kusto.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/microsoft-learn-github-kusto.png -------------------------------------------------------------------------------- /assets/microsoft-learn.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/microsoft-learn.png -------------------------------------------------------------------------------- /assets/moaw.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/moaw.png -------------------------------------------------------------------------------- /assets/printable-flashcards-generator-print.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/printable-flashcards-generator-print.png -------------------------------------------------------------------------------- /assets/printable-flashcards-generator.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/printable-flashcards-generator.png -------------------------------------------------------------------------------- /assets/workshop-website-repo-fork-actions.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/workshop-website-repo-fork-actions.png -------------------------------------------------------------------------------- /assets/workshop-website-repo-fork-commit.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/workshop-website-repo-fork-commit.png -------------------------------------------------------------------------------- /assets/workshop-website-repo-fork-edit.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/workshop-website-repo-fork-edit.png -------------------------------------------------------------------------------- /assets/workshop-website-repo-fork.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/workshop-website-repo-fork.png -------------------------------------------------------------------------------- /assets/workshop-website-repo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/microsoft/flashcards-workshop/c61e1942ce56b4df1e5db585a50eb26149b08ae1/assets/workshop-website-repo.png -------------------------------------------------------------------------------- /workshop.md: -------------------------------------------------------------------------------- 1 | --- 2 | published: true 3 | type: workshop 4 | title: Create self-study Flashcards using RAG (Retrieval Augmented Generation) with Microsoft Fabric and Azure AI Foundry 5 | short_title: Fabric Flashcards Workshop 6 | description: Learn how to use Microsoft Fabric and Azure AI Foundry to generate a set of study flashcards. Connect Fabric to Azure Blob Storage to store created flashcards and present in a static website hosted on GitHub Pages. 7 | level: beginner 8 | authors: 9 | - Alvaro Videla 10 | - Paul DeCarlo 11 | - Jasmine Greenaway 12 | contacts: 13 | - '@old_sound' 14 | duration_minutes: 60 15 | tags: openai, fabric, python, flashcards, Microsoft Fabric, Azure AI Foundry, Azure OpenAI, notebooks, data engineering, data pipelines 16 | banner_url: assets/banner.png # Optional. Should be a 1280x640px image 17 | #video_url: https://youtube.com/link # Optional. Link to a video of the workshop 18 | #audience: students # Optional. Audience of the workshop (students, pro devs, etc.) 19 | #wt_id: # Optional. Set advocacy tracking code for supported links 20 | #oc_id: # Optional. Set marketing tracking code for supported links 21 | #navigation_levels: 2 # Optional. Number of levels displayed in the side menu (default: 2) 22 | #sections_title: # Optional. Override titles for each section to be displayed in the side bar 23 | # - Section 1 title 24 | # - Section 2 title 25 | --- 26 | 27 | # Create Flashcards with Microsoft Fabric and Azure OpenAI 28 | 29 | In this workshop, you will learn how to use [Microsoft Fabric](https://www.microsoft.com/microsoft-fabric) with the [Azure OpenAI Service](https://azure.microsoft.com/products/ai-services/openai-service) in [Azure AI Foundry](https://learn.microsoft.com/azure/ai-foundry/azure-openai-in-ai-foundry) to generate a set of study flashcards. Learn how to use Microsoft Fabric with modern Retrieval-Augmented Generations (RAG) techniques and the latest foundation models in Azure AI Foundry, to develop materials to assist in your understanding of the overall Microsoft Fabric platform. 30 | 31 | In this example we will use [Microsoft Learn](https://learn.microsoft.com/training/) as the source material for the flashcards. 32 | 33 | We will fetch the Markdown files from the [Microsoft Learn GitHub repository](https://github.com/MicrosoftDocs/learn/) and import them into a Microsoft Fabric [Lakehouse](https://learn.microsoft.com/fabric/data-engineering/lakehouse-overview). 34 | 35 | Then we will use Azure OpenAI Service in Azure AI Foundry to generate the data that will be used as the basis for our set of study flashcards. 36 | 37 | Next, we will use [data pipelines](https://learn.microsoft.com/training/modules/use-data-factory-pipelines-fabric/) in Microsoft Fabric to copy the data to an external data store ([Azure Blob Storage](https://learn.microsoft.com/azure/storage/blobs/)) for public consumption. 38 | 39 | We will generate a static website to present your flashcards using the [GitHub Pages](https://docs.github.com/pages). 40 | 41 | ![Screenshot of static website hosted on GitHub Pages, displaying the contents of the Flashcards](assets/flashcards-site.png) 42 | 43 | ## Goals 44 | 45 | You'll learn how to: 46 | 47 | - Use Microsoft Fabric notebooks. 48 | - Load data from an external data source into a Microsoft Fabric Lakehouse. 49 | - Use Azure OpenAI Service in Azure AI Foundry to generate a set of study flashcards. 50 | - Run data pipelines to copy the data to an external data store (Azure Blob Storage) for public consumption. 51 | - Generate a static website using GitHub Pages to present and interact with your flashcards. 52 | 53 | ## Required Pre-requisites 54 | 55 | | | | 56 | |----------------------|------------------------------------------------------| 57 | | GitHub Account | [Create your free account on GitHub](https://github.com/signup) | 58 | | Access to Microsoft Fabric | [Accessing Microsoft Fabric for developers, startups and enterprises!](https://blog.fabric.microsoft.com/blog/accessing-microsoft-fabric-for-developers-startups-and-enterprises/) | 59 | | A workspace in Microsoft Fabric | [Create a Microsoft Fabric workspace](https://learn.microsoft.com/fabric/data-warehouse/tutorial-create-workspace) | 60 | | A Web browser | [Get Microsoft Edge](https://www.microsoft.com/edge) | 61 | 62 | 63 | ## Optional Pre-requisites 64 | 65 | >(Note: While it is possible to complete the workshop without access to these services, it is highly recommended use them in order to experience the full functionality of this content!) 66 | 67 | | | | 68 | |----------------------|------------------------------------------------------| 69 | | Azure account | [Get a free trial Azure account](https://azure.microsoft.com/free) | 70 | | Access to Azure OpenAI Service in Azure AI Foundry| [Azure Open AI in Azure AI Foundry](https://learn.microsoft.com/en-us/azure/ai-foundry/azure-openai-in-ai-foundry) | 71 | | Python knowledge | [Python for beginners](https://learn.microsoft.com/training/paths/beginner-python/) | 72 | 73 | --- 74 | 75 | # Environment Setup 76 | 77 | ## Account Provisioning 78 | 79 | **GitHub Account** - If you do not have an active GitHub account, proceed with steps to create a GitHub account at https://github.com/signup. This process should only take a few minutes to complete. 80 | 81 | 82 | **Access to Microsoft Fabric** - For those joining an in-person delivery of this workshop at FabCon, we will provide access to a Fabric tenant that will allow you access to an account that will have all of the necessary provisioning in place to complete the workshop steps. Proctors will provide you with details on how to access your account credentials. If you require access to a Fabric Free Trial Capacity, you can follow these [instructions](https://blog.fabric.microsoft.com/blog/accessing-microsoft-fabric-for-developers-startups-and-enterprises ) to satisfy this requirement. 83 | 84 | **Azure Account** - If you do not have an active Azure account setup, you should proceed to create an Azure free trial account. You can get started by visiting this [link](https://azure.microsoft.com/free). Azure Free trial accounts do not support deployment of Azure Open AI Service on AI Foundry, we will provide pre-generated inference results to accommodate situations for those who may be unable to access this service. Additionally, if for any reason, you are unable to create an Azure Account, you will still be able to complete a majority of the workshop content. For those who do not have access to an Azure account, you will follow different steps to reach the end goal, this will include skipping the deployment of the Azure Open AI Service in Azure AI Foundry and skipping the deployment of the Azure Storage Account. 85 | 86 | ## Deploy Azure Open AI Service in Azure AI Foundry 87 | 88 | **Azure Open AI Service in Azure AI Foundry** - You will use this service to generate the flashcards questions and answers. 89 | > If you are using an Azure account that does not have access to Azure Open AI Service in Azure Foundry (Azure Free Trial accounts are not supported!), it is suggested to read the instructions that follow for completeness, then skip to the next step which covers deployment of Azure Blob Storage. 90 | 91 | > The steps to deploy an Azure Open AI Service are described in additional detail in the [online documentation](https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/create-resource?pivots=web-portal) 92 | 93 | 1. Navigate to the [Azure Portal](https://portal.azure.com) and sign-in with your account. Select the `Create a resource` option as shown: 94 | 95 | ![Screenshot showing steps to create a resource in Microsoft Azure Portal](assets/azure-create-a-resource.png) 96 | 97 | 2. On the resulting page, search for the `Azure Open AI` service and click the `Create` button that is shown highlighted below : 98 | 99 | ![Screenshot showing steps to create Azure Open AI resource in Microsoft Azure Portal](assets/azure-create-azure-openai.png) 100 | 101 | 3. Provide the following details to prepare your service for deployment: 102 | 103 | **Subscription** - This should auto-populate based on your account settings 104 | 105 | **Resource group** - Select `Create new` and provide a name 106 | 107 | **Region** - Select an option from the drop-down, it is recommended to choose one in same geography as your Fabric tenant if possible 108 | 109 | **Pricing tier** - Standard S0 is recommended 110 | 111 | ![Screenshot showing steps to provide Azure details to prepare deployment of Open AI resource in Microsoft Azure Portal](assets/azure-create-azure-openai-details.png) 112 | 113 | When you are ready, select `Next` to proceed. On the resulting Network details page, ensure that you have selected option that states `All networks, including the internet, can access this resource`. You are now able to proceed to the `Review + submit` step to finalize deployment. When you are ready, select `Create` to begin the deployment. 114 | 115 | 4. You will need to wait for your deployment to complete, once this has completed, navigate to your newly deployed resource by selecting `Go to resource` on the resulting page. 116 | 117 |
118 | 119 | > If you have lost your way, you can always find your deployed service by navigating to the Resource Group that was selected for deployment in the previous step. Once successfully navigated, click the `Explore Azure AI Foundry portal` link on the overview page as shown below: 120 | 121 |
122 | 123 | ![Screenshot showing steps to navigate to Azure AI Foundry Portal from Microsoft Azure Portal](assets/azure-explore-ai-foundry.png) 124 | 125 | 5. Once you have completed navigation to [Azure AI Foundry](https://ai.azure.com/), select `Shared resources => Deployments`, the select `Deploy model` and select the `Deploy base model` option as shown: 126 | 127 | ![Screenshot showing steps to deploy base model from the Azure AI Foundry Portal](assets/azure-ai-foundry-deploy.png) 128 | 129 | 6. You will now be provided a number of options, we are specifically interested in deploying a model that supports the `Chat completion` task. At time of writing, `gpt-4o` is considered a good option. Select this model and select `Confirm` to proceed. 130 | 131 | ![Screenshot showing steps to deploy gpt4o model from the Azure AI Foundry Portal](assets/azure-ai-foundry-deploy-gpt4o.png) 132 | 133 | 7. On the resulting screen, leave options as they are with `Deployment type` set to `Global Standard` then select `Deploy` to begin the deployment. 134 | 135 | ![Screenshot showing steps to confirm and deploy a gpt4o model from the Azure AI Foundry Portal](assets/azure-ai-foundry-deploy-gpt4o-confirm.png) 136 | 137 | 8. You will then be navigated to your deployed instance of gpt-4o. In this screen you can find samples demonstrating how to call your Azure Open AI Service using various SDK. In addition, you can find your `Target URI` and `Key` that can be used to make web requests that call your AI model service from client code. These will be used in later instructions to call your deployed service from a Notebook in Microsoft Fabric. 138 | 139 | ![Screenshot showing your deployed gpt4o model in the Azure AI Foundry Portal](assets/azure-ai-foundry-deployed-gpt4o.png) 140 | 141 | ## Deploy Azure Storage Account 142 | 143 | **Azure Storage Account** - You will use this service to store the generated flashcard content, which will be used to render flashcard content in the accompanying static web app. 144 | > If you do not have access to an Azure account, it is still suggested to read the instructions that follow for completeness, and then skip to the next section. You will still be able to deploy a functional static web app, as this only requires an active GitHub account, but you will not be able to serve data to your static web app from Azure. 145 | 146 | 1. Navigate to the [Azure Portal](https://portal.azure.com) and sign-in with your account. Select the `Create a resource` option as shown: 147 | 148 | ![Screenshot showing steps to create a resource in Microsoft Azure Portal](assets/azure-create-a-resource.png) 149 | 150 | 151 | 2. On the resulting page, search for for the `Storage account` service and click the `Create` button highlighted below: 152 | 153 | ![Screenshot showing steps to create Storage Account resource in Microsoft Azure Portal](assets/azure-create-storage-account.png) 154 | 155 | 3. Provide the following details to prepare your service for deployment: 156 | 157 | **Subscription** - This should auto-populate based on your account settings 158 | 159 | **Resource group** - Select `Create new` and provide a name or select an existing Resource group 160 | 161 | **Storage account name** - The name must be unique across all existing storage account names in Azure. It must be 3 to 24 characters long, and can contain only lowercase letters and numbers. 162 | 163 | **Region** - Select an option from the drop-down, it is recommended to choose one in same geography as your Fabric tenant if possible 164 | 165 | **Primary service** - Select `Azure Blob Storage or Azure Data Lake Storage Gen 2` 166 | 167 | **Performance** - Select `Standard` 168 | 169 | **Redundancy** - Select `Geo-redundant storage (GRS)` 170 | 171 | ![Screenshot showing steps to provide Azure details to prepare deployment of an Azure Storage Account resource in the Microsoft Azure Portal](assets/azure-create-storage-account-details.png) 172 | 173 | It is suggested to leave all other options as they are set by default. You are now able to proceed to the `Review + submit` step to finalize deployment. When you are ready, select `Create` to begin the deployment. 174 | 175 | 4. You will need to wait for your deployment to complete, once this has finished, navigate to your newly deployed resource by selecting `Go to resource` on the resulting page. If you have lost your way, you can always find your deployed service by navigating to the Resource Group that was selected for deployment in the previous step. 176 | 177 | 5. Once you have successfully navigated to your storage account, on the left-hand side, expand `Settings` and select `Configuration`, then select `Enabled` under the option to `Allow Blob anonymous access` then select `Save` as shown: 178 | 179 | ![Screenshot showing steps to enable anonymous access for a storage account from Microsoft Azure Portal](assets/azure-enable-anonymous-access.png) 180 | 181 | 5. On the left-hand side, expand `Data management` and select `Static website`, then select `Enabled`, then select `Save` as shown: 182 | 183 | ![Screenshot showing steps to enable Static website for a storage account from Microsoft Azure Portal](assets/azure-enable-static-website.png) 184 | 185 | 6. On the left-hand side, expand `Settings` and select `Resource sharing (CORS)`, ensure that `Blob service` is highlighted then provide the following details to configure your web application and click `Save` as shown: 186 | 187 | **Allowed origins** - set this value to `*` 188 | 189 | **Allowed methods** - Select `Get` from the drop-down menu 190 | 191 | **Allowed headers** - set this value to `*` 192 | 193 | **Exposed headers** - set this value to `*` 194 | 195 | **Max age** - set this value to `1800` 196 | 197 | 198 | ![Screenshot showing steps to enable Resource sharing (CORS) for a storage account from Microsoft Azure Portal](assets/azure-enable-resource-sharing.png) 199 | 200 | 7. Finally navigate to `Storage browser => Blob containers` and select the `$web` checkbox, then click the 3 dots at the far-right and select `Change acccess level`: 201 | 202 | ![Screenshot showing steps to change access level for a storage account from Microsoft Azure Portal](assets/azure-change-access-level.png) 203 | 204 | 8. In the drop-down select `Blob (anonymous read access for blobs only)` then `Ok`: 205 | 206 | ![Screenshot showing steps to change access level to anonymous for a blob storage account from Microsoft Azure Portal](assets/azure-change-access-level-anonymous.png) 207 | 208 | 9. We are now done with the configuration of this service. We will return to this resource in future steps when creating the Fabric Data Pipeline. 209 | 210 | ## Create a Microsoft Fabric Workspace 211 | 212 | **Microsoft Fabric Workspace** - You will use Microsoft Fabric to: create a Lakehouse, run the provided notebook code, and create the Data pipeline that will copy your flashcard content to your Azure Storage account. The workspace provides a collaborative environment within the Microsoft Fabric platform where you can organize, manage, and share data-related assets. 213 | 214 | Access your Fabric instance by visiting [Microsoft Fabric online](https://app.fabric.microsoft.com), login with your credentials, and create a new Fabric-enabled workspace for this workshop. To accomplish this, on the home screen, select `New Workspace`, provide a value for `Name`, and ensure after expanding the `Advanced` section that you have selected a License mode that support creation of Fabric items as shown, then select `Apply`. 215 | 216 | ![Screenshot showing steps to create a workspace in Microsoft Fabric](assets/fabric-create-workspace.png) 217 | 218 | ## Create a Lakehouse in Microsoft Fabric 219 | 220 | **Lakehouse in Microsoft Fabric** - A lakehouse in Microsoft Fabric is a data architecture platform designed to store, manage, and analyze both structured and unstructured data in a single location. 221 | 222 | To learn more about Lakehouses in Microsoft Fabric, refer to [this Lakehouse tutorial](https://learn.microsoft.com/fabric/data-engineering/tutorial-build-lakehouse#create-a-lakehouse). 223 | 224 | 1. Create a new Lakehouse in your Microsoft Fabric workspace. To accomplish this, select `+ New item`, search for "Lakehouse" in the pane that opens, and select the `Lakehouse` item": 225 | 226 | ![Screenshot showing steps to create a new Lakehouse in Microsoft Fabric](assets/fabric-create-lakehouse.png) 227 | 228 | 2. Provide a name for your Lakehouse and select `Create` as shown: 229 | 230 | ![Screenshot showing steps to name a new Lakehouse in Microsoft Fabric](assets/fabric-name-lakehouse.png) 231 | 232 | ## Create a Notebook in Microsoft Fabric 233 | 234 | **Notebooks in Microsoft Fabric** - The Notebook is a primary code item for developing Apache Spark jobs and machine learning experiments. It is a web-based interactive service in Microsoft Fabric typically used by data scientists and data engineers. 235 | 236 | To learn more about Notebooks in Microsoft Fabric, refer to [the online documentation](https://learn.microsoft.com/fabric/data-engineering/how-to-use-notebook). 237 | 238 | 1. You should now be in the user interface of the Lakehouse. If you have lost your way, you can always find your Lakehouse by navigating to the Workspace that you created in the previous steps. Create a new notebook by selecting `Open Notebook` and then `new notebook` from the drop down menu. 239 | 240 | ![Screenshot of the New Notebook in the Lakehouse](assets/fabric-new-notebook.png) 241 | 242 | 3. Once the notebook is created, select the `Save as` icon in the upper left toolbar underneath `Home`, and save the notebook as `flashcards_workshop` as shown: 243 | 244 | ![Screenshot of the "Save As" Notebook dialog in the Lakehouse](assets/fabric-saveas-notebook.png) 245 | 246 | ## Install required Notebook libraries 247 | 248 | 1. Resume working in your recently created / saved Notebook and add the following code to the first cell to install the required Python Packages. 249 | 250 | ```bash 251 | #Install required Python Packages 252 | 253 | %pip install beautifulsoup4 #Python library for parsing data out of HTML and XML files 254 | %pip install azure-ai-inference #Azure AI Inference client library for Python 255 | ``` 256 | 257 | Run the cell by selecting the "Play" icon to the left of the cell. 258 | 259 |
260 | 261 | > When a notebook session is first created, non-standard libraries will not be available unless they have been installed into the session. If you receive errors in subsequent cells mentioning these libraries as missing, you likely need to re-run this cell as this usually indicates that your session was closed (relinquishing your installed libraries). 262 | 263 |
264 | 265 | 266 | ![Screenshot of the successful pip installation inside a Microsoft Fabric notebook](assets/fabric-notebook-pip-install.png) 267 | 268 | --- 269 | 270 | # Import and Process Data in your Lakehouse 271 | 272 | The first step is to import the data from an external source into your Lakehouse. For this workshop, we will use Microsoft Learn modules as our source material. We'll fetch the learn module Markdown files from the Microsoft Learn GitHub repository and import them into our Lakehouse. 273 | 274 | ![Microsoft Learn GitHub](assets/microsoft-learn.png) 275 | 276 | ## Import initial data and setup for processing in your Lakehouse 277 | 278 | To begin, we will familiarize with the problem set and the underlying data. Let's say we want to practice our knowledge on the [Get started with Real-Time Analytics in Microsoft Fabric](https://learn.microsoft.com/training/modules/get-started-kusto-fabric/) module. The source material for this module is available in the [Microsoft Learn GitHub repository](https://github.com/MicrosoftDocs/learn/). In this case navigate to the module folder [learn-pr/wwl/get-started-kusto-fabric](https://github.com/MicrosoftDocs/learn/tree/main/learn-pr/wwl/get-started-kusto-fabric). There you will find an `index.yml` file that contains the metadata for the module. This will list all the units and their respective Markdown files. To generate our flashcards we will need to specifiy a list of learn modules that we are interested in, we will then process this list to extract all relevant data from Microsoft Learn to assist in the generation of question and answer pairs in later steps. 279 | 280 | ![Screenshot of the Microsoft Learn GitHub Get started with Real-Time Analytics in Microsoft Fabric learn module](assets/microsoft-learn-github-kusto.png) 281 | 282 | 1. We will begin by uploading a CSV file containing a list of topics related to Microsoft Fabric along with an associated URL pointing to relevant module content on Microsoft Learn. We have prepared a list of topics with associated learn module urls. Start by downloading the file [topic_urls.csv](https://raw.githubusercontent.com/videlalvaro/fabcon-flashcards-workshop-site/refs/heads/main/initial_data/topic_urls.csv) 283 | 284 | >Note: You may need to "right-click => Save As" with filename `topic_urls.csv` 285 | 286 | Now, navigate to your notebook, select the `Files` folder icon in the Lakehouse explorer and right-click to reveal the options menu, from here select `New subfolder` and name this folder `initial_data`. Expand the `Files` tab to reveal the `initial_data` folder in the Lakehouse explorer, right-click it and select `Upload => Upload files`. Select the Folder icon in the `Upload files` pane, then navigate to the downloaded `topic_urls.csv` file and select `Upload`. The `topic_urls.csv` should now appear in your Lakehouse within the `initial_data` folder as shown: 287 | 288 | ![Screenshot showing topic_urls.csv successfully uploaded into the Fabric Lakehouse](assets/fabric-upload-csv.png) 289 | 290 | 2. Let's validate that the `topic_urls.csv` was uploaded to the appropriate location by reading the file and displaying it's contents within our notebook. Hover your mouse below the Log output of the previous cell to reveal the `+ Code` button, this will add an new code cell in the notebook. Add the following code to your notebook in the new cell: 291 | 292 | ```python 293 | #Open uploaded topic_urls.csv to validate it is in the appropriate location and able to be read 294 | with open('/lakehouse/default/Files/initial_data/topic_urls.csv') as list_of_modules: 295 | for line in list_of_modules: 296 | print(line) 297 | ``` 298 | 299 | Run the cell by selecting the "Play" icon to the left of the cell. 300 | 301 |
302 | 303 | > Going forward, it is expected that you run each cell in the notebook to execute the code. 304 | 305 |
306 | 307 | ![Screenshot showing output from reading the content of topic_urls.csv](assets/fabric-read-topics.png) 308 | 309 | 3. Add the following into a new code cell to import the packages and function definitions that will be used in later cells for fetching Markdown files from MS Learn and processing JSON / CSV formats. 310 | 311 | ```bash 312 | #Package imports and function definitions for fetching Markdown files and processing JSON / CSV formats from GitHub 313 | 314 | import re 315 | import requests 316 | import os 317 | import json 318 | import csv 319 | 320 | def parse_github_url(url): 321 | """ 322 | Parses a GitHub URL and extracts the owner, repository name, branch, and path. 323 | Sample: https://github.com/MicrosoftDocs/learn/blob/main/learn-pr/wwl/get-started-kusto-fabric/index.yml 324 | """ 325 | pattern = r"https://github\.com/(?P[^/]+)/(?P[^/]+)/blob/main/(?P[^/]+)/(?P.+)" 326 | match = re.match(pattern, url) 327 | if match: 328 | return match.groupdict() 329 | else: 330 | raise ValueError("Invalid GitHub URL format.") 331 | 332 | def download_and_save_file(file_url, lakehouse_save_path): 333 | """ 334 | Downloads a file from GitHub and saves it to the specified path in the Lakehouse. 335 | """ 336 | try: 337 | response = requests.get(file_url) 338 | if response.status_code == 200: 339 | # Create directories in Lakehouse if they don't exist 340 | print(f"lakehouse_save_path: {lakehouse_save_path}") 341 | dir_path = os.path.dirname(lakehouse_save_path) 342 | print(f"dir_path: {dir_path}") 343 | notebookutils.fs.mkdirs(dir_path) 344 | print(dir_path) 345 | # os.makedirs(dir_path, exist_ok=True) 346 | # Save the file content to Lakehouse 347 | notebookutils.fs.put(lakehouse_save_path, response.text, overwrite=True) 348 | # with open(lakehouse_save_path, 'w') as f: 349 | # f.write(response.text) 350 | print(f"Successfully downloaded {lakehouse_save_path}") 351 | return lakehouse_save_path 352 | else: 353 | print(f"Failed to download {file_url}: HTTP {response.status_code}") 354 | return False 355 | except Exception as e: 356 | print(f"Error downloading {file_url}: {e}") 357 | return False 358 | 359 | def download_folder_contents(contents, repo_path, base_save_path): 360 | """ 361 | Recursively downloads contents from a GitHub repository and saves them to the Lakehouse. 362 | """ 363 | 364 | saved_files = [] 365 | 366 | for item in contents: 367 | file_name = item['name'] 368 | download_url = item['download_url'] 369 | # Compute the relative path of the file within the module 370 | relative_path = item['path'].replace(repo_path + '/', '') 371 | print(f"relative_path: {relative_path}") 372 | # Construct the Lakehouse save path 373 | lakehouse_save_path = f"{base_save_path}/{relative_path}" 374 | # Download and save the file 375 | tmp = download_and_save_file(download_url, lakehouse_save_path) 376 | if tmp: 377 | saved_files.append(tmp) 378 | 379 | return saved_files 380 | ``` 381 | 382 | >Note: When a notebook session is first created, package imports and function definitions will not be available unless they have been imported / defined in the active session. If you receive errors in subsequent cells mentioning missing imports or defs, you likely need re-run this cell as this usually indicates that your session was closed (relinquishing your imports and function definitions). 383 | 384 | ![Screenshot of the successful Python imports and function definitions inside a Microsoft Fabric notebook](assets/fabric-notebook-imports.png) 385 | 386 | ## Fetch the markdown data 387 | 388 | 1. With our data processing functions now defined, we will use the BeautifulSoup package to parse the html content for all urls listed inside of `topic_urls.csv`, we will specifically scan the underlying metadata to produce a json document containing `module_url`, `topic`, `title`, and `index_yaml` (the link to the underlying markdown source for the module in question). Add the following to a new code cell, and run it. The resulting output will print out the url of each module as it parses through the list and display a table showing the resulting contents of the json document (`module_metadata.json`) in a Spark DataFrame. 389 | 390 | ```python 391 | #Obtain metadata for all modules in topic_urls.csv 392 | 393 | import requests 394 | from bs4 import BeautifulSoup 395 | 396 | module_metadata = [] 397 | 398 | with open('/lakehouse/default/Files/initial_data/topic_urls.csv') as list_of_modules: 399 | for line in list_of_modules: 400 | topic = line.split(",")[0] 401 | url = line.split(",")[1].strip() 402 | title = "" 403 | desc = "" 404 | index_yaml = "" 405 | print(url) 406 | 407 | response = requests.get(url) 408 | soup = BeautifulSoup(response.text, features="html.parser") 409 | 410 | metas = soup.find_all('meta') 411 | for meta in metas: 412 | # original_content_git_url 413 | # 414 | # https://github.com/MicrosoftDocs/learn-pr/blob/live/learn-pr/wwl/get-started-kusto-fabric/index.yml 415 | # https://github.com/MicrosoftDocs/learn/blob/main/learn-pr/wwl/get-started-kusto-fabric/index.yml 416 | 417 | if 'property' in meta.attrs and meta.attrs['property'] == 'og:title': 418 | title = meta.attrs['content'] 419 | # print(f"Title: {title}") 420 | if 'property' in meta.attrs and meta.attrs['property'] == 'og:description': 421 | desc = meta.attrs['content'] 422 | # print(f"Desc: {desc}") 423 | if 'name' in meta.attrs and meta.attrs['name'] == 'github_feedback_content_git_url': 424 | content = meta.attrs['content'] 425 | index_yaml = content 426 | # content = content.replace("https://github.com/MicrosoftDocs/learn/blob/main/", "") 427 | # content = content.replace("/index.yml", "") 428 | 429 | # file_list = list_markdown_files(f"{LEARN_PATH}/{content}/includes") 430 | temp = { 431 | "topic": topic, 432 | "module_url": url, 433 | "title": title, 434 | "desc": desc, 435 | "index_yaml": index_yaml 436 | } 437 | 438 | module_metadata.append(temp) 439 | 440 | with open("/lakehouse/default/Files/initial_data/module_metadata.json", "w") as f: 441 | json.dump(module_metadata, f) 442 | 443 | df = spark.read.option("multiline", "true").json("Files/initial_data/module_metadata.json") 444 | # df now is a Spark DataFrame containing JSON data from "Files/initial_data/module_metadata.json". 445 | display(df) 446 | ``` 447 | 448 | ![Screenshot of the code cell that produces module_metadata.json](assets/fabric-module-metadata1.png) 449 | 450 | ![Screenshot of the output of the code that produces module_metadata.json](assets/fabric-module-metadata2.png) 451 | 452 | The results of this cell should create a new file in the `initial_data` folder of your Lakehouse named `module_metadata.json`. If you see these file contents in the output of the DataFrame but no file of this name listed in the Lakehouse explorer Pane, right-click the `initial_data` folder in the Lakehouse Explorer and select `Refresh`, it should now be visible. 453 | 454 | 2. We now have the url to the associated `index_yaml` files for all modules that were originally specified in `topic_urls.csv`. These results are contained in the `module_metadata.json` produced in the previous step. Let's now loop through this list to get a full collection of all units within the module. The units contain the markdown that represents the actual content of the module. Once we have this data in our Lakehouse, we are ready to generate Questions and Answers from the contents of the units. To generate this list of module metadata with associated unit markdown files, create a new code cell and copy over this code snippet that pulls all associated markdown file urls (mds) for each module directly from GitHub: 455 | 456 | ```python 457 | #Obtain markdown files for all in units of each module listed in module_metadata.json 458 | 459 | # Define GitHub API URL 460 | GITHUB_API_URL = "https://api.github.com/repos" 461 | LAKEHOUSE_FILE_PATH = "Files" 462 | 463 | with open('/lakehouse/default/Files/initial_data/module_metadata.json') as list_of_modules: 464 | module_metadata = json.load(list_of_modules) 465 | module_metadata_with_mds = [] 466 | for metadata in module_metadata: 467 | # GITHUB_URL = "https://github.com/MicrosoftDocs/learn/blob/main/learn-pr/wwl/get-started-lakehouses/index.yml" 468 | # print(line.split(",")[1]) 469 | GITHUB_URL = metadata["index_yaml"] 470 | parsed_url = parse_github_url(GITHUB_URL) 471 | 472 | OWNER = parsed_url['owner'] 473 | REPO = parsed_url['repo'] 474 | BRANCH = parsed_url['branch'] 475 | REPO_PATH = parsed_url['path'] 476 | 477 | MODULE_NAME = os.path.dirname(REPO_PATH) 478 | MARKDOWN_PATH = f"{LAKEHOUSE_FILE_PATH}/markdown" 479 | 480 | api_url = f"{GITHUB_API_URL}/{OWNER}/{REPO}/contents/{BRANCH}/{MODULE_NAME}/includes" 481 | response = requests.get(api_url) 482 | if response.status_code == 200: 483 | contents = response.json() 484 | # print(contents) 485 | saved_mds = download_folder_contents(contents, REPO_PATH, MARKDOWN_PATH) 486 | metadata['markdowns'] = saved_mds 487 | module_metadata_with_mds.append(metadata) 488 | else: 489 | print(f"Failed to list contents of {REPO_PATH}: HTTP {response.status_code}") 490 | print(f"Response: {response.text}") 491 | 492 | with open("/lakehouse/default/Files/initial_data/module_metadata_with_mds.json", "w") as final_metadata: 493 | json.dump(module_metadata_with_mds, final_metadata) 494 | 495 | df = spark.read.option("multiline", "true").json("Files/initial_data/module_metadata_with_mds.json") 496 | # df now is a Spark DataFrame containing JSON data from "Files/initial_data/module_metadata_with_mds.json". 497 | display(df) 498 | ``` 499 | 500 | >Note: This cell usually takes about a minute or two to complete. Additionally, this particular cell may be subject to GitHub API rate-limiting / throttling. If many users execute this cell from a single outbound ip address, it may result in blocking of subsequent requests until the throttling is discontinued. If delivering this content as an interactive workshop, you may want to have a pre-generated result on hand, just in case! 501 | 502 | The results of this cell should create a new file in the `initial_data` folder of your Lakehouse named `module_metadata_with_mds.json`. If you see these file contents in the output of the DataFrame but no file of this name listed in the Lakehouse explorer Pane, right-click the `initial_data` folder in the Lakehouse explorer and select `Refresh`, it should now be visible. 503 | The results of this cell should also create new folder named `markdown` in the `Files` folder of the Lakhouse explorer pane. If you do not see the `markdown` folder under the `Files` folder in the Lakehouse explorer Pane, right-click the `Files` folder in the Lakehouse explorer and select `Refresh`, it should now be visible. 504 | 505 | ![Screenshot of the code cell that produces module_metadata_with_mds.json](assets/fabric-module-metadata-mds.png) 506 | 507 | --- 508 | 509 | # Generate Flashcards using Azure OpenAI Service in Azure AI Foundry 510 | 511 | Now that we have the Markdown files in our Lakehouse, we can use Azure OpenAI to generate a set of study flashcards. 512 | 513 | ## Setup Azure OpenAI 514 | 515 |
516 | 517 | > If you are using an Azure account that does not have access to Azure Open AI Service in Azure Foundry (Azure Free Trial accounts are not supported!), it is suggested to read the instructions that follow for completeness, then skip to the next step. 518 | 519 | If you are at a workshop, your proctor may share an API Key to allow you access to a service that has been configured for the event. 520 | 521 |
522 | 523 | 1. In your notebook, add the following code which is used to configure the Azure OpenAI client: 524 | 525 | ```python 526 | #Set prompts and configure Azure OpenAI client 527 | 528 | TASK_GUIDELINES = "Your task is help people learn from tutorials. You will receive a Markdown document, and extract from it pairs of questions and answers that will help the reader learn about the text." 529 | 530 | MODEL_GUIDELINES = """ 531 | Please follow these guidelines: 532 | 1. Questions and answers should be based on the input text. 533 | 2. Extract at least 5 different pairs of questions and answers. Questions and answers should be short. 534 | 3. Output should be valid JSON format. 535 | 4. Here's an example of your output format: [{"Q": "What is the name of the assistant?", "A": "Learn Assistant"}] 536 | 5. Output a plain array as explained above. Do not print anything else. 537 | """ 538 | 539 | AZURE_OPENAI_API_KEY="" 540 | AZURE_OPENAI_ENDPOINT="" 541 | AZURE_OPENAI_API_VERSION="2024-08-01-preview" 542 | ``` 543 | 544 | The idea is to tell the model to only generate questions and answers based on the input text. The model should generate at least 5 different pairs of questions and answers, and we provide a sample JSON format for the output, since our static web app will use that format to render interactive flashcards. 545 | 546 | 2. Now that we have configured the client, let's test our prompt by supplying it with the contents of one of the md files from our Lakehouse to ensure our Azure OpenAI Service is working as expected. Copy the following into a new cell and run it: 547 | 548 | ```python 549 | #Demonstrate example QnA request using content of single md file to test Azure OpenAI API Connection 550 | 551 | import os 552 | from azure.ai.inference import ChatCompletionsClient 553 | from azure.ai.inference.models import SystemMessage, UserMessage 554 | from azure.core.credentials import AzureKeyCredential 555 | 556 | model_name = "gpt-4o" 557 | 558 | client = ChatCompletionsClient( 559 | endpoint=AZURE_OPENAI_ENDPOINT, 560 | credential=AzureKeyCredential(AZURE_OPENAI_API_KEY), 561 | ) 562 | 563 | md = "Files/markdown/learn-pr/wwl/introduction-end-analytics-use-microsoft-fabric/includes/1-introduction.md" 564 | with open("/lakehouse/default/" + md, "r") as f: 565 | input_text = f.read() 566 | 567 | response = client.complete( 568 | messages=[ 569 | SystemMessage(content=TASK_GUIDELINES), 570 | SystemMessage(content=MODEL_GUIDELINES), 571 | UserMessage(content="Here's the document:"), 572 | UserMessage(content=input_text) 573 | ], 574 | max_tokens=4096, 575 | temperature=1.0, 576 | top_p=1.0, 577 | model=model_name 578 | ) 579 | 580 | print(response.choices[0].message.content) 581 | ``` 582 | 583 | You should see a collection of five QnA pairs generated based on the contents of the supplied markdown file: 584 | 585 | ![Screenshot of an example result that tests the Azure OpenAI API configuration](assets/fabric-openai-test.png) 586 | 587 | ## Generate the flashcards 588 | 589 | 1. Now we are almost ready to create flashcards by iterating through all of the markdown files that were retrieved in previous steps. First we will define a function that wil parse the model response into CSV format. Add the following to a code cell and execute it: 590 | 591 | ```python 592 | #Process JSON response from model and write specific data to a CSV file 593 | 594 | import csv 595 | 596 | def dump_to_csv(writer, file_metadata, md, model_response): 597 | try: 598 | qas = json.loads(model_response) 599 | # topic|module_url|unit_url|include_name|include_path|question|answer 600 | for qa in qas: 601 | include_name = md.split("/")[-1] 602 | print(f"processing: {include_name}") 603 | tmp = { 604 | "topic": file_metadata["topic"], 605 | "module_url": file_metadata["module_url"], 606 | "unit_url": file_metadata["module_url"] + include_name.replace(".md", ""), 607 | "include_name": include_name, 608 | "include_path": md.replace("Files/markdown/", ""), 609 | "question": qa["Q"], 610 | "answer": qa["A"] 611 | } 612 | writer.writerow(tmp.values()) 613 | except json.JSONDecodeError as e: 614 | with open("/lakehouse/default/Files/initial_data/errors.txt", "a") as error_file: 615 | print(e) 616 | error_file.write(md) 617 | error_file.write(e) 618 | error_file.write(model_response) 619 | ``` 620 | 621 | ![Screenshot of the dump_to_csv method defined in the notebook](assets/fabric-dumpcsv-definition.png) 622 | 623 | 2. With this method defined, we are now prepared to iterate through all of our markdown files and generate a full set of flashcards. For the purposes of demonstration, we will provide code that can be uncommented to do a full pass, but will only process the first markdown file. If you are attending a workshop, check with the proctors if you plan to execute this snippet with any modifications as it could be costly in terms of time and may render the OpenAI Service unusable if throttling is encountered. 624 | 625 | ```python 626 | #Demonstration of full loop to iterate through all markdown files and dump relevant content to CSV 627 | 628 | import json 629 | import csv 630 | 631 | with open("/lakehouse/default/Files/initial_data/module_metadata_with_mds.json") as f: 632 | file_list = json.load(f) 633 | 634 | csv_file = '/lakehouse/default/Files/initial_data/flashcards-uncurated.csv' 635 | 636 | with open(csv_file, mode='w', newline='') as file: 637 | writer = csv.writer(file, delimiter='|') 638 | # write the CSV headers 639 | writer.writerow(["topic", "module_url", "unit_url", "include_name", "include_path", "question", "answer"]) 640 | 641 | # Comment the following line if you want to create a full set of flashcards instead of just one learn module 642 | file_list = [file_list[0]] 643 | 644 | for file in file_list: 645 | print(file) 646 | for md in file['markdowns']: 647 | with open("/lakehouse/default/" + md, "r") as f: 648 | input_text = f.read() 649 | 650 | response = client.complete( 651 | messages=[ 652 | SystemMessage(content=TASK_GUIDELINES), 653 | SystemMessage(content=MODEL_GUIDELINES), 654 | UserMessage(content="Here's the document:"), 655 | UserMessage(content=input_text) 656 | ], 657 | max_tokens=4096, 658 | temperature=1.0, 659 | top_p=1.0, 660 | model=model_name 661 | ) 662 | model_response = response.choices[0].message.content 663 | dump_to_csv(writer, file, md, model_response) 664 | # Also remove the following break to generate QnA pairs for all markdown files. This can take a long time to complete depending on the amount of markdown files to be processed. 665 | break 666 | ``` 667 | 668 | 669 | As the code runs, you should see the generated questions and answers in the output. The questions and answers are saved to a file in the `initial_data` subfolder in the Lakehouse explorer named `flashcard-uncurated.csv`. The name of this file denotes that these results have not yet gone through human review, which is not only good for quality assurance but also to ensure that [Responsible AI Guidelines](https://www.microsoft.com/ai/principles-and-approach) are employed. If you do not see this file, hover your mouse above the `initial_data` subfolder in the Lakehouse expplorer, right-click, and select `Refresh`, it should now be visible. 670 | 671 |
672 | 673 | > A way to apply [Responsible AI](https://learn.microsoft.com/legal/cognitive-services/openai/overview) practices would be to review each question and answer generated by the model to ensure they are correct and relevant to the source material. 674 | 675 |
676 | 677 | 3. Let's take a look at the contents of this file by adding and running a cell with the following: 678 | 679 | ```python 680 | #View contents of generated flashcards-uncurated.csv 681 | 682 | import pandas as pd 683 | # Load data into pandas DataFrame from "/lakehouse/default/Files/initial_data/flashcards-uncurated.csv" 684 | df = pd.read_csv("/lakehouse/default/Files/initial_data/flashcards-uncurated.csv", delimiter="|") 685 | display(df) 686 | ``` 687 | 688 | ![Screenshot of the generated flashcards_uncurated.csv in the notebook](assets/fabric-qna-uncurated.png) 689 | 690 | ## Review the generated flashcards 691 | 692 | We need to ensure that we have generated quality results to ensure are QnA pairs are not erroneous or otherwise unfit for use as a legitimate study aid. We will employ human-in-the-loop to validate our results before we ship them off for rendering in our static web app. 693 | 694 | Navigate to your Lakehouse, select the `Files` tab and click `Refresh`. You should see two folders in your Lakehouse (`initial_data` and `markdown`). Click into `initial_data` and select `flashcards-uncurated.csv`. Review that the content to ensure that it meets quality statndard and make any necessary modifications to the content if necessary by saving off, deleting from the lakehouse, and re-uploading back to the Lakehouse as `flashcards-uncurated.csv`. When you are satisfied, navigate back to the `initial_data` folder and rename `flashcards-uncurated.csv` to `flashcards-curated.csv` by left-clicking the filename and selecting `Rename`. 695 | 696 | ![Screenshot of the human-reviewed flashcards_curated.csv file in the lakehouse](assets/fabric-qna-curated.png) 697 | 698 | ## Prepare generated results for rendering in Static Web App 699 | 700 | 1. Navigate back to your Notebook. Now that we have our generated QnA pairs, we need to do some minor transformations in order to prepare our data for rendering in the static Web App. The following will generate a `topics.json` file which represents the topic for a given collection of individual questions and answers, these derive from the topic tags originating in `initial_data/topic_urls.json` that were applied to the output when producing `initial_data/module_metadata.json`. The `topic.json` that we will create here defines the topics or categories for our Question and Answer pairs to allow us to organize them accordingly in the static web app user interface. 701 | 702 | ```python 703 | #Generate a topics.json file that defines the topics for our Question and Answer pairs 704 | #Allowing us to organize them accordingly in the static web app user interface 705 | 706 | import json 707 | 708 | topics = [] 709 | 710 | with open("/lakehouse/default/Files/initial_data/module_metadata.json") as f: 711 | data = json.load(f) 712 | for row in data: 713 | # https://github.com/MicrosoftDocs/learn/blob/main/learn-pr/wwl-data-ai/get-started-with-graphql-microsoft-fabric/index.yml 714 | tmp = { 715 | "title": row['title'], 716 | "summary": row["desc"], 717 | "module": row["index_yaml"].split("/")[-2] 718 | } 719 | topics.append(tmp) 720 | 721 | notebookutils.fs.mkdirs("Files/final_data") 722 | notebookutils.fs.put("Files/final_data/topics.json", json.dumps(topics)) 723 | 724 | df = spark.read.option("multiline", "true").json("Files/final_data/topics.json") 725 | # df now is a Spark DataFrame containing JSON data from "Files/final_data/topics.json". 726 | display(df) 727 | ``` 728 | 729 | If you see these file contents in the output of the DataFrame but do not see a folder named `final_data` in the Lakehouse explorer Pane, right-click the `Files` folder in the Lakehouse Explorer and select `Refresh`, it should now be visible. 730 | 731 | ![Screenshot of the output to generate the topics.json file in the lakehouse](assets/fabric-topics-generated.png) 732 | 733 | 734 | 2. Next, we will generate a `generated-QAs.json` that is a full collection of all QnA pairs along with a source pointing to the url from which the question was derived during generation. This will be used by the static web app to render actual QnA contents. To create this file, run the following in a new code cell: 735 | 736 | 737 | ```python 738 | #Generate a generated-QAs.json file that defines our Question and Answer pairs 739 | #Allowing us to render them accordingly in the static web app user interface 740 | 741 | import pandas as pd 742 | 743 | # Specify the path to your CSV file and the custom delimiter 744 | file_path = '/lakehouse/default/Files/initial_data/flashcards-curated.csv' 745 | delimiter = '|' # Example delimiter 746 | 747 | # Read the CSV file 748 | df = pd.read_csv(file_path, delimiter=delimiter, header=0) 749 | 750 | websiteQAs = {} 751 | 752 | for index, row in df.iterrows(): 753 | module_id = row['include_path'].split("/")[-3] 754 | if module_id not in websiteQAs: 755 | websiteQAs[module_id] = [] 756 | else: 757 | tmp = { 758 | "q": row['question'], 759 | "a": row['answer'], 760 | "source": row['include_name'] 761 | } 762 | websiteQAs[module_id].append(tmp) 763 | 764 | notebookutils.fs.put("Files/final_data/generated-QAs.json", json.dumps(websiteQAs)) 765 | 766 | df = spark.read.option("multiline", "true").json("Files/final_data/generated-QAs.json") 767 | # df now is a Spark DataFrame containing JSON data from "Files/final_data/generated-QAs.json". 768 | display(df) 769 | 770 | ``` 771 | 772 | If you see these file contents in the output of the DataFrame but do not see a file in the `final_data` folder named `generated-QAs.json` in the Lakehouse explorer Pane, right-click the `final_data` folder in the Lakehouse Explorer and select `Refresh`, it should now be visible. 773 | 774 | ![Screenshot of the output to generate the generated-QAs.json file in the lakehouse](assets/fabric-generated-qas.png) 775 | 776 | 777 | --- 778 | 779 | # Create a Fabric Data Pipeline 780 | 781 | In this section you are going to learn how to use Fabric Data Pipelines to copy data from your Lakehouse to Azure Storage. This is one of the many ways you can use to connect your data in Fabric with external applications. 782 | 783 | ## Update Lakehouse with full-set of QnA pairs 784 | 785 | If you have followed all workshop instructions up to this point without modification, you would have generated a subset of QnA pairs since for sample purposes we didn't want to over extend your Azure OpenAI quota. We will begin this section under that assumption. 786 | 787 | Of course, our end goal is to produce a static web app with a full-set of questions so you can use it to study for a real exam! To do achieve this, we will start by updating the `topics.json` and `generated-QAs.json` files that are currently in our Lakehouse with known-working, hand-curated topics questions that cover 12 unique topics in Microsoft Fabric. 788 | 789 | 1. Download a copy of the full [topics.json](https://raw.githubusercontent.com/videlalvaro/fabcon-flashcards-workshop-site/refs/heads/main/src/topics.json) 790 | 791 | >Note: You may need to "right-click => Save As" with filename `topics.json` 792 | 793 | 2. Download a copy of the full [generated-QAs.json](https://raw.githubusercontent.com/videlalvaro/fabcon-flashcards-workshop-site/refs/heads/main/src/generated-QAs.json) 794 | 795 | >Note: You may need to "right-click => Save As" with filename `generated-QAs.json` 796 | 797 | 3. Navigate to your Lakehouse in Microsoft Fabric and expand the `Files` folder, then select the `final_data` subfolder. Right-Click the `final_data` subfolder and select `Upload => Upload files`. Ensure that the "Overwrite if files already exist" option is selected and upload both of the newly downloaded `topics.json` and `generated-QAs.json` into the `final_data` subfolder as shown: 798 | 799 | ![Screenshot showing upload and overwriting of the full topics.json and generated-QAs.json files into the lakehouse](assets/fabric-upload-full-topic-and-qas.png) 800 | 801 | ## Copy the JSON Data to Azure Blob Storage 802 | 803 | > For this section, if you do not have access to an Azure account, it is still suggested to read the instructions that follow for completeness, and then skip to the next section. You will still be able to deploy a functional static web app, as this only requires an active GitHub account, but you will not be able to serve data to your static web app from Azure. 804 | 805 | Now that we have the files that represent a full set of flashcards in our Lakehouse, we can run a [data pipeline](https://learn.microsoft.com/training/modules/use-data-factory-pipelines-fabric/) to copy the data to an external data store ([Azure Blob Storage](https://learn.microsoft.com/training/modules/explore-azure-blob-storage/)) for public consumption. 806 | 807 | 1. Select the `Workspace` button, and select your existing workspace. In the workspace view, select `+ New item`, then search for `Data pipeline` to create a new data pipeline. Name it `flashcards_pipeline`, and then select `Create`. 808 | 809 | ![Screenshot of the New Item view with the data pipeline option visible](assets/fabric-create-pipeline.png) 810 | 811 | 2. Inside the pipeline, select "Copy data assistant" to start building the pipeline with a with a `Copy data` activity. In the next view you will configure the Copy data activity. 812 | 813 | ![Screenshot of a data pipeline with an empty canvas.](assets/fabric-new-data-pipeline.png) 814 | 815 | In the `Choose a data source` view, select the Lakehouse you previously created. 816 | In the `Connect to data source` view, select `Files` as the root folder, then in the directory open the `final_data` folder and select `topics.json` and select **Next**. 817 | 818 | ![Screenshot of the New Connection view for a Lakehouse configured as the pipeline source](assets/data-pipeline-source-connection-lakehouse-topics.png) 819 | 820 | In the `Choose data destination` view, search and select `Azure Blobs`. 821 | 822 | ![Screenshot of the New Connection view for Azure Blob storage selected to be the pipeline destination](assets/data-pipeline-connection.png) 823 | 824 | Enter the following information about the Azure Storage account that you created in the previous step: 825 | 826 | - Account name or URL: Storage account name 827 | - Connection: Create new connection 828 | - Connection name: FlashCards Azure Blob Storage 829 | - Data gateway: (none) 830 | - Authentication kind: Shared Access Signature (SAS) - see steps below for how to create this 831 | 832 | Create a SAS token in your Azure Blob Storage account in the Portal in the Shared Access Signature view and check `Service`, `Container`, and `Object` under **Allowed resource types**. The other default values can be left as is. 833 | 834 | ![Screenshot of the Azure Blob Storage Shared Access Signature view](assets/azure-blob-storage-sas.png) 835 | 836 | Select `Generate SAS token` and copy the `SAS token` value to the clipboard and paste it into the `SAS token` field back over in your `Copy Data => Connect to data destination` prompt in Microsoft Fabric then select `Next`. 837 | 838 | ![Screenshot of the New Connection view for Azure Blob storage configured as the pipeline destination ](assets/data-pipeline-connection-azure-blob.png) 839 | 840 | In the `File path` view, select `Browse`, and then select `$web` then select `OK` and then select `Next`. 841 | 842 | ![Screenshot of the "Connect to data destination fields part 1](assets/data-pipeline-connect-to-destination1.png) 843 | 844 | In the next view, set the file format to `JSON` and leave the default values as they are and select `Next`. 845 | 846 | ![Screenshot of the "Connect to data destination fields part 2](assets/data-pipeline-connect-to-destination2.png) 847 | 848 | Review the configuration of the source and destination and confirm **Start data transfer immediately** is checked and select `Save + Run` and the pipeline will begin to run. 849 | 850 | ![Screenshot the review and save view](assets/data-pipeline-copy-confirm.png) 851 | 852 | 3. Select the `Copy data` item and rename the activity to `Copy JSON file to Azure Blob Storage` in the `General` tab. If you click into the empty space in the pipeline editor, you can monitor the progress of the pipeline in by selecting the `Output` tab. 853 | 854 | 855 | ![Screenshot of the pipeline running](assets/data-pipeline-running.png) 856 | 857 | 858 | Once the pipeline is complete, the `topics.json` file will be in the configured destination of the Azure Blob Storage account. 859 | 860 | ![Screenshot of the Azure Blob Storage account with the topics.json file](assets/azure-blob-storage-topics.png) 861 | 862 | 4. Back in the pipline editor, select the `Copy Data` item again and select the "Clone" icon at the bottom of the item. 863 | 864 | ![Screenshot of the the Clone icon selected in the pipeline editor](assets/data-pipeline-clone.png) 865 | 866 | 867 | Select the newly duplicated `Copy Data` item and configure the source as shown to create another Copy data item for the `generated-QAs.json` file: 868 | 869 | ![Screenshot of the the Clone icon selected in the pipeline editor](assets/data-pipeline-clone-source.png) 870 | 871 | Now save and run your pipeline, once it completes, you should see both files have successfully copied into your Azure storage account: 872 | 873 | ![Screenshot of the pipeline completed](assets/data-pipeline-completed.png) 874 | 875 | ![Screenshot of the Azure Blob Storage account with the topics.json and generated-QAs json files](assets/azure-blob-storage-topics-and-qas.png) 876 | 877 | If you reached this point you have successfully used Pipelines to copy data from your Lakehouse to Azure Blob Storage. Of course there's much more you can do with pipelines [Orchestrate processes and data movement with Microsoft Fabric](https://learn.microsoft.com/training/modules/use-data-factory-pipelines-fabric/) 878 | 879 | --- 880 | 881 | 882 | # Publish the Flashcards to a Static Web App 883 | 884 | We are now ready to display our flashcards in a static web app on GitHub. [GitHub Pages](https://docs.github.com/pages) is a static site hosting service that allows you to host a website directly from a repository on GitHub. It can be used to host personal, organizational, or project websites. GitHub Pages takes HTML, CSS, and JavaScript files straight from a repository on GitHub, optionally runs them through a build process, and publishes as a website. 885 | 886 | ## Fork a copy of the Flashcard Web App 887 | 888 | 1. Navigate to the [Flashcards Web App repository](https://github.com/videlalvaro/fabcon-flashcards-workshop-site) while logged into your GitHub account and fork the repository. 889 | 890 | ![Screenshot of the Fabric Flashcards website repository](assets/workshop-website-repo.png) 891 | 892 | 2. Ensure that "Copy the main branch only" option is checked and select `Create fork`: 893 | 894 | ![Screenshot of the Fabric Flashcards website repository being forked](assets/workshop-website-repo-fork.png) 895 | 896 | 897 | ## Change the source of the topics and generated QAs 898 | 899 | You will update the web app to use your own `topics.json` and `generated-QAs.json` as the data source of the web app. For those with access to an Azure Account, we will serve this from your blob storage account which should now have the latest and greatest curated versions of these files. For those without access to an Azure Account, we will serve these files from your own GitHub repo. 900 | 901 | 1. Inside your forked repo, navigate to the `src/app/page.tsx` path of the website repository. Look for the pencil icon to edit the contents of this file: 902 | 903 | ![Screenshot of the Fabric Flashcards website repository fork being edited](assets/workshop-website-repo-fork-edit.png) 904 | 905 | 906 | 2. Update the existing `topicsURL` and `QAsURL` variables adding as follows, depending on whether you have access to an Azure Account. 907 | 908 | If you have access to an Azure Account, navigate to the `Storage browser`, select `Blob containers`, select then `$web` and select the `topics.json file` and copy the URL by clicking the clipboard icon (this value will become `topicsURL`): 909 | 910 | ![Screenshot showing how to copy topics.json URL from Azure storage in Azure Portal](assets/azure-blob-storage-copy-topics.png) 911 | 912 | Simlarly, if you have access to an Azure Account, navigate to the `Storage browser`, select `Blob containers`, and then `$web` and select the `generated-QAs.json file` and copy the URL by clicking the clipboard icon (this value will become `QAsURL`): 913 | 914 | ![Screenshot showing how to copy generated-QAs.json file URL from Azure storage in Azure Portal](assets/azure-blob-storage-copy-qas.png) 915 | 916 | If you do not have access to an Azure Account, modify the `topicsURL` and `QAsURL` variables as shown (this will serve your web app using resources from your forked repo!): 917 | 918 | ```javascript 919 | const topicsURL = "https://raw.githubusercontent.com/[your GitHub handle]/fabcon-flashcards-workshop-site/refs/heads/main/src/topics.json" 920 | 921 | const QAsURL = "https://raw.githubusercontent.com/[your GitHub handle]/fabcon-flashcards-workshop-site/refs/heads/main/src/generated-QAs.json" 922 | ``` 923 | 924 | 3. Once you have made your changes in the editor, select `Commit changes` and provide a desciption if you like, then select `Commit changes` again to `Commit diretly to the main branch`: 925 | 926 | ![Screenshot of the Fabric Flashcards website repository fork committing a change](assets/workshop-website-repo-fork-commit.png) 927 | 928 | ## Enable GitHub Pages 929 | 930 | 1. Navigate to your forked repository and select `Settings`, then navigate to **Pages** under **Code and automation**. Select `GitHub Actions` in the dropdown under `Source`. This will configure your repository to a deploy a static website to GitHub pages using GitHub Actions. 931 | 932 | ![Screenshot of the Pages view in the repository's settings.](assets/github-actions-config.png) 933 | 934 | 2. Next, navigate to **Actions**, when prompted, select `I understand my workflows, go ahead and enable them` 935 | 936 | ![Screenshot of notification that workflows aren't configured to run on the repository.](assets/github-actions-confirm.png) 937 | 938 | 3. Select the Action titled `Deploy Next.js site to Pages` then choose `Run workflow` and ensure that `Branch:main` is selected then click the `Run workflow` button to kick off the deployment. 939 | 940 | ![Screenshot showing where to select to start the deployment workflow.](assets/github-actions-run.png) 941 | 942 | 4. The deployment should now be in progress, it should take a minute or so to complete 943 | 944 | ![Screenshot showing the deployment workflow running.](assets/github-actions-running.png) 945 | 946 | 5. When you have confirmed the deployment has completed successfully, navigate to `Settings` then select **Pages** under **Code and automation**. Notice the mention that "Your site is live at" that then displays the URL for your static web. Click the `Visit site` link to navigate to your newly deployed flashcard site. 947 | 948 | ![Screenshot showing the live URL of your deployed static web site.](assets/github-pages-live.png) 949 | 950 | 6. If using a devtools capable browser, press 'F12' to open the devtools pane, then refresh your web site, and select the `Network` tab. Verify that your `topic.json` and `generated-QAs.json` are being served from your modified location. The example below shows a static site that is now pulling data from Azure Storage: 951 | 952 | 953 | ![Screenshot sources of topics.json and generated-QAs.json in the deployed static web site.](assets/github-pages-live-inspect.png) 954 | 955 | So like that you can see how you can host a static website on GitHub Pages while at the same time serving dynamicly generated data from Azure. How cool is that! 956 | 957 | --- 958 | 959 | # Conclusion 960 | 961 | This concludes this workshop, we hope you enjoyed it and learned something new. 962 | 963 | In this workshop, you learned how to use Microsoft Fabric and Azure OpenAI Service in Azure AI Foundry to generate a set of study flashcards. You connected Fabric to Azure Blob Storage to store the flashcards and then served them in a static web app to make them available to anyone, anywhere, anytime! 964 | 965 | Special thanks to [Kevin Comba Gatimu](https://github.com/kelcho-spense), Microsoft MVP, who assisted in development of the mechanism used for fetching markdown files from Github! 966 | 967 | ## Clean up resources 968 | 969 |
970 | 971 | > After completing the workshop, you may to delete your Microsoft Fabric items and / or Azure Resources that were created (especially if you do not plan on using them further) to avoid incurring unnecessary costs! 972 | 973 |
974 | 975 | ## Resources 976 | 977 | To learn more about Retrieval Augmented Generation (RAG) using Azure Search an Azure OpenAI, refer to the following resources: 978 | 979 | - [Retrieval Augmented Generation (RAG) in Azure AI Search](https://learn.microsoft.com/azure/search/retrieval-augmented-generation-overview) 980 | - [Use Azure OpenAI in Fabric with Python SDK and Synapse ML (preview)](https://learn.microsoft.com/fabric/data-science/ai-services/how-to-use-openai-sdk-synapse) 981 | - [Azure OpenAI for big data](https://microsoft.github.io/SynapseML/docs/Explore%20Algorithms/OpenAI/) --------------------------------------------------------------------------------