├── macros ├── .gitkeep ├── combinations.sql ├── create_description_model.sql └── create_metadata_model.sql ├── integration_tests ├── seeds │ ├── dummy_seed.csv │ └── dummy_seed.yml ├── packages.yml ├── models │ ├── dummy_models │ │ ├── dummy_model_1.yml │ │ ├── dummy_model_2.yml │ │ ├── dummy_model_2.sql │ │ ├── source.yml │ │ └── dummy_model_1.sql │ ├── metadata_tests │ │ ├── 01_metadata_test.sql │ │ ├── 02_metadata_test_with_granularity.sql │ │ ├── 03_metadata_test_resource_type.sql │ │ ├── 04_metadata_test_show_resource_type.sql │ │ ├── 05_metadata_test_undefined.sql │ │ ├── 06_metadata_test_undefined_as_null.sql │ │ └── 07_metadata_test_files.sql │ └── description_tests │ │ └── description_test_files.sql └── dbt_project.yml ├── .gitignore ├── dbt_project.yml ├── LICENSE └── README.md /macros/.gitkeep: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /integration_tests/seeds/dummy_seed.csv: -------------------------------------------------------------------------------- 1 | seed_column 2 | 1 -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | .DS_Store 2 | target/ 3 | dbt_packages/ 4 | logs/ 5 | -------------------------------------------------------------------------------- /integration_tests/packages.yml: -------------------------------------------------------------------------------- 1 | packages: 2 | - local: ../ 3 | -------------------------------------------------------------------------------- /integration_tests/models/dummy_models/dummy_model_1.yml: -------------------------------------------------------------------------------- 1 | version: 2 2 | 3 | models: 4 | - name: 'dummy_model_1' 5 | 6 | columns: 7 | - name: "dummy" 8 | description: "the description of the dummy column of dummy_model_1" -------------------------------------------------------------------------------- /integration_tests/models/dummy_models/dummy_model_2.yml: -------------------------------------------------------------------------------- 1 | version: 2 2 | 3 | models: 4 | - name: 'dummy_model_2' 5 | description: "description of dummy_model_2" 6 | 7 | columns: 8 | - name: "dummy" 9 | description: "the description of the dummy column of dummy_model_2" -------------------------------------------------------------------------------- /integration_tests/models/dummy_models/dummy_model_2.sql: -------------------------------------------------------------------------------- 1 | {{ config( 2 | meta={ 3 | 'main_subject': 'people' 4 | , 'owner': 'Bob' 5 | , 'business_questions': 'How many employees ...?' 6 | } 7 | )}} 8 | 9 | select * 10 | from {{ source('raw', 'dummy_raw') }} -------------------------------------------------------------------------------- /integration_tests/models/metadata_tests/01_metadata_test.sql: -------------------------------------------------------------------------------- 1 | {{ metalog.create_metadata_model( 2 | metadata = [ 3 | "main_subject" 4 | , "owner" 5 | , "business_questions" 6 | , "business_rules" 7 | , "todos" 8 | ] 9 | )}} -------------------------------------------------------------------------------- /dbt_project.yml: -------------------------------------------------------------------------------- 1 | name: 'metalog' 2 | version: '1.0.0' 3 | config-version: 2 4 | require-dbt-version: [">=1.0.0", "<2.0.0"] 5 | 6 | profile: 'metalog' 7 | 8 | macro-paths: ["macros"] 9 | 10 | target-path: "target" 11 | clean-targets: 12 | - "target" 13 | - "dbt_packages" 14 | 15 | 16 | 17 | -------------------------------------------------------------------------------- /integration_tests/seeds/dummy_seed.yml: -------------------------------------------------------------------------------- 1 | version: 2 2 | 3 | seeds: 4 | - name: 'dummy_seed' 5 | description: "stores" 6 | config: 7 | meta: { 8 | 'main_subject': 'sales' 9 | , 'owner': 'Carl' 10 | } 11 | 12 | columns: 13 | - name: "seed_column" 14 | description: "the description of the dummy seed column of dummy_seed" -------------------------------------------------------------------------------- /integration_tests/models/description_tests/description_test_files.sql: -------------------------------------------------------------------------------- 1 | {{ metalog.create_description_model( 2 | resource_type = [ 3 | "model" 4 | , "seed" 5 | , "source" 6 | ] 7 | , show_resource_type = True 8 | , files = [ 9 | '.*.sql' 10 | ] 11 | , exclude_files = [ 12 | 'models/metadata.*' 13 | ] 14 | )}} -------------------------------------------------------------------------------- /integration_tests/models/metadata_tests/02_metadata_test_with_granularity.sql: -------------------------------------------------------------------------------- 1 | {{ metalog.create_metadata_model( 2 | metadata = [ 3 | "main_subject" 4 | , "owner" 5 | , "business_questions" 6 | , "business_rules" 7 | , "todos" 8 | ] 9 | , granularity = [ 10 | "business_questions" 11 | , "business_rules" 12 | , "todos" 13 | ] 14 | )}} -------------------------------------------------------------------------------- /integration_tests/models/dummy_models/source.yml: -------------------------------------------------------------------------------- 1 | version: 2 2 | 3 | sources: 4 | - name: raw 5 | description: "dummy raw source." 6 | schema: raw 7 | tables: 8 | - name: dummy_raw 9 | description: "dummy raw source table" 10 | columns: 11 | - name: source_column 12 | description: "dummy source column." 13 | meta: { 14 | 'main_subject': 'sales' 15 | , 'owner': 'Carl' 16 | } -------------------------------------------------------------------------------- /integration_tests/dbt_project.yml: -------------------------------------------------------------------------------- 1 | name: 'metalog_integration_tests' 2 | version: '1.0' 3 | config-version: 2 4 | 5 | profile: 'metalog_integration_tests' 6 | 7 | model-paths: ["models"] 8 | analysis-paths: ["analyses"] 9 | test-paths: ["tests"] 10 | seed-paths: ["seeds"] 11 | macro-paths: ["macros"] 12 | 13 | target-path: "target" # directory which will store compiled SQL files 14 | clean-targets: # directories to be removed by `dbt clean` 15 | - "target" 16 | - "dbt_packages" -------------------------------------------------------------------------------- /integration_tests/models/dummy_models/dummy_model_1.sql: -------------------------------------------------------------------------------- 1 | {{ config( 2 | meta={ 3 | 'main_subject': 'sales' 4 | , 'owner': 'Alice' 5 | , 'business_questions': [ 6 | 'How many stores of type ...?' 7 | , 'How many stores in ...?' 8 | ] 9 | , 'business_rules': [ 10 | 'Stores of type A receive code B ...' 11 | , 'Consider only stores open after ...' 12 | ] 13 | , 'todos': 'change to incremental' 14 | } 15 | )}} 16 | 17 | select 1 as dummy -------------------------------------------------------------------------------- /integration_tests/models/metadata_tests/03_metadata_test_resource_type.sql: -------------------------------------------------------------------------------- 1 | {{ metalog.create_metadata_model( 2 | metadata = [ 3 | "main_subject" 4 | , "owner" 5 | , "business_questions" 6 | , "business_rules" 7 | , "todos" 8 | ] 9 | , granularity = [ 10 | "business_questions" 11 | , "business_rules" 12 | , "todos" 13 | ] 14 | , resource_type = [ 15 | "model" 16 | , "seed" 17 | ] 18 | )}} -------------------------------------------------------------------------------- /integration_tests/models/metadata_tests/04_metadata_test_show_resource_type.sql: -------------------------------------------------------------------------------- 1 | {{ metalog.create_metadata_model( 2 | metadata = [ 3 | "main_subject" 4 | , "owner" 5 | , "business_questions" 6 | , "business_rules" 7 | , "todos" 8 | ] 9 | , granularity = [ 10 | "business_questions" 11 | , "business_rules" 12 | , "todos" 13 | ] 14 | , resource_type = [ 15 | "model" 16 | , "seed" 17 | ] 18 | , show_resource_type = False 19 | )}} -------------------------------------------------------------------------------- /integration_tests/models/metadata_tests/05_metadata_test_undefined.sql: -------------------------------------------------------------------------------- 1 | {{ metalog.create_metadata_model( 2 | metadata = [ 3 | "main_subject" 4 | , "owner" 5 | , "business_questions" 6 | , "business_rules" 7 | , "todos" 8 | ] 9 | , granularity = [ 10 | "business_questions" 11 | , "business_rules" 12 | , "todos" 13 | ] 14 | , resource_type = [ 15 | "model" 16 | , "seed" 17 | ] 18 | , show_resource_type = True 19 | , undefined = "Not defined" 20 | )}} -------------------------------------------------------------------------------- /macros/combinations.sql: -------------------------------------------------------------------------------- 1 | {% macro combinations(lists) %} 2 | 3 | {% set all_combinations = [] %} 4 | 5 | {{ metalog.recursive_combinations(lists, 0, [], all_combinations) }} 6 | 7 | {{ return(all_combinations) }} 8 | 9 | {% endmacro %} 10 | 11 | {% macro recursive_combinations(lists, id, current_combination, all_combinations) %} 12 | 13 | {% if id == lists | length %} 14 | {{ all_combinations.append(current_combination) }} 15 | {% endif %} 16 | 17 | {% for item in lists[id] %} 18 | {{ metalog.recursive_combinations(lists, id + 1, current_combination + [item], all_combinations) }} 19 | {% endfor %} 20 | 21 | {% endmacro %} -------------------------------------------------------------------------------- /integration_tests/models/metadata_tests/06_metadata_test_undefined_as_null.sql: -------------------------------------------------------------------------------- 1 | {{ metalog.create_metadata_model( 2 | metadata = [ 3 | "main_subject" 4 | , "owner" 5 | , "business_questions" 6 | , "business_rules" 7 | , "todos" 8 | ] 9 | , granularity = [ 10 | "business_questions" 11 | , "business_rules" 12 | , "todos" 13 | ] 14 | , resource_type = [ 15 | "model" 16 | , "seed" 17 | ] 18 | , show_resource_type = True 19 | , undefined = "Not defined" 20 | , undefined_as_null = True 21 | )}} -------------------------------------------------------------------------------- /integration_tests/models/metadata_tests/07_metadata_test_files.sql: -------------------------------------------------------------------------------- 1 | {{ metalog.create_metadata_model( 2 | metadata = [ 3 | "main_subject" 4 | , "owner" 5 | , "business_questions" 6 | , "business_rules" 7 | , "todos" 8 | ] 9 | , granularity = [ 10 | "business_questions" 11 | , "business_rules" 12 | , "todos" 13 | ] 14 | , resource_type = [ 15 | "model" 16 | , "seed" 17 | , "source" 18 | ] 19 | , show_resource_type = True 20 | , undefined = "Not defined" 21 | , undefined_as_null = True 22 | , files = [ 23 | "models/.*" 24 | , "seeds/.*" 25 | , ".*/source.yml" 26 | ] 27 | , exclude_files = [ 28 | 'models/metadata_tests/.*' 29 | 'models/description_tests/.*' 30 | ] 31 | )}} -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2023 techindicium 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /macros/create_description_model.sql: -------------------------------------------------------------------------------- 1 | {%- macro create_description_model( 2 | resource_type=["model"] 3 | , show_resource_type=True 4 | , files=[] 5 | , exclude_files=[] 6 | ) -%} 7 | 8 | {%- if execute -%} 9 | 10 | {%- set rows_list = metalog.get_rows(resource_type, files, exclude_files) -%} 11 | 12 | {%- if rows_list | length == 0 -%} 13 | 14 | {{ exceptions.raise_compiler_error("No description found for the provided parameters\nPlease check the descriptions of your resources") }} 15 | 16 | {%- endif -%} 17 | 18 | {%- for row in rows_list -%} 19 | 20 | select "{{ row[0] }}" as resource_name 21 | 22 | {%- if show_resource_type -%} 23 | , "{{ row[1] }}" as resource_type 24 | {%- endif -%} 25 | , "{{ row[2] }}" as resource_description 26 | , "{{ row[3] }}" as column_name 27 | , "{{ row[4] }}" as columns_description 28 | {% if not loop.last %} 29 | union all 30 | {% endif %} 31 | 32 | {%- endfor -%} 33 | 34 | {%- endif -%} 35 | 36 | {%- endmacro -%} 37 | 38 | 39 | {%- macro get_rows( 40 | resource_type_list 41 | , files_list 42 | , exclude_files_list 43 | ) -%} 44 | 45 | {% set re = modules.re %} 46 | 47 | {%- set rows_list = [] -%} 48 | 49 | {%- for node in graph.nodes.values() if node.resource_type in resource_type_list -%} 50 | 51 | {# "Check if node is in the provided files" #} 52 | {%- set valid_files = [] -%} 53 | {%- if files_list -%} 54 | {%- for file in files_list if re.match(file, node.original_file_path, re.IGNORECASE) -%} 55 | {%- if exclude_files_list -%} 56 | {%- for file_exclude in exclude_files_list if not re.match(file_exclude, node.original_file_path, re.IGNORECASE) -%} 57 | {{ valid_files.append(1) }} 58 | {%- endfor -%} 59 | {%- else -%} 60 | {{ valid_files.append(1) }} 61 | {%- endif -%} 62 | {%- endfor -%} 63 | {%- else -%} 64 | {{ valid_files.append(1) }} 65 | {%- endif -%} 66 | 67 | {%- if valid_files -%} 68 | 69 | {%- for column in node.columns.values() -%} 70 | 71 | {%- set node_columns_list = [] -%} 72 | {{ node_columns_list.append(node.unique_id.split(".")[2]) }} 73 | {{ node_columns_list.append(node.resource_type) }} 74 | {{ node_columns_list.append(node.description) }} 75 | {{ node_columns_list.append(column.name) }} 76 | {{ node_columns_list.append(column.description) }} 77 | 78 | {{ rows_list.append(node_columns_list) }} 79 | 80 | {%- endfor -%} 81 | 82 | {%- endif -%} 83 | 84 | {%- endfor -%} 85 | 86 | {%- for source in graph.sources.values() if "source" in resource_type_list -%} 87 | 88 | {# "Check if source is in the provided files" #} 89 | {%- set valid_files = [] -%} 90 | {%- if files_list -%} 91 | {%- for file in files_list if re.match(file, source.original_file_path, re.IGNORECASE) -%} 92 | {%- if exclude_files_list -%} 93 | {%- for file_exclude in exclude_files_list if not re.match(file_exclude, source.original_file_path, re.IGNORECASE) -%} 94 | {{ valid_files.append(1) }} 95 | {%- endfor -%} 96 | {%- else -%} 97 | {{ valid_files.append(1) }} 98 | {%- endif -%} 99 | {%- endfor -%} 100 | {%- else -%} 101 | {{ valid_files.append(1) }} 102 | {%- endif -%} 103 | 104 | {%- if valid_files -%} 105 | 106 | {%- for column in source.columns.values() -%} 107 | 108 | {%- set source_columns_list = [] -%} 109 | {{ source_columns_list.append(source.unique_id.split(".")[2]) }} 110 | {{ source_columns_list.append(source.resource_type) }} 111 | {{ source_columns_list.append(source.description) }} 112 | {{ source_columns_list.append(column.name) }} 113 | {{ source_columns_list.append(column.description) }} 114 | 115 | {{ rows_list.append(source_columns_list) }} 116 | 117 | {%- endfor -%} 118 | 119 | {%- endif -%} 120 | 121 | {%- endfor -%} 122 | 123 | {{ return(rows_list) }} 124 | 125 | {%- endmacro -%} -------------------------------------------------------------------------------- /macros/create_metadata_model.sql: -------------------------------------------------------------------------------- 1 | {%- macro create_metadata_model( 2 | metadata 3 | , granularity=[] 4 | , resource_type=["model"] 5 | , show_resource_type=True 6 | , undefined="Undefined" 7 | , undefined_as_null=False 8 | , files=[] 9 | , exclude_files=[] 10 | ) -%} 11 | {%- if execute -%} 12 | 13 | {%- set rows_list = metalog.get_metadata(metadata, granularity, resource_type, undefined, files, exclude_files) -%} 14 | 15 | {%- if rows_list | length == 0 -%} 16 | {{ exceptions.raise_compiler_error("No metadata found for the provided parameters\nPlease check the metadata and resource type provided") }} 17 | {%- endif -%} 18 | 19 | {%- for row in rows_list -%} 20 | 21 | select "{{ row[0] }}" as resource_name 22 | 23 | {%- if show_resource_type -%} 24 | , "{{ row[1] }}" as resource_type 25 | {%- endif -%} 26 | 27 | {%- for i in range(metadata | length) -%} 28 | 29 | {%- if undefined_as_null and row[i+2] == undefined -%} 30 | , null as {{metadata[i]}} 31 | {%- else -%} 32 | , "{{ row[i+2] }}" as {{metadata[i]}} 33 | {%- endif -%} 34 | 35 | {%- endfor -%} 36 | 37 | {% if not loop.last %} 38 | union all 39 | {% endif %} 40 | 41 | {%- endfor -%} 42 | 43 | {%- endif -%} 44 | 45 | {%- endmacro -%} 46 | 47 | 48 | {%- macro get_metadata( 49 | metadata_list 50 | , granularity_list 51 | , resource_type_list 52 | , undefined 53 | , files_list 54 | , exclude_files_list 55 | ) -%} 56 | 57 | {% set re = modules.re %} 58 | 59 | {%- set rows_list = [] -%} 60 | 61 | {%- for node in graph.nodes.values() if node.resource_type in resource_type_list -%} 62 | 63 | {# "Check if node is in the provided files" #} 64 | {%- set valid_files = [] -%} 65 | {%- if files_list -%} 66 | {%- for file in files_list if re.match(file, node.original_file_path, re.IGNORECASE) -%} 67 | {%- if exclude_files_list -%} 68 | {%- for file_exclude in exclude_files_list if not re.match(file_exclude, node.original_file_path, re.IGNORECASE) -%} 69 | {{ valid_files.append(1) }} 70 | {%- endfor -%} 71 | {%- else -%} 72 | {{ valid_files.append(1) }} 73 | {%- endif -%} 74 | {%- endfor -%} 75 | {%- else -%} 76 | {{ valid_files.append(1) }} 77 | {%- endif -%} 78 | 79 | {%- if valid_files -%} 80 | 81 | {%- set granularity_values_list = [] -%} 82 | 83 | {%- for metadata in granularity_list -%} 84 | 85 | {%- set values_list = [] -%} 86 | 87 | {# "If the provided metadata in granularity is a string" #} 88 | {# "just append the string into values_list" #} 89 | {%- if node.meta[metadata] is string -%} 90 | {{ values_list.append(node.meta[metadata]) }} 91 | 92 | {# "If the provided metadata in granularity is a list" #} 93 | {# "then append each value into values_list" #} 94 | {%- else -%} 95 | {%- for item in node.meta[metadata] -%} 96 | {{ values_list.append(item) }} 97 | {%- endfor -%} 98 | 99 | {%- endif -%} 100 | 101 | {# "If the model has no metadata from the granularity list" #} 102 | {# "append the undefined argument string" #} 103 | {%- if values_list == [] -%} 104 | {%- set values_list = [undefined] -%} 105 | {%- endif -%} 106 | 107 | {{ granularity_values_list.append(values_list) }} 108 | 109 | {%- endfor -%} 110 | 111 | {%- set all_combinations = metalog.combinations(granularity_values_list) -%} 112 | 113 | {# "The if block below is used to get a length of 1 for all_combinations 114 | if there is none combination" #} 115 | {%- if all_combinations == [] -%} 116 | {%- set all_combinations = [[]] -%} 117 | {%- endif -%} 118 | 119 | {%- for c in range(all_combinations | length) -%} 120 | 121 | {%- set node_row = [] -%} 122 | 123 | {%- set unique_id_splitted = node.unique_id.split(".") -%} 124 | 125 | {{ node_row.append(unique_id_splitted[2]) }} 126 | {{ node_row.append(node.resource_type) }} 127 | 128 | {%- for metadata in metadata_list -%} 129 | 130 | {%- if not node.meta[metadata] or not node.meta -%} 131 | {{ node_row.append(undefined) }} 132 | {%- else -%} 133 | 134 | {%- if metadata not in granularity_list -%} 135 | {{ node_row.append(node.meta[metadata] | string) }} 136 | {%- else -%} 137 | {%- set idx = granularity_list.index(metadata) -%} 138 | {{ node_row.append(all_combinations[c][idx] | string) }} 139 | {%- endif -%} 140 | 141 | {%- endif -%} 142 | 143 | {%- endfor -%} 144 | 145 | {{ rows_list.append(node_row) }} 146 | 147 | {%- endfor -%} 148 | 149 | {%- endif -%} 150 | 151 | {%- endfor -%} 152 | 153 | {%- for source in graph.sources.values() if source.resource_type in resource_type_list -%} 154 | 155 | {# "Check if source is in the provided files" #} 156 | {%- set valid_files = [] -%} 157 | {%- if files_list -%} 158 | {%- for file in files_list if re.match(file, source.original_file_path, re.IGNORECASE) -%} 159 | {%- if exclude_files_list -%} 160 | {%- for file_exclude in exclude_files_list if not re.match(file_exclude, source.original_file_path, re.IGNORECASE) -%} 161 | {{ valid_files.append(1) }} 162 | {%- endfor -%} 163 | {%- else -%} 164 | {{ valid_files.append(1) }} 165 | {%- endif -%} 166 | {%- endfor -%} 167 | {%- else -%} 168 | {{ valid_files.append(1) }} 169 | {%- endif -%} 170 | 171 | {%- if valid_files -%} 172 | 173 | {%- set granularity_values_list = [] -%} 174 | 175 | {%- for metadata in granularity_list -%} 176 | 177 | {%- set values_list = [] -%} 178 | 179 | {# "If the provided metadata in granularity is a string" #} 180 | {# "just append the string into values_list" #} 181 | {%- if source.meta[metadata] is string -%} 182 | {{ values_list.append(source.meta[metadata]) }} 183 | 184 | {# "If the provided metadata in granularity is a list" #} 185 | {# "then append each value into values_list" #} 186 | {%- else -%} 187 | {%- for item in source.meta[metadata] -%} 188 | {{ values_list.append(item) }} 189 | {%- endfor -%} 190 | 191 | {%- endif -%} 192 | 193 | {# "If the model has no metadata from the granularity list" #} 194 | {# "append the undefined argument string" #} 195 | {%- if values_list == [] -%} 196 | {%- set values_list = [undefined] -%} 197 | {%- endif -%} 198 | 199 | {{ granularity_values_list.append(values_list) }} 200 | 201 | {%- endfor -%} 202 | 203 | {%- set all_combinations = metalog.combinations(granularity_values_list) -%} 204 | 205 | {# "The if block below is used to get a length of 1 for all_combinations 206 | if there is none combination" #} 207 | {%- if all_combinations == [] -%} 208 | {%- set all_combinations = [[]] -%} 209 | {%- endif -%} 210 | 211 | {%- for c in range(all_combinations | length) -%} 212 | 213 | {%- set source_row = [] -%} 214 | 215 | {%- set unique_id_splitted = source.unique_id.split(".") -%} 216 | 217 | {{ source_row.append(unique_id_splitted[2]) }} 218 | {{ source_row.append(source.resource_type) }} 219 | 220 | {%- for metadata in metadata_list -%} 221 | 222 | {%- if not source.meta[metadata] or not source.meta -%} 223 | {{ source_row.append(undefined) }} 224 | {%- else -%} 225 | 226 | {%- if metadata not in granularity_list -%} 227 | {{ source_row.append(source.meta[metadata] | string) }} 228 | {%- else -%} 229 | {%- set idx = granularity_list.index(metadata) -%} 230 | {{ source_row.append(all_combinations[c][idx] | string) }} 231 | {%- endif -%} 232 | 233 | {%- endif -%} 234 | 235 | {%- endfor -%} 236 | 237 | {{ rows_list.append(source_row) }} 238 | 239 | {%- endfor -%} 240 | 241 | {%- endif -%} 242 | 243 | {%- endfor -%} 244 | 245 | {{ return(rows_list) }} 246 | 247 | {%- endmacro -%} -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # :notebook: dbt-metalog: Your metadata's catalog 2 | 3 | Have you ever found yourself jotting down essential business rules, questions, technical owners, or ToDos in a **separate Excel sheet  - far from your code -, only to forget about them later?** 4 | 5 | Do you **struggle to keep track of all the metadata** that's critical for effective data management and analysis? 6 | 7 | Worry no more! We have a solution - dbt-metalog. 8 | 9 | **Create lightweight and totally customizable models from your metadata.** 10 | 11 | **Easily create models for:** 12 | 13 | :white_check_mark: Business rules. 14 | 15 | :white_check_mark: Business questions. 16 | 17 | :white_check_mark: Tech owners. 18 | 19 | :white_check_mark: Requesting areas/persons. 20 | 21 | :white_check_mark: Date the model was created. 22 | 23 | :white_check_mark: ToDo's. 24 | 25 | :white_check_mark: Any metadata you want... 26 | 27 | **Choose your metadata by:** 28 | 29 | :white_check_mark: name. 30 | 31 | :white_check_mark: resource type. 32 | 33 | :white_check_mark: file. 34 | 35 | # :mag_right: Content 36 | * :running: [Quickstart](#running-quickstart) 37 | * * [Requirements](#requirements) 38 | * * [Installation](#installation) 39 | * * [Package Limitations](#package-limitations) 40 | * :gear: [Macros](#gear-macros) 41 | * * [{{ create_metadata_model( ) }}](#create_metadata_model-source) 42 | * * [{{ create_description_model( ) }}](#create_description_model-source) 43 | * :wrench: [Troubleshooting](#wrench-troubleshooting) 44 | * :writing_hand: [ToDos](#writing_hand-todos) 45 | 46 | 47 | 48 | # :running: Quickstart 49 | 50 | New to dbt packages? Read more about them [here](https://docs.getdbt.com/docs/building-a-dbt-project/package-management/). 51 | 52 | ## Requirements 53 | dbt version 54 | * ```dbt version >= 1.0.0``` 55 | 56 | ## Installation 57 | 58 | 1. Include this package in your `packages.yml` file. 59 | ```yaml 60 | packages: 61 | - package: techindicium/metalog 62 | version: 1.0.0 63 | ``` 64 | 65 | 2. Run `dbt deps` to install the package. 66 | 67 | ## Package Limitations 68 | > **Warning** If your project is too large (too many models with too many metadata), there is a chance the generated SQL by the macros exceed the query length limit of your DW. Then you will get an error. 69 | 70 | 71 | 72 | 73 | # :gear: Macros 74 | ## create_metadata_model ([source](macros/create_metadata_model.sql)) 75 | 76 | This macro generates SQL for creating **customizable tables or views from the [metadata](https://docs.getdbt.com/reference/resource-configs/meta) of your nodes and sources**. You have the **flexibility to select the specific metadata** you want to include in your table or view. If a node **does not contain the specified metadata, it will be displayed as "Undefined"**, but you can alter this default text to your preference. 77 | 78 | > **Note** For this README, every time you see "node" consider also "sources" 79 | 80 | The macro will check the metadata defined in your nodes. If you are new to metadata in dbt, check the documentation [here](https://docs.getdbt.com/reference/configs-and-properties). 81 | 82 | Nodes can be: 83 | * Models 84 | * Sources 85 | * Seeds 86 | * Snapshots 87 | * Tests 88 | * Analyses 89 | * Macros 90 | 91 | For a model, you can define the ```meta```config in 92 | * Your model, inside the config block. 93 | * A config property, in a ```.yml file``` 94 | * The ```dbt_project.yml``` under configs under models. 95 | 96 | For others resource types, [check the docs](https://docs.getdbt.com/reference/resource-configs/meta). 97 | 98 | 99 | > **Warning**: **Currently this package does not supports dicts in the meta config, just single values or lists.** 100 | 101 | 102 | ### Arguments 103 | - ```metadata``` (required): A ```list``` of the metadata which will be the columns of your model. 104 | - ```granularity``` (optional) (default = ```[]```): A ```list``` of th metadata which must be separated in different rows. They must be wrote in the meta config as lists. 105 | - ```resource_type``` (optional) (default = ```['model']```): A ```list``` of the resource types you want to read the metadata from. Options: 106 | - model 107 | - source 108 | - seed 109 | - snapshot 110 | - tests 111 | - analysis 112 | - macros 113 | - ```undefined``` (optional) (default = ```'Undefined'```): A ```string``` which overrides the default string shown when the metadata is not found for that model. 114 | - ```undefined_as_null``` (optional) (default = ```'False'```): A ```booelan```, when True undefined metadata will be displayed as null. 115 | - ```show_resource_type```(optional) (default = ```True```): A ```boolean``` to show or hide the ```resource_type``` column in your resulting model. 116 | - ```files```(optional) (default = []): A ```list``` of regex specifying the files to include, e.g. If you want to include all files in models, then ```files=['models/.*']```. 117 | - ```exclude_files```(optional) (default = []): A ```list``` of regex specifying the files to exclude, e.g. If you want to exclude all staging files, then ```files=['.*stg_.*']```. 118 | 119 | ### Usage 120 | 121 | #### Define the metadata in your nodes 122 | So, for example take a look at the [dummy_model_1](https://github.com/techindicium/dbt-metalog/blob/main/integration_tests/models/dummy_models/dummy_model_1.sql) inside the ```integration_tests``` folder 123 | 124 | ```sql 125 | {{ config( 126 | meta={ 127 | 'main_subject': 'sales' 128 | , 'owner': 'Alice' 129 | , 'business_questions': [ 130 | 'How many stores of type ...?' 131 | , 'How many stores in ...?' 132 | ] 133 | , 'business_rules': [ 134 | 'Stores of type A receive code B ...' 135 | , 'Consider only stores open after ...' 136 | ] 137 | , 'todos': 'change to incremental' 138 | } 139 | )}} 140 | 141 | select 1 as dummy 142 | ``` 143 | 144 | #### Create a model which uses the ```create_metadata_model``` macro. 145 | 146 | Use the ```create_metadata_model``` macro passing as argument a list of the metadata you want to include in your model. Let's see the [01_metadata_test](https://github.com/techindicium/dbt-metalog/blob/main/integration_tests/models/metadata_tests/01_metadata_test.sql) model (You can create a model with any name you want) as an example: 147 | ```sql 148 | {{ metalog.create_metadata_model( 149 | metadata = [ 150 | "main_subject" 151 | , "owner" 152 | , "business_questions" 153 | , "business_rules" 154 | , "todos" 155 | ] 156 | )}} 157 | ``` 158 | 159 | > **Note**: **The default materialization for dbt models is view. If you want to change to table, change the [```materialized``` configuration property ](https://docs.getdbt.com/docs/build/materializations).** 160 | 161 | #### Run your model 162 | Just run it! 163 | ```shell 164 | dbt run -s metadata_view 165 | ``` 166 | 167 | > **Note** Suppose we have the following nodes in our project: [dummy_model_1](https://github.com/techindicium/dbt-metalog/blob/main/integration_tests/models/dummy_models/dummy_model_1.sql), [dummy_model_2](https://github.com/techindicium/dbt-metalog/blob/main/integration_tests/models/dummy_models/dummy_model_2.sql) and [dummy_seed](https://github.com/techindicium/dbt-metalog/blob/main/integration_tests/seeds/dummy_seed.csv) 168 | 169 | 170 | The output view, using the meta defined in our nodes will be: 171 | 172 | | **resource_name** | **resource_type** | **main_subject** | **owner** | **business_questions** | **business_rules** | **todos** | 173 | |-------------------|-------------------|------------------|-----------|---------------------------------------------------------------|----------------------------------------------------------------------------------|-----------------------| 174 | | dummy_model_1 | model | sales | Alice | "['How many stores of type ...?', 'How many stores in ...?']" | "['Stores of type A receive code B ...', 'Consider only stores open after ...']" | change to incremental | 175 | | dummy_model_2 | model | people | Bob | How many employees ...? | Undefined | Undefined | 176 | 177 | 178 | 179 | ## Additional customization 180 | 181 | ### granularity 182 | 183 | **If you want to break a metadata into different rows, you can use the ```granularity_list``` argument. For example, if you want to break the ```business_questions``` meta into different lines, change your model to:** 184 | 185 | ```sql 186 | {{ metalog.create_metadata_model( 187 | metadata = [ 188 | "main_subject" 189 | , "owner" 190 | , "business_questions" 191 | , "business_rules" 192 | , "todos" 193 | ] 194 | , granularity = [ 195 | "business_questions" 196 | ] 197 | )}} 198 | ``` 199 | 200 | Now you have only a business question per row. 201 | 202 | | **resource_name** | **resource_type** | **main_subject** | **owner** | **business_questions** | **business_rules** | **todos** | 203 | |-------------------|-------------------|------------------|-----------|------------------------------|----------------------------------------------------------------------------------|-----------------------| 204 | | dummy_model_1 | model | sales | Alice | How many stores of type ...? | "['Stores of type A receive code B ...', 'Consider only stores open after ...']" | change to incremental | 205 | | dummy_model_1 | model | sales | Alice | How many stores in ...? | "['Stores of type A receive code B ...', 'Consider only stores open after ...']" | change to incremental | 206 | | dummy_model_2 | model | people | Bob | How many employees ...? | Undefined | Undefined | 207 | 208 | 209 | 210 | You can pass more than one metadata to the ```granularity_list```, for example: 211 | 212 | [02_metadata_test_with_granularity](https://github.com/techindicium/dbt-metalog/blob/main/integration_tests/models/metadata_tests/02_metadata_test_with_granularity.sql) 213 | ```sql 214 | {{ metalog.create_metadata_model( 215 | metadata = [ 216 | "main_subject" 217 | , "owner" 218 | , "business_questions" 219 | , "business_rules" 220 | , "todos" 221 | ] 222 | , granularity = [ 223 | "business_questions" 224 | , "business_rules" 225 | , "todos" 226 | ] 227 | )}} 228 | ``` 229 | 230 | Now each row have a unique business question, business rule and a todo. 231 | 232 | | **resource_name** | **resource_type** | **main_subject** | **owner** | **business_questions** | **business_rules** | **todos** | 233 | |-------------------|-------------------|------------------|-----------|------------------------------|-------------------------------------|-----------------------| 234 | | dummy_model_1 | model | sales | Alice | How many stores of type ...? | Stores of type A receive code B ... | change to incremental | 235 | | dummy_model_1 | model | sales | Alice | How many stores of type ...? | Consider only stores open after ... | change to incremental | 236 | | dummy_model_1 | model | sales | Alice | How many stores in ...? | Stores of type A receive code B ... | change to incremental | 237 | | dummy_model_1 | model | sales | Alice | How many stores in ...? | Consider only stores open after ... | change to incremental | 238 | | dummy_model_2 | model | people | Bob | How many employees ...? | Undefined | Undefined | 239 | 240 | 241 | 242 | ### resource_type 243 | You can ask the macro to include metadata of more resource types with the ```resource_type```argument. Let's include ```seeds``` along with models. 244 | 245 | [03_metadata_test_resource_type](https://github.com/techindicium/dbt-metalog/blob/main/integration_tests/models/metadata_tests/03_metadata_test_resource_type.sql) 246 | ```sql 247 | {{ metalog.create_metadata_model( 248 | metadata = [ 249 | "main_subject" 250 | , "owner" 251 | , "business_questions" 252 | , "business_rules" 253 | , "todos" 254 | ] 255 | , granularity = [ 256 | "business_questions" 257 | , "business_rules" 258 | , "todos" 259 | ] 260 | , resource_type = [ 261 | "model" 262 | , "seed" 263 | ] 264 | )}} 265 | ``` 266 | 267 | Now you can see also the metadata from the ```dummy_seed```. 268 | | **resource_name** | **resource_type** | **main_subject** | **owner** | **business_questions** | **business_rules** | **todos** | 269 | |-------------------|-------------------|------------------|-----------|------------------------------|-------------------------------------|-----------------------| 270 | | dummy_model_1 | model | sales | Alice | How many stores of type ...? | Stores of type A receive code B ... | change to incremental | 271 | | dummy_model_1 | model | sales | Alice | How many stores of type ...? | Consider only stores open after ... | change to incremental | 272 | | dummy_model_1 | model | sales | Alice | How many stores in ...? | Stores of type A receive code B ... | change to incremental | 273 | | dummy_model_1 | model | sales | Alice | How many stores in ...? | Consider only stores open after ... | change to incremental | 274 | | dummy_model_2 | model | people | Bob | How many employees ...? | Undefined | Undefined | 275 | | dummy_seed | seed | sales | Carl | Undefined | Undefined | Undefined | 276 | 277 | 278 | ### show_resource_type 279 | 280 | You can hide the ```resource_type``` column passing the argument ```show_resource_type=False``` 281 | 282 | [04_metadata_test_show_resource_type](https://github.com/techindicium/dbt-metalog/blob/main/integration_tests/models/metadata_tests/04_metadata_test_show_resource_type.sql) 283 | ```sql 284 | {{ metalog.create_metadata_model( 285 | metadata = [ 286 | "main_subject" 287 | , "owner" 288 | , "business_questions" 289 | , "business_rules" 290 | , "todos" 291 | ] 292 | , granularity = [ 293 | "business_questions" 294 | , "business_rules" 295 | , "todos" 296 | ] 297 | , resource_type = [ 298 | "model" 299 | , "seed" 300 | ] 301 | , show_resource_type = False 302 | )}} 303 | ``` 304 | 305 | The ```resource_type``` column was removed 306 | 307 | | **resource_name** | **main_subject** | **owner** | **business_questions** | **business_rules** | **todos** | 308 | |-------------------|------------------|-----------|------------------------------|-------------------------------------|-----------------------| 309 | | dummy_model_1 | sales | Alice | How many stores of type ...? | Stores of type A receive code B ... | change to incremental | 310 | | dummy_model_1 | sales | Alice | How many stores of type ...? | Consider only stores open after ... | change to incremental | 311 | | dummy_model_1 | sales | Alice | How many stores in ...? | Stores of type A receive code B ... | change to incremental | 312 | | dummy_model_1 | sales | Alice | How many stores in ...? | Consider only stores open after ... | change to incremental | 313 | | dummy_model_2 | people | Bob | How many employees ...? | Undefined | Undefined | 314 | | dummy_seed | sales | Carl | Undefined | Undefined | Undefined | 315 | 316 | 317 | ### undefined 318 | You can override the default 'Undefined' string with ```undefined```. 319 | 320 | [05_metadata_test_undefined](https://github.com/techindicium/dbt-metalog/blob/main/integration_tests/models/metadata_tests/05_metadata_test_undefined.sql) 321 | ```sql 322 | {{ metalog.create_metadata_model( 323 | metadata = [ 324 | "main_subject" 325 | , "owner" 326 | , "business_questions" 327 | , "business_rules" 328 | , "todos" 329 | ] 330 | , granularity = [ 331 | "business_questions" 332 | , "business_rules" 333 | , "todos" 334 | ] 335 | , resource_type = [ 336 | "model" 337 | , "seed" 338 | ] 339 | , show_resource_type = True 340 | , undefined = "Not defined" 341 | )}} 342 | ``` 343 | 344 | The undefined metadata are displayed as 'Not defined'. 345 | | **resource_name** | **resource_type** | **main_subject** | **owner** | **business_questions** | **business_rules** | **todos** | 346 | |-------------------|-------------------|------------------|-----------|------------------------------|-------------------------------------|-----------------------| 347 | | dummy_model_1 | model | sales | Alice | How many stores of type ...? | Stores of type A receive code B ... | change to incremental | 348 | | dummy_model_1 | model | sales | Alice | How many stores of type ...? | Consider only stores open after ... | change to incremental | 349 | | dummy_model_1 | model | sales | Alice | How many stores in ...? | Stores of type A receive code B ... | change to incremental | 350 | | dummy_model_1 | model | sales | Alice | How many stores in ...? | Consider only stores open after ... | change to incremental | 351 | | dummy_model_2 | model | people | Bob | How many employees ...? | Not defined | Not defined | 352 | | dummy_seed | seed | sales | Carl | Not defined | Not defined | Not defined | 353 | 354 | 355 | ### undefined_as_null 356 | You can also set the undefined metadata to appear as null values with ```undefined_as_null``` 357 | 358 | [06_metadata_test_undefined_as_null](https://github.com/techindicium/dbt-metalog/blob/main/integration_tests/models/metadata_tests/06_metadata_test_undefined_as_null.sql) 359 | ```sql 360 | {{ metalog.create_metadata_model( 361 | metadata = [ 362 | "main_subject" 363 | , "owner" 364 | , "business_questions" 365 | , "business_rules" 366 | , "todos" 367 | ] 368 | , granularity = [ 369 | "business_questions" 370 | , "business_rules" 371 | , "todos" 372 | ] 373 | , resource_type = [ 374 | "model" 375 | , "seed" 376 | ] 377 | , show_resource_type = True 378 | , undefined = "Not defined" 379 | , undefined_as_null = True 380 | )}} 381 | ``` 382 | 383 | The undefined metadata are displayed as null. 384 | 385 | | **resource_name** | **resource_type** | **main_subject** | **owner** | **business_questions** | **business_rules** | **todos** | 386 | |-------------------|-------------------|------------------|-----------|------------------------------|-------------------------------------|-----------------------| 387 | | dummy_model_1 | model | sales | Alice | How many stores of type ...? | Stores of type A receive code B ... | change to incremental | 388 | | dummy_model_1 | model | sales | Alice | How many stores of type ...? | Consider only stores open after ... | change to incremental | 389 | | dummy_model_1 | model | sales | Alice | How many stores in ...? | Stores of type A receive code B ... | change to incremental | 390 | | dummy_model_1 | model | sales | Alice | How many stores in ...? | Consider only stores open after ... | change to incremental | 391 | | dummy_model_2 | model | people | Bob | How many employees ...? | | | 392 | | dummy_seed | seed | sales | Carl | | | | 393 | 394 | 395 | ### files 396 | You can select the files you want to include (```files```) or to exclude (```exclude_files```) 397 | 398 | [07_metadata_test_files](https://github.com/techindicium/dbt-metalog/blob/main/integration_tests/models/metadata_tests/07_metadata_test_files.sql) 399 | ```sql 400 | {{ metalog.create_metadata_model( 401 | metadata = [ 402 | "main_subject" 403 | , "owner" 404 | , "business_questions" 405 | , "business_rules" 406 | , "todos" 407 | ] 408 | , granularity = [ 409 | "business_questions" 410 | , "business_rules" 411 | , "todos" 412 | ] 413 | , resource_type = [ 414 | "model" 415 | , "seed" 416 | , "source" 417 | ] 418 | , show_resource_type = True 419 | , undefined = "Not defined" 420 | , undefined_as_null = True 421 | , files = [ 422 | "models/.*" 423 | , "seeds/.*" 424 | , ".*/source.yml" 425 | ] 426 | , exclude_files = [ 427 | 'models/metadata_tests/.*' 428 | 'models/description_tests/.*' 429 | ] 430 | )}} 431 | ``` 432 | | **resource_name** | **resource_type** | **main_subject** | **owner** | **business_questions** | **business_rules** | **todos** | 433 | |-------------------|-------------------|------------------|-----------|------------------------------|-------------------------------------|-----------------------| 434 | | dummy_model_1 | model | sales | Alice | How many stores of type ...? | Stores of type A receive code B ... | change to incremental | 435 | | dummy_model_1 | model | sales | Alice | How many stores of type ...? | Consider only stores open after ... | change to incremental | 436 | | dummy_model_1 | model | sales | Alice | How many stores in ...? | Stores of type A receive code B ... | change to incremental | 437 | | dummy_model_1 | model | sales | Alice | How many stores in ...? | Consider only stores open after ... | change to incremental | 438 | | dummy_model_2 | model | people | Bob | How many employees ...? | | | 439 | | dummy_seed | seed | sales | Carl | | | | 440 | | raw | source | | | | | | 441 | 442 | 443 | ## create_description_model ([source](macros/create_description_model.sql)) 444 | 445 | This macro generates SQL for creating **tables or views from the description of your nodes and its columns**. You have the **flexibility to select the specific files** you want to include in your table or view. 446 | 447 | The macro will check the description defined in your nodes. 448 | 449 | Nodes can be: 450 | * Models 451 | * Sources 452 | * Seeds 453 | * Snapshots 454 | * Tests 455 | * Analyses 456 | * Macros 457 | 458 | ### Arguments 459 | - ```resource_type``` (optional) (default = ```['model']```): A ```list``` of the resource types you want to read the metadata from. Options: 460 | - model 461 | - source 462 | - seed 463 | - snapshot 464 | - tests 465 | - analysis 466 | - macros 467 | - ```show_resource_type```(optional) (default = ```True```): A ```boolean``` to show or hide the ```resource_type``` column in your resulting model. 468 | - ```files```(optional) (default = []): A ```list``` of regex specifying the files to include, e.g. If you want to include all files in models, then ```files=['models/.*']```. 469 | - ```exclude_files```(optional) (default = []): A ```list``` of regex specifying the files to exclude, e.g. If you want to exclude all staging files, then ```files=['.*stg_.*']```. 470 | 471 | ### Usage 472 | 473 | #### Define the description of your nodes 474 | So, for example take a look at the [dummy_model_2.yml](https://github.com/techindicium/dbt-metalog/blob/main/integration_tests/models/dummy_models/dummy_model_2.yml) inside the ```integration_tests``` folder 475 | 476 | ```yaml 477 | version: 2 478 | 479 | models: 480 | - name: 'dummy_model_2' 481 | description: "description of dummy_model_2" 482 | 483 | columns: 484 | - name: "dummy" 485 | description: "the description of the dummy column of dummy_model_2" 486 | ``` 487 | 488 | #### Create a model which uses the ```create_description_model``` macro. 489 | 490 | Use the ```create_description_model``` macro passing the arguments of your choice 491 | 492 | [description_test_files](https://github.com/techindicium/dbt-metalog/blob/main/integration_tests/models/description_tests/description_test_files.sql) 493 | ```sql 494 | {{ metalog.create_description_model( 495 | resource_type = [ 496 | "model" 497 | , "seed" 498 | , "source" 499 | ] 500 | , show_resource_type = True 501 | , files = [ 502 | '.*.sql' 503 | ] 504 | , exclude_files = [ 505 | 'models/metadata.*' 506 | ] 507 | )}} 508 | ``` 509 | 510 | > **Note**: **The default materialization for dbt models is view. If you want to change to table, change the [```materialized``` configuration property ](https://docs.getdbt.com/docs/build/materializations).** 511 | 512 | #### Run your model 513 | Just run it! 514 | ```shell 515 | dbt run -s metadata_view 516 | ``` 517 | 518 | > **Note** Suppose we have the following nodes in our project: [dummy_model_1](https://github.com/techindicium/dbt-metalog/blob/main/integration_tests/models/dummy_models/dummy_model_1.sql), [dummy_model_2](https://github.com/techindicium/dbt-metalog/blob/main/integration_tests/models/dummy_models/dummy_model_2.sql) and [dummy_seed](https://github.com/techindicium/dbt-metalog/blob/main/integration_tests/seeds/dummy_seed.csv) 519 | 520 | 521 | The output view, using the meta defined in our nodes will be: 522 | 523 | | **resource_name** | **resource_type** | **resource_description** | **column_name** | **columns_description** | 524 | |-------------------|-------------------|------------------------------|-----------------|------------------------------------------------------| 525 | | dummy_model_1 | model | | dummy | the description of the dummy column of dummy_model_1 | 526 | | dummy_model_2 | model | description of dummy_model_2 | dummy | the description of the dummy column of dummy_model_2 | 527 | 528 | 529 | # :wrench: Troubleshooting 530 | ### Unclosed string literal 531 | ``` 532 | Database Error in model description_model (models/metadata_catalog/description_model.sql) 533 | Syntax error: Unclosed string literal at [2473:196] 534 | compiled Code at target/run/marketing_mas/models/metadata_catalog/description_model.sql 535 | ``` 536 | Check if there is a double quote in any of your descriptions. If so, remove it or replace by single quotes. 537 | 538 | ### The query is too large 539 | ``` 540 | Database Error in model description models_ view (models/metadata catalog/description models_ view.sql) 541 | The query is too large. The maximum standard SQL query length is 1024.00K characters, including comments and white space characters. 542 | compiled Code at target/run/marketing_mas/models/metadata_catalog/description_models_view.sql 543 | ``` 544 | It is a current limitation of the package. As it passes the metadata/descriptions to the SQL query, if there is a massive number of metadata/descriptions there is a chance the query exceeds the limits of your DW. 545 | 546 | # :writing_hand: ToDos 547 | * Implement CI 548 | * Create PR template 549 | * Workaround the query limit problem 550 | --------------------------------------------------------------------------------