├── README.md ├── data ├── .gitkeep ├── raw_customers.csv ├── raw_orders.csv └── raw_payments.csv ├── dbt_project.yml ├── lightdash-dockerfile ├── lightdash-entrypoint.sh ├── models ├── customers.sql ├── docs.md ├── orders.sql ├── overview.md ├── payments.sql ├── schema.yml └── staging │ ├── schema.yml │ ├── stg_customers.sql │ ├── stg_orders.sql │ └── stg_payments.sql ├── profiles └── profiles.yml └── screenshots ├── .DS_Store ├── 1.png ├── 2.png ├── 3.png ├── 4.png ├── 5.png ├── 6.png ├── 7.png └── 8.png /README.md: -------------------------------------------------------------------------------- 1 | # Deploy a Lightdash project with GCP Cloud Run 2 | 3 | This repo shows you how to deploy your dbt project with Lightdash to GCP Cloud Run. 4 | 5 | ## 1. Add three extra files to your dbt repo 6 | 7 | This directory is a standard dbt project repo with 3 extra files: 8 | - `lightdash-dockerfile`: you need to create your own simple dockerfile to deploy Lightdash with your dbt project 9 | - `lightdash-entrypoint.sh`: (optional) - an optional script if you need to run any dbt commands before deploying lightdash 10 | - `profiles/profiles.yml`: credentials to access your data warehouse 11 | 12 | ``` 13 | . 14 | ├── data/ 15 | ├── dbt_project.yml 16 | ├── lightdash-dockerfile 17 | ├── lightdash-entrypoint.sh 18 | ├── models/ 19 | └── profiles 20 | └── profiles.yml 21 | ``` 22 | 23 | ## 2. Configure `profiles.yml` 24 | 25 | Your `profiles.yml` should contain a profile matching the profile in your `dbt_project.yml`. Do not put secrets in here 26 | instead use the `env_var` function. You can see an example in this repo. 27 | 28 | **Bigquery tip** - by default cloud run can access bigquery, so you don't need to pass any credentials (see the `./profiles.yml` in this repo) 29 | 30 | ## 3. Configure Cloud Run to deploy from your repo 31 | 32 | Now you can use this repo to launch a Cloud Run container using Docker. The steps below show you how to do this with the GCP UI using the github integration. 33 | 34 | ### 3.1 Create a new service 35 | 36 | [https://console.cloud.google.com/run](https://console.cloud.google.com/run) 37 | 38 | ![s1](screenshots/1.png) 39 | ![s2](screenshots/2.png) 40 | 41 | ### 3.2 Link to github repo 42 | 43 | Select "Deploy continuously... from a source repository" and point it to your github repo. Select your branch and the name of the docker file (in our case it's `lightdash-dockerfile` 44 | 45 | ![s3](screenshots/3.png) 46 | ![s4](screenshots/4.png) 47 | 48 | ### 3.3 Update container configuration 49 | 50 | Hit advanced settings and check the port is set to `8080`. Also set minimum instances to 1: Lightdash is slow to startup so it's better to keep it live. 51 | 52 | ![s5](screenshots/5.png) 53 | ![s6](screenshots/6.png) 54 | 55 | ### 3.4 Update secrets 56 | 57 | Set environment variables to populate all the values of your `profiles.yml` file that use the `env_var` function. 58 | 59 | In our example we don't consider these secrets so we add them under "environment variables". However, if you need to pass database passwords or other secrets to your `profiles.yml` file, then you can use the "secrets" functionality. 60 | 61 | ![s7](screenshots/7.png) 62 | 63 | ### 3.5 Setup network access 64 | 65 | In our example we just make it public. You may want to run it in a VPC, like your internal company network. 66 | 67 | ![s8](screenshots/8.png) 68 | -------------------------------------------------------------------------------- /data/.gitkeep: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lightdash/lightdash-production-example/de672eca9890d15d186258ac78006282af410b40/data/.gitkeep -------------------------------------------------------------------------------- /data/raw_customers.csv: -------------------------------------------------------------------------------- 1 | id,first_name,last_name 2 | 1,Michael,P. 3 | 2,Shawn,M. 4 | 3,Kathleen,P. 5 | 4,Jimmy,C. 6 | 5,Katherine,R. 7 | 6,Sarah,R. 8 | 7,Martin,M. 9 | 8,Frank,R. 10 | 9,Jennifer,F. 11 | 10,Henry,W. 12 | 11,Fred,S. 13 | 12,Amy,D. 14 | 13,Kathleen,M. 15 | 14,Steve,F. 16 | 15,Teresa,H. 17 | 16,Amanda,H. 18 | 17,Kimberly,R. 19 | 18,Johnny,K. 20 | 19,Virginia,F. 21 | 20,Anna,A. 22 | 21,Willie,H. 23 | 22,Sean,H. 24 | 23,Mildred,A. 25 | 24,David,G. 26 | 25,Victor,H. 27 | 26,Aaron,R. 28 | 27,Benjamin,B. 29 | 28,Lisa,W. 30 | 29,Benjamin,K. 31 | 30,Christina,W. 32 | 31,Jane,G. 33 | 32,Thomas,O. 34 | 33,Katherine,M. 35 | 34,Jennifer,S. 36 | 35,Sara,T. 37 | 36,Harold,O. 38 | 37,Shirley,J. 39 | 38,Dennis,J. 40 | 39,Louise,W. 41 | 40,Maria,A. 42 | 41,Gloria,C. 43 | 42,Diana,S. 44 | 43,Kelly,N. 45 | 44,Jane,R. 46 | 45,Scott,B. 47 | 46,Norma,C. 48 | 47,Marie,P. 49 | 48,Lillian,C. 50 | 49,Judy,N. 51 | 50,Billy,L. 52 | 51,Howard,R. 53 | 52,Laura,F. 54 | 53,Anne,B. 55 | 54,Rose,M. 56 | 55,Nicholas,R. 57 | 56,Joshua,K. 58 | 57,Paul,W. 59 | 58,Kathryn,K. 60 | 59,Adam,A. 61 | 60,Norma,W. 62 | 61,Timothy,R. 63 | 62,Elizabeth,P. 64 | 63,Edward,G. 65 | 64,David,C. 66 | 65,Brenda,W. 67 | 66,Adam,W. 68 | 67,Michael,H. 69 | 68,Jesse,E. 70 | 69,Janet,P. 71 | 70,Helen,F. 72 | 71,Gerald,C. 73 | 72,Kathryn,O. 74 | 73,Alan,B. 75 | 74,Harry,A. 76 | 75,Andrea,H. 77 | 76,Barbara,W. 78 | 77,Anne,W. 79 | 78,Harry,H. 80 | 79,Jack,R. 81 | 80,Phillip,H. 82 | 81,Shirley,H. 83 | 82,Arthur,D. 84 | 83,Virginia,R. 85 | 84,Christina,R. 86 | 85,Theresa,M. 87 | 86,Jason,C. 88 | 87,Phillip,B. 89 | 88,Adam,T. 90 | 89,Margaret,J. 91 | 90,Paul,P. 92 | 91,Todd,W. 93 | 92,Willie,O. 94 | 93,Frances,R. 95 | 94,Gregory,H. 96 | 95,Lisa,P. 97 | 96,Jacqueline,A. 98 | 97,Shirley,D. 99 | 98,Nicole,M. 100 | 99,Mary,G. 101 | 100,Jean,M. 102 | -------------------------------------------------------------------------------- /data/raw_orders.csv: -------------------------------------------------------------------------------- 1 | id,user_id,order_date,status 2 | 1,1,2018-01-01,returned 3 | 2,3,2018-01-02,completed 4 | 3,94,2018-01-04,completed 5 | 4,50,2018-01-05,completed 6 | 5,64,2018-01-05,completed 7 | 6,54,2018-01-07,completed 8 | 7,88,2018-01-09,completed 9 | 8,2,2018-01-11,returned 10 | 9,53,2018-01-12,completed 11 | 10,7,2018-01-14,completed 12 | 11,99,2018-01-14,completed 13 | 12,59,2018-01-15,completed 14 | 13,84,2018-01-17,completed 15 | 14,40,2018-01-17,returned 16 | 15,25,2018-01-17,completed 17 | 16,39,2018-01-18,completed 18 | 17,71,2018-01-18,completed 19 | 18,64,2018-01-20,returned 20 | 19,54,2018-01-22,completed 21 | 20,20,2018-01-23,completed 22 | 21,71,2018-01-23,completed 23 | 22,86,2018-01-24,completed 24 | 23,22,2018-01-26,return_pending 25 | 24,3,2018-01-27,completed 26 | 25,51,2018-01-28,completed 27 | 26,32,2018-01-28,completed 28 | 27,94,2018-01-29,completed 29 | 28,8,2018-01-29,completed 30 | 29,57,2018-01-31,completed 31 | 30,69,2018-02-02,completed 32 | 31,16,2018-02-02,completed 33 | 32,28,2018-02-04,completed 34 | 33,42,2018-02-04,completed 35 | 34,38,2018-02-06,completed 36 | 35,80,2018-02-08,completed 37 | 36,85,2018-02-10,completed 38 | 37,1,2018-02-10,completed 39 | 38,51,2018-02-10,completed 40 | 39,26,2018-02-11,completed 41 | 40,33,2018-02-13,completed 42 | 41,99,2018-02-14,completed 43 | 42,92,2018-02-16,completed 44 | 43,31,2018-02-17,completed 45 | 44,66,2018-02-17,completed 46 | 45,22,2018-02-17,completed 47 | 46,6,2018-02-19,completed 48 | 47,50,2018-02-20,completed 49 | 48,27,2018-02-21,completed 50 | 49,35,2018-02-21,completed 51 | 50,51,2018-02-23,completed 52 | 51,71,2018-02-24,completed 53 | 52,54,2018-02-25,return_pending 54 | 53,34,2018-02-26,completed 55 | 54,54,2018-02-26,completed 56 | 55,18,2018-02-27,completed 57 | 56,79,2018-02-28,completed 58 | 57,93,2018-03-01,completed 59 | 58,22,2018-03-01,completed 60 | 59,30,2018-03-02,completed 61 | 60,12,2018-03-03,completed 62 | 61,63,2018-03-03,completed 63 | 62,57,2018-03-05,completed 64 | 63,70,2018-03-06,completed 65 | 64,13,2018-03-07,completed 66 | 65,26,2018-03-08,completed 67 | 66,36,2018-03-10,completed 68 | 67,79,2018-03-11,completed 69 | 68,53,2018-03-11,completed 70 | 69,3,2018-03-11,completed 71 | 70,8,2018-03-12,completed 72 | 71,42,2018-03-12,shipped 73 | 72,30,2018-03-14,shipped 74 | 73,19,2018-03-16,completed 75 | 74,9,2018-03-17,shipped 76 | 75,69,2018-03-18,completed 77 | 76,25,2018-03-20,completed 78 | 77,35,2018-03-21,shipped 79 | 78,90,2018-03-23,shipped 80 | 79,52,2018-03-23,shipped 81 | 80,11,2018-03-23,shipped 82 | 81,76,2018-03-23,shipped 83 | 82,46,2018-03-24,shipped 84 | 83,54,2018-03-24,shipped 85 | 84,70,2018-03-26,placed 86 | 85,47,2018-03-26,shipped 87 | 86,68,2018-03-26,placed 88 | 87,46,2018-03-27,placed 89 | 88,91,2018-03-27,shipped 90 | 89,21,2018-03-28,placed 91 | 90,66,2018-03-30,shipped 92 | 91,47,2018-03-31,placed 93 | 92,84,2018-04-02,placed 94 | 93,66,2018-04-03,placed 95 | 94,63,2018-04-03,placed 96 | 95,27,2018-04-04,placed 97 | 96,90,2018-04-06,placed 98 | 97,89,2018-04-07,placed 99 | 98,41,2018-04-07,placed 100 | 99,85,2018-04-09,placed 101 | -------------------------------------------------------------------------------- /data/raw_payments.csv: -------------------------------------------------------------------------------- 1 | id,order_id,payment_method,amount 2 | 1,1,credit_card,1000 3 | 2,2,credit_card,2000 4 | 3,3,coupon,100 5 | 4,4,coupon,2500 6 | 5,5,bank_transfer,1700 7 | 6,6,credit_card,600 8 | 7,7,credit_card,1600 9 | 8,8,credit_card,2300 10 | 9,9,gift_card,2300 11 | 10,9,bank_transfer,0 12 | 11,10,bank_transfer,2600 13 | 12,11,credit_card,2700 14 | 13,12,credit_card,100 15 | 14,13,credit_card,500 16 | 15,13,bank_transfer,1400 17 | 16,14,bank_transfer,300 18 | 17,15,coupon,2200 19 | 18,16,credit_card,1000 20 | 19,17,bank_transfer,200 21 | 20,18,credit_card,500 22 | 21,18,credit_card,800 23 | 22,19,gift_card,600 24 | 23,20,bank_transfer,1500 25 | 24,21,credit_card,1200 26 | 25,22,bank_transfer,800 27 | 26,23,gift_card,2300 28 | 27,24,coupon,2600 29 | 28,25,bank_transfer,2000 30 | 29,25,credit_card,2200 31 | 30,25,coupon,1600 32 | 31,26,credit_card,3000 33 | 32,27,credit_card,2300 34 | 33,28,bank_transfer,1900 35 | 34,29,bank_transfer,1200 36 | 35,30,credit_card,1300 37 | 36,31,credit_card,1200 38 | 37,32,credit_card,300 39 | 38,33,credit_card,2200 40 | 39,34,bank_transfer,1500 41 | 40,35,credit_card,2900 42 | 41,36,bank_transfer,900 43 | 42,37,credit_card,2300 44 | 43,38,credit_card,1500 45 | 44,39,bank_transfer,800 46 | 45,40,credit_card,1400 47 | 46,41,credit_card,1700 48 | 47,42,coupon,1700 49 | 48,43,gift_card,1800 50 | 49,44,gift_card,1100 51 | 50,45,bank_transfer,500 52 | 51,46,bank_transfer,800 53 | 52,47,credit_card,2200 54 | 53,48,bank_transfer,300 55 | 54,49,credit_card,600 56 | 55,49,credit_card,900 57 | 56,50,credit_card,2600 58 | 57,51,credit_card,2900 59 | 58,51,credit_card,100 60 | 59,52,bank_transfer,1500 61 | 60,53,credit_card,300 62 | 61,54,credit_card,1800 63 | 62,54,bank_transfer,1100 64 | 63,55,credit_card,2900 65 | 64,56,credit_card,400 66 | 65,57,bank_transfer,200 67 | 66,58,coupon,1800 68 | 67,58,gift_card,600 69 | 68,59,gift_card,2800 70 | 69,60,credit_card,400 71 | 70,61,bank_transfer,1600 72 | 71,62,gift_card,1400 73 | 72,63,credit_card,2900 74 | 73,64,bank_transfer,2600 75 | 74,65,credit_card,0 76 | 75,66,credit_card,2800 77 | 76,67,bank_transfer,400 78 | 77,67,credit_card,1900 79 | 78,68,credit_card,1600 80 | 79,69,credit_card,1900 81 | 80,70,credit_card,2600 82 | 81,71,credit_card,500 83 | 82,72,credit_card,2900 84 | 83,73,bank_transfer,300 85 | 84,74,credit_card,3000 86 | 85,75,credit_card,1900 87 | 86,76,coupon,200 88 | 87,77,credit_card,0 89 | 88,77,bank_transfer,1900 90 | 89,78,bank_transfer,2600 91 | 90,79,credit_card,1800 92 | 91,79,credit_card,900 93 | 92,80,gift_card,300 94 | 93,81,coupon,200 95 | 94,82,credit_card,800 96 | 95,83,credit_card,100 97 | 96,84,bank_transfer,2500 98 | 97,85,bank_transfer,1700 99 | 98,86,coupon,2300 100 | 99,87,gift_card,3000 101 | 100,87,credit_card,2600 102 | 101,88,credit_card,2900 103 | 102,89,bank_transfer,2200 104 | 103,90,bank_transfer,200 105 | 104,91,credit_card,1900 106 | 105,92,bank_transfer,1500 107 | 106,92,coupon,200 108 | 107,93,gift_card,2600 109 | 108,94,coupon,700 110 | 109,95,coupon,2400 111 | 110,96,gift_card,1700 112 | 111,97,bank_transfer,1400 113 | 112,98,bank_transfer,1000 114 | 113,99,credit_card,2400 115 | -------------------------------------------------------------------------------- /dbt_project.yml: -------------------------------------------------------------------------------- 1 | 2 | name: 'jaffle_shop' 3 | version: '0.1' 4 | profile: 'jaffle_shop' 5 | config-version: 2 6 | 7 | source-paths: ["models"] 8 | analysis-paths: ["analysis"] 9 | test-paths: ["tests"] 10 | data-paths: ["data"] 11 | macro-paths: ["macros"] 12 | 13 | target-path: "target" 14 | clean-targets: 15 | - "target" 16 | - "dbt_modules" 17 | - "logs" 18 | 19 | models: 20 | jaffle_shop: 21 | materialized: table 22 | staging: 23 | materialized: view 24 | -------------------------------------------------------------------------------- /lightdash-dockerfile: -------------------------------------------------------------------------------- 1 | FROM lightdash/lightdash:latest 2 | 3 | COPY ./dbt_project.yml /usr/app/dbt/dbt_project.yml 4 | COPY ./data /usr/app/dbt/data 5 | COPY ./models /usr/app/dbt/models 6 | COPY ./profiles /usr/app/profiles 7 | COPY ./lightdash-entrypoint.sh /usr/bin/lightdash-entrypoint.sh 8 | 9 | EXPOSE 8080 10 | 11 | ENTRYPOINT ["/usr/bin/lightdash-entrypoint.sh"] 12 | -------------------------------------------------------------------------------- /lightdash-entrypoint.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # -- OPTIONAL -- 4 | # Only needed if you want to run dbt commands on deploying lightdash 5 | cd /usr/app/dbt 6 | dbt seed --full-refresh 7 | dbt run 8 | cd /usr/app/packages/backend 9 | # -------------- 10 | 11 | 12 | exec yarn start 13 | -------------------------------------------------------------------------------- /models/customers.sql: -------------------------------------------------------------------------------- 1 | with customers as ( 2 | 3 | select * from {{ ref('stg_customers') }} 4 | 5 | ), 6 | 7 | orders as ( 8 | 9 | select * from {{ ref('stg_orders') }} 10 | 11 | ), 12 | 13 | payments as ( 14 | 15 | select * from {{ ref('stg_payments') }} 16 | 17 | ), 18 | 19 | customer_orders as ( 20 | 21 | select 22 | customer_id, 23 | 24 | min(order_date) as first_order, 25 | max(order_date) as most_recent_order, 26 | count(order_id) as number_of_orders 27 | from orders 28 | 29 | group by 1 30 | 31 | ), 32 | 33 | customer_payments as ( 34 | 35 | select 36 | orders.customer_id, 37 | sum(amount) as total_amount 38 | 39 | from payments 40 | 41 | left join orders using (order_id) 42 | 43 | group by 1 44 | 45 | ), 46 | 47 | final as ( 48 | 49 | select 50 | customers.customer_id, 51 | customers.first_name, 52 | customers.last_name, 53 | customer_orders.first_order, 54 | customer_orders.most_recent_order, 55 | customer_orders.number_of_orders, 56 | customer_payments.total_amount as customer_lifetime_value 57 | 58 | from customers 59 | 60 | left join customer_orders using (customer_id) 61 | 62 | left join customer_payments using (customer_id) 63 | 64 | ) 65 | 66 | select * from final 67 | -------------------------------------------------------------------------------- /models/docs.md: -------------------------------------------------------------------------------- 1 | {% docs orders_status %} 2 | 3 | Orders can be one of the following statuses: 4 | 5 | | status | description | 6 | |----------------|------------------------------------------------------------------------------------------------------------------------| 7 | | placed | The order has been placed but has not yet left the warehouse | 8 | | shipped | The order has ben shipped to the customer and is currently in transit | 9 | | completed | The order has been received by the customer | 10 | | return_pending | The customer has indicated that they would like to return the order, but it has not yet been received at the warehouse | 11 | | returned | The order has been returned by the customer and received at the warehouse | 12 | 13 | 14 | {% enddocs %} 15 | -------------------------------------------------------------------------------- /models/orders.sql: -------------------------------------------------------------------------------- 1 | {% set payment_methods = ['credit_card', 'coupon', 'bank_transfer', 'gift_card'] %} 2 | 3 | with orders as ( 4 | 5 | select * from {{ ref('stg_orders') }} 6 | 7 | ), 8 | 9 | payments as ( 10 | 11 | select * from {{ ref('stg_payments') }} 12 | 13 | ), 14 | 15 | order_payments as ( 16 | 17 | select 18 | order_id, 19 | 20 | {% for payment_method in payment_methods -%} 21 | sum(case when payment_method = '{{ payment_method }}' then amount else 0 end) as {{ payment_method }}_amount, 22 | {% endfor -%} 23 | 24 | sum(amount) as total_amount 25 | 26 | from payments 27 | 28 | group by 1 29 | 30 | ), 31 | 32 | final as ( 33 | 34 | select 35 | orders.order_id, 36 | orders.customer_id, 37 | orders.order_date, 38 | orders.status, 39 | 40 | {% for payment_method in payment_methods -%} 41 | 42 | order_payments.{{ payment_method }}_amount, 43 | 44 | {% endfor -%} 45 | 46 | order_payments.total_amount as amount 47 | 48 | from orders 49 | 50 | left join order_payments using (order_id) 51 | 52 | ) 53 | 54 | select * from final 55 | -------------------------------------------------------------------------------- /models/overview.md: -------------------------------------------------------------------------------- 1 | {% docs __overview__ %} 2 | 3 | ## Data Documentation for Jaffle Shop 4 | 5 | `jaffle_shop` is a fictional ecommerce store. 6 | 7 | This [dbt](https://www.getdbt.com/) project is for testing out code. 8 | 9 | The source code can be found [here](https://github.com/clrcrl/jaffle_shop). 10 | 11 | {% enddocs %} 12 | -------------------------------------------------------------------------------- /models/payments.sql: -------------------------------------------------------------------------------- 1 | select * from {{ ref('stg_payments') }} 2 | -------------------------------------------------------------------------------- /models/schema.yml: -------------------------------------------------------------------------------- 1 | version: 2 2 | 3 | models: 4 | - name: customers 5 | description: This table has basic information about a customer, as well as some derived facts based on a customer's orders 6 | 7 | columns: 8 | - name: customer_id 9 | description: This is a unique identifier for a customer 10 | tests: 11 | - unique 12 | - not_null 13 | meta: 14 | metrics: 15 | total_customers: 16 | type: count_distinct 17 | 18 | - name: first_name 19 | description: Customer's first name. PII. 20 | 21 | - name: last_name 22 | description: Customer's last name. PII. 23 | 24 | - name: first_order 25 | description: Date (UTC) of a customer's first order 26 | 27 | - name: most_recent_order 28 | description: Date (UTC) of a customer's most recent order 29 | 30 | - name: number_of_orders 31 | description: Count of the number of orders a customer has placed 32 | 33 | - name: total_order_amount 34 | description: Total value (AUD) of a customer's orders 35 | 36 | - name: orders 37 | description: This table has basic information about orders, as well as some derived facts based on payments 38 | 39 | meta: 40 | joins: 41 | - join: customers 42 | sql_on: ${customers.customer_id} = ${orders.customer_id} 43 | 44 | 45 | columns: 46 | - name: order_id 47 | tests: 48 | - unique 49 | - not_null 50 | description: This is a unique identifier for an order 51 | meta: 52 | metrics: 53 | total_orders: 54 | type: count 55 | more_orders: 56 | type: count 57 | 58 | - name: customer_id 59 | description: Foreign key to the customers table 60 | tests: 61 | - not_null 62 | - relationships: 63 | to: ref('customers') 64 | field: customer_id 65 | 66 | - name: order_date 67 | description: Date (UTC) that the order was placed 68 | meta: 69 | metrics: 70 | first_ordered_at: 71 | type: min 72 | last_ordered_at: 73 | type: max 74 | 75 | - name: status 76 | description: '{{ doc("orders_status") }}' 77 | tests: 78 | - accepted_values: 79 | values: ['placed', 'shipped', 'completed', 'return_pending', 'returned'] 80 | 81 | - name: amount 82 | description: Total amount (AUD) of the order 83 | tests: 84 | - not_null 85 | 86 | - name: credit_card_amount 87 | description: Amount of the order (AUD) paid for by credit card 88 | tests: 89 | - not_null 90 | 91 | - name: coupon_amount 92 | description: Amount of the order (AUD) paid for by coupon 93 | tests: 94 | - not_null 95 | 96 | - name: bank_transfer_amount 97 | description: Amount of the order (AUD) paid for by bank transfer 98 | tests: 99 | - not_null 100 | 101 | - name: gift_card_amount 102 | description: Amount of the order (AUD) paid for by gift card 103 | tests: 104 | - not_null 105 | 106 | - name: payments 107 | description: This table has information about individual payments 108 | 109 | meta: 110 | joins: 111 | - join: orders 112 | sql_on: ${orders.order_id} = ${payments.order_id} 113 | - join: customers 114 | sql_on: ${customers.customer_id} = ${orders.customer_id} 115 | 116 | columns: 117 | - name: payment_id 118 | description: This is a unique identifier for a payment 119 | meta: 120 | metrics: 121 | payment_count: 122 | type: count 123 | description: count of all payments 124 | 125 | - name: order_id 126 | description: Foreign key to the orders table 127 | 128 | - name: payment_method 129 | description: Method of payment used, for example credit card 130 | 131 | - name: amount 132 | description: Total amount (AUD) of the individual payment 133 | meta: 134 | metrics: 135 | total_revenue: 136 | type: sum 137 | description: Sum of all payments 138 | -------------------------------------------------------------------------------- /models/staging/schema.yml: -------------------------------------------------------------------------------- 1 | version: 2 2 | 3 | models: 4 | - name: stg_customers 5 | columns: 6 | - name: customer_id 7 | tests: 8 | - unique 9 | - not_null 10 | 11 | - name: stg_orders 12 | columns: 13 | - name: order_id 14 | tests: 15 | - unique 16 | - not_null 17 | - name: status 18 | tests: 19 | - accepted_values: 20 | values: ['placed', 'shipped', 'completed', 'return_pending', 'returned'] 21 | 22 | - name: stg_payments 23 | columns: 24 | - name: payment_id 25 | tests: 26 | - unique 27 | - not_null 28 | - name: payment_method 29 | tests: 30 | - accepted_values: 31 | values: ['credit_card', 'coupon', 'bank_transfer', 'gift_card'] 32 | -------------------------------------------------------------------------------- /models/staging/stg_customers.sql: -------------------------------------------------------------------------------- 1 | with source as ( 2 | 3 | {#- 4 | Normally we would select from the table here, but we are using seeds to load 5 | our data in this project 6 | #} 7 | select * from {{ ref('raw_customers') }} 8 | 9 | ), 10 | 11 | renamed as ( 12 | 13 | select 14 | id as customer_id, 15 | first_name, 16 | last_name 17 | 18 | from source 19 | 20 | ) 21 | 22 | select * from renamed 23 | -------------------------------------------------------------------------------- /models/staging/stg_orders.sql: -------------------------------------------------------------------------------- 1 | with source as ( 2 | 3 | {#- 4 | Normally we would select from the table here, but we are using seeds to load 5 | our data in this project 6 | #} 7 | select * from {{ ref('raw_orders') }} 8 | 9 | ), 10 | 11 | renamed as ( 12 | 13 | select 14 | id as order_id, 15 | user_id as customer_id, 16 | order_date, 17 | status 18 | 19 | from source 20 | 21 | ) 22 | 23 | select * from renamed 24 | -------------------------------------------------------------------------------- /models/staging/stg_payments.sql: -------------------------------------------------------------------------------- 1 | with source as ( 2 | 3 | {#- 4 | Normally we would select from the table here, but we are using seeds to load 5 | our data in this project 6 | #} 7 | select * from {{ ref('raw_payments') }} 8 | 9 | ), 10 | 11 | renamed as ( 12 | 13 | select 14 | id as payment_id, 15 | order_id, 16 | payment_method, 17 | 18 | --`amount` is currently stored in cents, so we convert it to dollars 19 | amount / 100 as amount 20 | 21 | from source 22 | 23 | ) 24 | 25 | select * from renamed 26 | -------------------------------------------------------------------------------- /profiles/profiles.yml: -------------------------------------------------------------------------------- 1 | jaffle_shop: 2 | target: jaffle 3 | outputs: 4 | jaffle: 5 | type: bigquery 6 | method: oauth 7 | project: "{{ env_var('GCP_PROJECT_ID') }}" 8 | schema: "{{ env_var('GCP_DATASET_ID') }}" 9 | threads: 1 10 | timeout_seconds: 300 11 | location: "{{ env_var('GCP_DATASET_LOCATION') }}" 12 | priority: interactive 13 | retries: 1 14 | 15 | -------------------------------------------------------------------------------- /screenshots/.DS_Store: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lightdash/lightdash-production-example/de672eca9890d15d186258ac78006282af410b40/screenshots/.DS_Store -------------------------------------------------------------------------------- /screenshots/1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lightdash/lightdash-production-example/de672eca9890d15d186258ac78006282af410b40/screenshots/1.png -------------------------------------------------------------------------------- /screenshots/2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lightdash/lightdash-production-example/de672eca9890d15d186258ac78006282af410b40/screenshots/2.png -------------------------------------------------------------------------------- /screenshots/3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lightdash/lightdash-production-example/de672eca9890d15d186258ac78006282af410b40/screenshots/3.png -------------------------------------------------------------------------------- /screenshots/4.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lightdash/lightdash-production-example/de672eca9890d15d186258ac78006282af410b40/screenshots/4.png -------------------------------------------------------------------------------- /screenshots/5.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lightdash/lightdash-production-example/de672eca9890d15d186258ac78006282af410b40/screenshots/5.png -------------------------------------------------------------------------------- /screenshots/6.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lightdash/lightdash-production-example/de672eca9890d15d186258ac78006282af410b40/screenshots/6.png -------------------------------------------------------------------------------- /screenshots/7.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lightdash/lightdash-production-example/de672eca9890d15d186258ac78006282af410b40/screenshots/7.png -------------------------------------------------------------------------------- /screenshots/8.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/lightdash/lightdash-production-example/de672eca9890d15d186258ac78006282af410b40/screenshots/8.png --------------------------------------------------------------------------------