├── .gitignore ├── .pre-commit-config.yaml ├── cohort_1 ├── .gitignore ├── LICENSE ├── README.md ├── requirements.txt ├── week1_bootstrap_evals │ ├── .gitignore │ ├── README.md │ ├── make_product_reviews.ipynb │ ├── make_synthetic_questions.ipynb │ ├── make_synthetic_questions.py │ ├── metrics.ipynb │ ├── reviews.json │ ├── scoring_utils.py │ ├── synthetic_eval_dataset.json │ └── synthetic_finetune_dataset.jsonl ├── week2_question_classification │ ├── classify_with_yaml │ │ ├── example.py │ │ ├── example.yaml │ │ ├── rag_classifer.py │ │ ├── readme.md │ │ ├── requirements.txt │ │ └── yaml_classifier.py │ └── example_notebook │ │ ├── README.md │ │ ├── analyze_clusters.ipynb │ │ ├── make_prod_questions.ipynb │ │ ├── prod_questions.json │ │ └── question_types.py ├── week3_seperate_indices │ ├── README.md │ └── tool_creation.ipynb ├── week4_routing │ ├── README.md │ ├── benchmark_tool_retrieval.ipynb │ ├── funcs_to_call.py │ └── utils.py ├── week5 │ ├── pyproject.toml │ └── uv.lock ├── week5_fine_tuning │ ├── README.md │ ├── cohere_fine_tuning.ipynb │ ├── eval_sbert.py │ ├── finetune_sbert.py │ └── ft_dataset.jsonl └── week6_product │ └── streaming_examples.ipynb └── cohort_2 ├── .env.example ├── .gitignore ├── README.md ├── convert.py ├── office-hours ├── README.md ├── merge.py ├── move-files.py ├── week1-summary.md ├── week2-summary.md ├── week3-summary.md ├── week3 │ └── week3-summary.md └── week4-summary.md ├── pyproject.toml ├── requirements.txt ├── week0 ├── 1. Using Jupyter Notebooks.ipynb ├── 2. LanceDB.ipynb ├── 3. Using Pydantic Evals.ipynb ├── README.md ├── assets │ └── kernel.png └── uv.md ├── week1 ├── 1. synthetic_questions.ipynb ├── 1. synthetic_questions_logfire.ipynb ├── 1. synthetic_questions_modal.ipynb ├── 2. benchmark_retrieval.ipynb ├── 2. benchmark_retrieval_logfire.ipynb ├── 3. visualise_results.ipynb ├── 3. visualise_results_logfire.ipynb ├── README.md ├── modal_vllm.py ├── questions.yaml └── questions_schema.json ├── week2 ├── 1. Synthetic-Transactions-Logfire.ipynb ├── 1. Synthetic-Transactions.ipynb ├── 2. Finetune Cohere Logfire.ipynb ├── 2. Finetune Cohere.ipynb ├── 3. Open Source Models.ipynb ├── 3. Open Source Models_logfire.ipynb ├── README.md ├── data │ ├── categories.json │ ├── cleaned.jsonl │ ├── eval_transactions.jsonl │ ├── hard_negatives.png │ ├── semi-hard-negative.png │ └── train_transactions.jsonl ├── helpers.py └── label.py ├── week4 ├── 1. Generate Dataset.ipynb ├── 2. Topic Modelling.ipynb ├── 3. Classifier.ipynb ├── README.md ├── assets │ └── matrix.png ├── categories.yml ├── config.yml ├── data │ ├── cleaned.jsonl │ ├── filtered.jsonl │ ├── html │ │ ├── can-i-change-my-return-after-its-been-reported.html │ │ ├── can-i-pay-in-store-using-klarna.html │ │ ├── can-i-use-klarna-to-pay-for-subscriptions-and-on-demand-services.html │ │ ├── can-i-use-klarna-to-purchase-gift-cards.html │ │ ├── does-klarna-charge-an-annual-fee.html │ │ ├── how-can-i-add-remove-or-change-my-card-or-bank-account.html │ │ ├── how-can-i-avoid-late-fees.html │ │ ├── how-can-i-change-my-email-address.html │ │ ├── how-can-i-extend-my-due-date.html │ │ ├── how-can-i-get-approved-to-pay-with-klarna.html │ │ ├── how-can-i-make-a-formal-complaint-to-klarna.html │ │ ├── how-can-i-merge-my-klarna-accounts.html │ │ ├── how-can-i-pay-for-my-klarna-card-purchases.html │ │ ├── how-can-i-pay.html │ │ ├── how-can-i-protect-myself-against-identity-theft.html │ │ ├── how-can-i-report-a-fraud-attempt.html │ │ ├── how-can-i-request-the-deletion-of-my-personal-data.html │ │ ├── how-can-i-use-klarna-on-apple-pay.html │ │ ├── how-do-i-add-money-into-my-klarna-balance.html │ │ ├── how-do-i-apply-for-a-klarna-card.html │ │ ├── how-do-i-block-or-cancel-my-klarna-card.html │ │ ├── how-do-i-cancel-my-order.html │ │ ├── how-do-i-change-my-billing-address.html │ │ ├── how-do-i-change-my-name.html │ │ ├── how-do-i-change-the-card-for-a-purchase.html │ │ ├── how-do-i-close-my-klarna-account.html │ │ ├── how-do-i-log-in.html │ │ ├── how-do-i-return-an-order-purchased-with-a-gift-card.html │ │ ├── how-do-i-return-my-order.html │ │ ├── how-do-i-use-my-klarna-balance.html │ │ ├── how-long-do-refunds-take.html │ │ ├── how-long-does-it-take-for-my-problem-to-be-solved.html │ │ ├── how-long-does-it-take-for-my-return-to-be-registered.html │ │ ├── how-much-am-i-eligible-to-spend.html │ │ ├── i-cant-log-in-what-can-i-do.html │ │ ├── i-cant-open-a-klarna-balance-what-can-i-do.html │ │ ├── i-dont-recognize-a-charge-what-should-i-do.html │ │ ├── i-have-a-payment-due-but-get-an-error-message-when-i-try-to-pay-what-can-i-do.html │ │ ├── i-have-a-problem-with-the-order-purchased-with-my-gift-card-what-should-i-do.html │ │ ├── i-havent-received-my-order-do-i-have-to-pay.html │ │ ├── if-the-store-only-confirms-part-of-my-order-will-my-payment-plan-be-adjusted.html │ │ ├── is-shopping-with-klarna-safe.html │ │ ├── klarna-perform-credit-check.html │ │ ├── my-order-was-cancelled-why-am-i-still-being-charged.html │ │ ├── my-problem-has-been-solved-what-should-i-do.html │ │ ├── my-purchase-was-denied-why-do-i-still-have-a-pending-authorization-hold-on-my-card.html │ │ ├── my-return-hasnt-been-registered-what-should-i-do.html │ │ ├── theres-something-wrong-with-my-order-do-i-have-to-pay.html │ │ ├── what-happens-after-ive-reported-my-problem.html │ │ ├── what-happens-if-i-cant-pay-on-time.html │ │ ├── what-happens-if-my-purchase-is-for-a-different-amount-than-the-one-time-card.html │ │ ├── what-happens-when-i-report-a-return.html │ │ ├── what-if-i-never-use-the-one-time-card-and-want-to-cancel-it.html │ │ ├── what-is-a-one-time-card-and-how-does-it-work.html │ │ ├── what-is-financing-and-how-does-it-work.html │ │ ├── what-is-pay-later-in-30-days.html │ │ ├── what-is-phishing-and-how-can-i-identify-it.html │ │ ├── what-should-i-do-if-the-store-doesnt-respond.html │ │ ├── whats-the-status-of-the-investigation-into-my-problem.html │ │ ├── whats-the-status-of-the-investigation-of-my-return.html │ │ ├── when-is-my-payment-due.html │ │ ├── when-will-i-receive-my-klarna-card-refund.html │ │ ├── when-will-my-klarna-card-be-delivered.html │ │ ├── when-will-my-order-be-delivered.html │ │ ├── where-can-i-find-my-payment-plan.html │ │ ├── where-can-i-use-klarna.html │ │ ├── which-payment-methods-are-accepted-by-klarna.html │ │ ├── who-can-use-klarna.html │ │ ├── why-are-not-all-payment-methods-available-at-the-checkout.html │ │ ├── why-cant-i-pay-with-klarna-anymore.html │ │ ├── why-did-i-receive-a-reminder-after-returning-my-order.html │ │ ├── why-did-klarna-charge-me-twice.html │ │ ├── why-do-i-need-to-verify-my-details-when-paying.html │ │ ├── why-do-i-see-two-orders-in-the-klarna-app-when-i-only-placed-one.html │ │ ├── why-is-my-debt-in-collection.html │ │ ├── why-was-i-charged-a-late-fee.html │ │ ├── why-was-i-denied-a-klarna-card.html │ │ ├── why-was-my-klarna-card-purchase-declined.html │ │ ├── why-was-my-purchase-not-approved-with-klarna.html │ │ ├── why-was-the-first-installment-of-my-payment-plan-higher.html │ │ ├── why-was-the-investigation-of-my-problem-closed-even-though-my-problem-has-not-been-solved-yet.html │ │ ├── why-was-the-investigation-of-my-return-closed-even-though-my-return-hasnt-been-registered-yet.html │ │ └── why-was-the-one-time-card-purchase-declined.html │ ├── md │ │ ├── can-i-change-my-return-after-its-been-reported.md │ │ ├── can-i-pay-in-store-using-klarna.md │ │ ├── can-i-use-klarna-to-pay-for-subscriptions-and-on-demand-services.md │ │ ├── can-i-use-klarna-to-purchase-gift-cards.md │ │ ├── does-klarna-charge-an-annual-fee.md │ │ ├── how-can-i-add-remove-or-change-my-card-or-bank-account.md │ │ ├── how-can-i-avoid-late-fees.md │ │ ├── how-can-i-change-my-email-address.md │ │ ├── how-can-i-extend-my-due-date.md │ │ ├── how-can-i-get-approved-to-pay-with-klarna.md │ │ ├── how-can-i-make-a-formal-complaint-to-klarna.md │ │ ├── how-can-i-merge-my-klarna-accounts.md │ │ ├── how-can-i-pay-for-my-klarna-card-purchases.md │ │ ├── how-can-i-pay.md │ │ ├── how-can-i-protect-myself-against-identity-theft.md │ │ ├── how-can-i-report-a-fraud-attempt.md │ │ ├── how-can-i-request-the-deletion-of-my-personal-data.md │ │ ├── how-can-i-use-klarna-on-apple-pay.md │ │ ├── how-do-i-add-money-into-my-klarna-balance.md │ │ ├── how-do-i-apply-for-a-klarna-card.md │ │ ├── how-do-i-block-or-cancel-my-klarna-card.md │ │ ├── how-do-i-cancel-my-order.md │ │ ├── how-do-i-change-my-billing-address.md │ │ ├── how-do-i-change-my-name.md │ │ ├── how-do-i-change-the-card-for-a-purchase.md │ │ ├── how-do-i-close-my-klarna-account.md │ │ ├── how-do-i-log-in.md │ │ ├── how-do-i-return-an-order-purchased-with-a-gift-card.md │ │ ├── how-do-i-return-my-order.md │ │ ├── how-do-i-use-my-klarna-balance.md │ │ ├── how-long-do-refunds-take.md │ │ ├── how-long-does-it-take-for-my-problem-to-be-solved.md │ │ ├── how-long-does-it-take-for-my-return-to-be-registered.md │ │ ├── how-much-am-i-eligible-to-spend.md │ │ ├── i-cant-log-in-what-can-i-do.md │ │ ├── i-cant-open-a-klarna-balance-what-can-i-do.md │ │ ├── i-dont-recognize-a-charge-what-should-i-do.md │ │ ├── i-have-a-payment-due-but-get-an-error-message-when-i-try-to-pay-what-can-i-do.md │ │ ├── i-have-a-problem-with-the-order-purchased-with-my-gift-card-what-should-i-do.md │ │ ├── i-havent-received-my-order-do-i-have-to-pay.md │ │ ├── if-the-store-only-confirms-part-of-my-order-will-my-payment-plan-be-adjusted.md │ │ ├── is-shopping-with-klarna-safe.md │ │ ├── klarna-perform-credit-check.md │ │ ├── my-order-was-cancelled-why-am-i-still-being-charged.md │ │ ├── my-problem-has-been-solved-what-should-i-do.md │ │ ├── my-purchase-was-denied-why-do-i-still-have-a-pending-authorization-hold-on-my-card.md │ │ ├── my-return-hasnt-been-registered-what-should-i-do.md │ │ ├── theres-something-wrong-with-my-order-do-i-have-to-pay.md │ │ ├── what-happens-after-ive-reported-my-problem.md │ │ ├── what-happens-if-i-cant-pay-on-time.md │ │ ├── what-happens-if-my-purchase-is-for-a-different-amount-than-the-one-time-card.md │ │ ├── what-happens-when-i-report-a-return.md │ │ ├── what-if-i-never-use-the-one-time-card-and-want-to-cancel-it.md │ │ ├── what-is-a-one-time-card-and-how-does-it-work.md │ │ ├── what-is-financing-and-how-does-it-work.md │ │ ├── what-is-pay-later-in-30-days.md │ │ ├── what-is-phishing-and-how-can-i-identify-it.md │ │ ├── what-should-i-do-if-the-store-doesnt-respond.md │ │ ├── whats-the-status-of-the-investigation-into-my-problem.md │ │ ├── whats-the-status-of-the-investigation-of-my-return.md │ │ ├── when-is-my-payment-due.md │ │ ├── when-will-i-receive-my-klarna-card-refund.md │ │ ├── when-will-my-klarna-card-be-delivered.md │ │ ├── when-will-my-order-be-delivered.md │ │ ├── where-can-i-find-my-payment-plan.md │ │ ├── where-can-i-use-klarna.md │ │ ├── which-payment-methods-are-accepted-by-klarna.md │ │ ├── who-can-use-klarna.md │ │ ├── why-are-not-all-payment-methods-available-at-the-checkout.md │ │ ├── why-cant-i-pay-with-klarna-anymore.md │ │ ├── why-did-i-receive-a-reminder-after-returning-my-order.md │ │ ├── why-did-klarna-charge-me-twice.md │ │ ├── why-do-i-need-to-verify-my-details-when-paying.md │ │ ├── why-do-i-see-two-orders-in-the-klarna-app-when-i-only-placed-one.md │ │ ├── why-is-my-debt-in-collection.md │ │ ├── why-was-i-charged-a-late-fee.md │ │ ├── why-was-i-denied-a-klarna-card.md │ │ ├── why-was-my-klarna-card-purchase-declined.md │ │ ├── why-was-my-purchase-not-approved-with-klarna.md │ │ ├── why-was-the-first-installment-of-my-payment-plan-higher.md │ │ ├── why-was-the-investigation-of-my-problem-closed-even-though-my-problem-has-not-been-solved-yet.md │ │ ├── why-was-the-investigation-of-my-return-closed-even-though-my-return-hasnt-been-registered-yet.md │ │ └── why-was-the-one-time-card-purchase-declined.md │ ├── questions.jsonl │ ├── results_df.csv │ └── synthetic_questions.jsonl └── label.py ├── week5 ├── 1. Generate Dataset.ipynb ├── 2. Metadata Filtering Logfire.ipynb ├── 2. Metadata Filtering.ipynb ├── 3. Text-2-SQL.ipynb ├── 4. PDF-Parser.ipynb ├── Cohere-Embed-v4.ipynb ├── README.md ├── data │ ├── documents │ │ ├── maybank-world-mastercard-tnc.pdf │ │ ├── mb-regional-golf-program-tnc.pdf │ │ └── personal-credit-card-tnc.pdf │ └── invoice.pdf ├── helpers.py ├── init.sql ├── queries.json └── taxonomy.yml └── week6 ├── 1. Evaluate Tools.ipynb ├── 2. Generate Dataset Logfire.ipynb ├── 2. Generate Dataset.ipynb ├── 3. Improving Performance Logfire.ipynb ├── 3. Improving Performance.ipynb ├── README.md ├── helpers.py ├── queries.jsonl ├── queries.yaml ├── queries_schema.json └── raw_commands.json /.gitignore: -------------------------------------------------------------------------------- 1 | # Python version files (recursive) 2 | **/.python-version 3 | 4 | # uv lock files (recursive) 5 | **/uv.lock 6 | 7 | # Transcript files and merged text files 8 | **/*.txt 9 | cohort_2/md 10 | .DS_Store 11 | *.vtt 12 | 13 | # Participation statistics outputs 14 | participation.py 15 | **/participation_stats.csv 16 | **/visualization/*.png 17 | **/visualization/ 18 | docs/.cache/ 19 | -------------------------------------------------------------------------------- /.pre-commit-config.yaml: -------------------------------------------------------------------------------- 1 | repos: 2 | - repo: https://github.com/pre-commit/pre-commit-hooks 3 | rev: v4.5.0 4 | hooks: 5 | - id: trailing-whitespace 6 | - id: end-of-file-fixer 7 | - id: check-yaml 8 | - id: check-added-large-files 9 | 10 | - repo: https://github.com/psf/black 11 | rev: 24.2.0 12 | hooks: 13 | - id: black 14 | language_version: python3 15 | 16 | - repo: https://github.com/astral-sh/ruff-pre-commit 17 | rev: v0.3.0 18 | hooks: 19 | - id: ruff 20 | args: [--fix] -------------------------------------------------------------------------------- /cohort_1/.gitignore: -------------------------------------------------------------------------------- 1 | .DS_Store 2 | # Byte-compiled / optimized / DLL files 3 | __pycache__/ 4 | *.py[cod] 5 | *$py.class 6 | 7 | # C extensions 8 | *.so 9 | 10 | # Distribution / packaging 11 | .Python 12 | build/ 13 | develop-eggs/ 14 | dist/ 15 | downloads/ 16 | eggs/ 17 | .eggs/ 18 | lib/ 19 | lib64/ 20 | parts/ 21 | sdist/ 22 | var/ 23 | wheels/ 24 | share/python-wheels/ 25 | *.egg-info/ 26 | .installed.cfg 27 | *.egg 28 | MANIFEST 29 | 30 | # PyInstaller 31 | # Usually these files are written by a python script from a template 32 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 33 | *.manifest 34 | *.spec 35 | 36 | # Installer logs 37 | pip-log.txt 38 | pip-delete-this-directory.txt 39 | 40 | # Unit test / coverage reports 41 | htmlcov/ 42 | .tox/ 43 | .nox/ 44 | .coverage 45 | .coverage.* 46 | .cache 47 | nosetests.xml 48 | coverage.xml 49 | *.cover 50 | *.py,cover 51 | .hypothesis/ 52 | .pytest_cache/ 53 | cover/ 54 | 55 | # Translations 56 | *.mo 57 | *.pot 58 | 59 | # Django stuff: 60 | *.log 61 | local_settings.py 62 | db.sqlite3 63 | db.sqlite3-journal 64 | 65 | # Flask stuff: 66 | instance/ 67 | .webassets-cache 68 | 69 | # Scrapy stuff: 70 | .scrapy 71 | 72 | # Sphinx documentation 73 | docs/_build/ 74 | 75 | # PyBuilder 76 | .pybuilder/ 77 | target/ 78 | 79 | # Jupyter Notebook 80 | .ipynb_checkpoints 81 | 82 | # IPython 83 | profile_default/ 84 | ipython_config.py 85 | 86 | # pyenv 87 | # For a library or package, you might want to ignore these files since the code is 88 | # intended to run in multiple environments; otherwise, check them in: 89 | # .python-version 90 | 91 | # pipenv 92 | # According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control. 93 | # However, in case of collaboration, if having platform-specific dependencies or dependencies 94 | # having no cross-platform support, pipenv may install dependencies that don't work, or not 95 | # install all needed dependencies. 96 | #Pipfile.lock 97 | 98 | # poetry 99 | # Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control. 100 | # This is especially recommended for binary packages to ensure reproducibility, and is more 101 | # commonly ignored for libraries. 102 | # https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control 103 | #poetry.lock 104 | 105 | # pdm 106 | # Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control. 107 | #pdm.lock 108 | # pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it 109 | # in version control. 110 | # https://pdm.fming.dev/latest/usage/project/#working-with-version-control 111 | .pdm.toml 112 | .pdm-python 113 | .pdm-build/ 114 | 115 | # PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm 116 | __pypackages__/ 117 | 118 | # Celery stuff 119 | celerybeat-schedule 120 | celerybeat.pid 121 | 122 | # SageMath parsed files 123 | *.sage.py 124 | 125 | # Environments 126 | .env 127 | .venv 128 | env/ 129 | venv/ 130 | ENV/ 131 | env.bak/ 132 | venv.bak/ 133 | 134 | # Spyder project settings 135 | .spyderproject 136 | .spyproject 137 | 138 | # Rope project settings 139 | .ropeproject 140 | 141 | # mkdocs documentation 142 | /site 143 | 144 | # mypy 145 | .mypy_cache/ 146 | .dmypy.json 147 | dmypy.json 148 | 149 | # Pyre type checker 150 | .pyre/ 151 | 152 | # pytype static type analyzer 153 | .pytype/ 154 | 155 | # Cython debug symbols 156 | cython_debug/ 157 | 158 | # PyCharm 159 | # JetBrains specific template is maintained in a separate JetBrains.gitignore that can 160 | # be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore 161 | # and can be added to the global gitignore or merged into this file. For a more nuclear 162 | # option (not recommended) you can uncomment the following to ignore the entire idea folder. 163 | #.idea/ 164 | -------------------------------------------------------------------------------- /cohort_1/LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2024 567-labs 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /cohort_1/README.md: -------------------------------------------------------------------------------- 1 | # systematically-improving-rag -------------------------------------------------------------------------------- /cohort_1/requirements.txt: -------------------------------------------------------------------------------- 1 | cohere 2 | diskcache 3 | openai 4 | instructor 5 | pydantic 6 | lancedb 7 | matplotlib 8 | modal 9 | pandas 10 | rich 11 | sentence-transformers 12 | jupyter 13 | ipykernel 14 | wandb -------------------------------------------------------------------------------- /cohort_1/week1_bootstrap_evals/.gitignore: -------------------------------------------------------------------------------- 1 | lancedb 2 | cohere_cache 3 | -------------------------------------------------------------------------------- /cohort_1/week1_bootstrap_evals/README.md: -------------------------------------------------------------------------------- 1 | This folder includes code from week 1 of [Systematically Improving RAG Applications`](https://maven.com/applied-llms/rag-playbook). 2 | 3 | It shows you how to bootstrap evaluation of a RAG system with synthetic evaluations so you can measure the impact of changes in your retrieval pipelines. 4 | 5 | The code uses the example of a hardware e-commerce site building a RAG feature to answer consumer questions based on historical product reviews. The notebook of interest is `make_synthetic_questions.ipynb`. That notebook explains more of what we are trying to achieve with synthetic evaluation. 6 | 7 | However a RAG product starts with a corpus of documents to retrieve. For you to be able to run `make_synthetic_questions.ipynb` we also generate synthetic product reviews. That happens in `make_product_reviews.ipynb`. You won't use `make_product_reviews.ipynb` when applying this code to your own projects. 8 | 9 | Finally, we calculate metrics on these questions in `metrics.ipynb`. This uses a simple approach to search, but gives you a baseline you can iterate from. 10 | -------------------------------------------------------------------------------- /cohort_1/week1_bootstrap_evals/scoring_utils.py: -------------------------------------------------------------------------------- 1 | import cohere 2 | from diskcache import Cache 3 | import lancedb 4 | import os 5 | from typing import List, Dict 6 | 7 | from pydantic import BaseModel 8 | from concurrent.futures import ThreadPoolExecutor 9 | 10 | 11 | cohere_api_key = os.environ["COHERE_API_KEY"] 12 | 13 | 14 | class EvalQuestion(BaseModel): 15 | question: str 16 | answer: str 17 | chunk_id: str 18 | question_with_context: str 19 | 20 | 21 | def score(hits): 22 | n_retrieval_requests = len(hits) 23 | total_retrievals = sum(len(l) for l in hits) 24 | true_positives = sum(sum(sublist) for sublist in hits) 25 | precision = true_positives / total_retrievals if total_retrievals > 0 else 0 26 | recall = true_positives / n_retrieval_requests if n_retrieval_requests > 0 else 0 27 | return {"precision": precision, "recall": recall} 28 | 29 | 30 | def run_reranked_request( 31 | q: EvalQuestion, 32 | reviews_table: lancedb.table.LanceTable, 33 | max_n_return_vals: int, 34 | n_to_rerank: int = 40, 35 | model: str = "rerank-english-v3.0", 36 | ) -> List[bool]: 37 | cache = Cache("./cohere_cache") 38 | cache_key = f"{q.question_with_context}_{max_n_return_vals}_{model}".replace( 39 | "?", "" 40 | ) 41 | 42 | cached_result = cache.get(cache_key) 43 | if cached_result is not None: 44 | return cached_result 45 | 46 | initial_results = ( 47 | reviews_table.search(q.question_with_context) 48 | .select(["id", "review"]) 49 | .limit(n_to_rerank) 50 | .to_list() 51 | ) 52 | 53 | texts = [r["review"] for r in initial_results] 54 | 55 | # Rerank using Cohere 56 | co = cohere.Client(cohere_api_key) 57 | reranked = co.rerank( 58 | query=q.question_with_context, 59 | documents=texts, 60 | top_n=max_n_return_vals, 61 | model=model, 62 | ) 63 | 64 | # Map reranked results back to original IDs 65 | reranked_ids = [initial_results[r.index]["id"] for r in reranked.results] 66 | result = [str(q.chunk_id) == str(r) for r in reranked_ids] 67 | cache.set(cache_key, result) 68 | return result 69 | 70 | 71 | def score_reranked_search( 72 | eval_questions: List[EvalQuestion], 73 | reviews_table: lancedb.table.LanceTable, 74 | k_values: List[int], 75 | n_to_rerank: int = 40, 76 | model="rerank-english-v3.0", 77 | ) -> Dict[int, Dict[str, float]]: 78 | max_k = max(k_values) 79 | with ThreadPoolExecutor() as executor: 80 | all_hits = list( 81 | executor.map( 82 | lambda q: run_reranked_request( 83 | q, reviews_table, max_k, n_to_rerank, model 84 | ), 85 | eval_questions, 86 | ) 87 | ) 88 | 89 | results = {} 90 | for k in k_values: 91 | hits = [h[:k] for h in all_hits] 92 | results[k] = score(hits) 93 | 94 | return results 95 | -------------------------------------------------------------------------------- /cohort_1/week2_question_classification/classify_with_yaml/example.py: -------------------------------------------------------------------------------- 1 | import openai 2 | import instructor 3 | from yaml_classifier import YamlClassifier 4 | from pydantic import BaseModel, Field, field_validator, ValidationInfo 5 | from typing import List 6 | 7 | client = instructor.from_openai(openai.OpenAI()) 8 | 9 | classifier = YamlClassifier.load("example.yaml") 10 | 11 | 12 | class Prediction(BaseModel): 13 | correct_labels: List[str] = Field( 14 | description="The predicted label(s) as a list of strings" 15 | ) 16 | 17 | @field_validator("correct_labels") 18 | def validate_labels(cls, v, info: ValidationInfo): 19 | labels = info.context["labels"] 20 | for label in v: 21 | if label not in labels: 22 | raise ValueError(f"Label {label} not in {labels}") 23 | return v 24 | 25 | 26 | class PredictionWithReasoning(Prediction): 27 | reasoning: str = Field( 28 | description="A detailed explanation of the thought process leading to the prediction, including key factors considered, comparisons to label descriptions and examples, and how the query's content and intent align with the chosen label" 29 | ) 30 | 31 | 32 | # Example without reasoning 33 | 34 | resp = classifier.predict( 35 | query="When was the last time i ask you about dinner?", 36 | response_model=Prediction, 37 | client=client, 38 | model="gpt-4o-mini", 39 | ) 40 | 41 | print(resp.model_dump_json(indent=2)) 42 | # > {"correct_labels": ["time_filter_requirement"]} 43 | 44 | # Example with batch prediction with asyncio 45 | from asyncio import run, gather 46 | 47 | client = instructor.from_openai(openai.AsyncOpenAI()) 48 | 49 | examples = [ 50 | "When was the last time I asked you about dinner?", 51 | "Can you draft an email to my team about the project deadline?", 52 | "Do I have permission to access the financial reports?", 53 | "I need to draft a report on our current stock prices for the board meeting tomorrow.", 54 | "Can you give me access to the server room and write an email to IT about it?", 55 | "What were yesterday's headlines about our company's financial performance?", 56 | ] 57 | 58 | 59 | async def run_predictions(): 60 | tasks = [ 61 | classifier.apredict( 62 | client=client, # type: ignore 63 | model="gpt-4o-mini", 64 | query=query, 65 | response_model=Prediction, 66 | ) 67 | for query in examples 68 | ] 69 | return await gather(*tasks) 70 | 71 | 72 | resp = run(run_predictions()) 73 | 74 | for r in resp: 75 | print(r.model_dump_json(indent=2)) 76 | -------------------------------------------------------------------------------- /cohort_1/week2_question_classification/classify_with_yaml/example.yaml: -------------------------------------------------------------------------------- 1 | task: query_type_classification 2 | description: > 3 | Classify user queries into predefined categories based on the type of task requested. 4 | This classification system aims to categorize incoming user queries into specific task types, 5 | such as authorization requests, content drafting, or time-sensitive information retrieval. 6 | 7 | The purpose is to streamline query processing, improve response accuracy, and potentially 8 | route queries to appropriate handling systems or personnel. This classification helps in 9 | prioritizing tasks, applying relevant security measures, and ensuring that time-sensitive 10 | queries are handled promptly. 11 | 12 | The system uses a set of predefined labels, each with a clear description and examples, 13 | to accurately categorize a wide range of user inputs. This approach enables efficient 14 | query handling and enhances the overall user experience by providing more targeted and 15 | timely responses to various types of requests. 16 | 17 | # For each label we show the top `n_examples` examples automatically 18 | n_examples: 2 19 | # Number of similar examples to fetch if we use RAGClassifier vs YAMLClassifer 20 | n_similar_examples: 2 21 | 22 | labels: 23 | - name: authorization 24 | description: Queries related to granting or verifying access permissions 25 | examples: 26 | positive: 27 | - "Can you give me access to the financial reports?" 28 | - "I need permission to edit the shared document." 29 | negative: 30 | - "What's the weather like today?" 31 | - "Can you draft an email for me?" 32 | 33 | - name: drafting 34 | description: Queries requesting the creation or composition of content 35 | examples: 36 | positive: 37 | - "Write a summary of the quarterly report." 38 | - "Can you draft a reply to this customer complaint?" 39 | negative: 40 | - "What's the capital of France?" 41 | - "Do I have permission to access the server room?" 42 | 43 | - name: time_filter_requirement 44 | description: Queries that may need to be filtered based on time-sensitive information 45 | examples: 46 | positive: 47 | - "What are the current stock prices for Tech Co.?" 48 | - "Give me today's headlines from major news outlets." 49 | negative: 50 | - "Who was the first president of the United States?" 51 | - "What's the chemical formula for water?" -------------------------------------------------------------------------------- /cohort_1/week2_question_classification/classify_with_yaml/rag_classifer.py: -------------------------------------------------------------------------------- 1 | import chromadb 2 | from jinja2 import Template 3 | from yaml_classifier import YamlClassifier 4 | from textwrap import dedent 5 | from pydantic import Field 6 | import chromadb.utils.embedding_functions as embedding_functions 7 | import os 8 | from pydantic import Field 9 | import instructor 10 | import openai 11 | 12 | 13 | class RAGClassifier(YamlClassifier): 14 | fetch_n_examples: int | None = Field(default=2) 15 | __db = None 16 | 17 | def load_db(self, collection_name: str): 18 | if collection_name: 19 | chroma_client = chromadb.Client() 20 | self.__db = chroma_client.get_or_create_collection( 21 | name=collection_name, 22 | embedding_function=self.get_embedding_function(), 23 | ) 24 | return self 25 | 26 | def get_embedding_function(self): 27 | return embedding_functions.OpenAIEmbeddingFunction( 28 | api_key=os.environ.get("OPENAI_API_KEY"), 29 | model_name="text-embedding-3-small", 30 | ) 31 | 32 | def fit(self, collection_name: str): 33 | chroma_client = chromadb.Client() 34 | 35 | self.__db = chroma_client.get_or_create_collection( 36 | name=collection_name, embedding_function=self.get_embedding_function() 37 | ) 38 | 39 | all_examples = [] 40 | for label in self.labels: 41 | for example in label.examples.positive: 42 | all_examples.append({"text": example, "label": label.name}) 43 | for example in label.examples.negative: 44 | all_examples.append({"text": example, "label": label.name}) 45 | 46 | # Upsert all examples into the database 47 | self.__db.upsert( 48 | documents=[example["text"] for example in all_examples], 49 | ids=[str(i) for i in range(len(all_examples))], 50 | metadatas=[{"label": example["label"]} for example in all_examples], 51 | ) 52 | 53 | def get_user_query(self, query: str) -> str: 54 | if self.__db is None: 55 | raise ValueError("Database not initialized, call cls.fit() first") 56 | results = self.__db.query( 57 | query_texts=[query], 58 | n_results=self.fetch_n_examples, 59 | ) 60 | formatted_results = [ 61 | (doc, metadata["label"], distance) 62 | for doc, metadata, distance in zip( 63 | results["documents"][0], 64 | results["metadatas"][0], 65 | results["distances"][0], 66 | ) 67 | ] 68 | 69 | template = Template( 70 | dedent( 71 | """ 72 | Classify the following document: 73 | 74 | 75 | {{ query }} 76 | 77 | 78 | Similar examples: 79 | 80 | {% for doc, label, distance in formatted_results %} 81 | 82 | {{ "%.2f"|format(distance) }} 83 | 84 | {{ doc }} 85 | 86 | {% endfor %} 87 | 88 | 89 | Provide your classification based on the above information. 90 | """ 91 | ) 92 | ) 93 | 94 | return template.render(query=query, formatted_results=formatted_results) 95 | 96 | 97 | if __name__ == "__main__": 98 | client = instructor.from_openai(openai.OpenAI()) 99 | 100 | classifier = RAGClassifier.load("example.yaml") 101 | classifier.fit("example") 102 | # classifier.load_db("example") 103 | 104 | print("# Example of User Query") 105 | print( 106 | classifier.get_user_query( 107 | "When can i expect to see the next episode of the show?" 108 | ) 109 | ) 110 | -------------------------------------------------------------------------------- /cohort_1/week2_question_classification/classify_with_yaml/readme.md: -------------------------------------------------------------------------------- 1 | # Classification Module 2 | 3 | This module provides two main classifiers for text classification tasks: `YamlClassifier` and `RAGClassifier`. Both classifiers are designed to categorize user queries into predefined labels based on a set of examples and descriptions. 4 | 5 | ## YamlClassifier 6 | 7 | The `YamlClassifier` is a simple yet powerful classifier that uses a YAML configuration file to define the classification task, labels, and examples. 8 | 9 | ### Features 10 | 11 | - Load classification configuration from a YAML file 12 | - Support for multiple labels with positive and negative examples 13 | - Customizable number of examples to use for each label 14 | - Synchronous and asynchronous prediction methods 15 | 16 | ## RAGClassifier 17 | 18 | > Before running the `RAGClassifier` script, make sure to set an OPENAI_API_KEY variable in your shell. 19 | 20 | The `RAGClassifier` is a classifier that uses a retreival model to classify user queries into predefined labels based on a set of examples and descriptions. 21 | 22 | ### Features 23 | 24 | - Inherits all features from YamlClassifier 25 | - Uses a vector database (ChromaDB) to store and retrieve similar examples 26 | - Dynamically fetches similar examples for each query during classification 27 | - Customizable number of similar examples to fetch (default: 2) 28 | - Provides distance metrics for retrieved examples to aid in classification 29 | - Supports fitting the classifier with examples and loading pre-fitted databases 30 | -------------------------------------------------------------------------------- /cohort_1/week2_question_classification/classify_with_yaml/requirements.txt: -------------------------------------------------------------------------------- 1 | instructor==1.3.7 2 | openai==1.38.0 3 | jinja2==3.1.4 4 | pyyaml==6.0.1 5 | chromadb==0.5.5 6 | 7 | -------------------------------------------------------------------------------- /cohort_1/week2_question_classification/classify_with_yaml/yaml_classifier.py: -------------------------------------------------------------------------------- 1 | from pydantic import BaseModel, Field 2 | from typing import List 3 | from pydantic import BaseModel, field_validator 4 | from instructor import Instructor, AsyncInstructor 5 | from jinja2 import Template 6 | from textwrap import dedent 7 | from typing import Type, TypeVar 8 | 9 | T = TypeVar("T", bound=BaseModel) 10 | 11 | 12 | class Example(BaseModel): 13 | positive: List[str] 14 | negative: List[str] 15 | 16 | 17 | class Label(BaseModel): 18 | name: str 19 | description: str 20 | examples: Example 21 | 22 | @field_validator("name") 23 | def name_must_be_snake_case(cls, v): 24 | import re 25 | 26 | if not re.match(r"^[a-z0-9_]+$", v): 27 | raise ValueError("Label name must be in snake_case") 28 | return v 29 | 30 | 31 | class YamlClassifier(BaseModel): 32 | task: str 33 | description: str 34 | labels: List[Label] 35 | n_examples: int | None = Field( 36 | default=100, description="Number of examples to use for each label" 37 | ) 38 | 39 | @classmethod 40 | def load(cls, fn: str): 41 | import yaml 42 | 43 | with open(fn, "r") as file: 44 | data = yaml.safe_load(file) 45 | 46 | return cls(**data) 47 | 48 | def to_system_messages(self) -> str: 49 | template = Template( 50 | dedent( 51 | """ 52 | 53 | {{ task }} 54 | 55 | 56 | 57 | {{ description }} 58 | 59 | 60 | 61 | {% for label in labels %} 62 | 89 | {% endfor %} 90 | 91 | 92 | Instructions: 93 | 1. Carefully read the user's query. 94 | 2. Compare the query to the descriptions and examples for each label. 95 | 3. Use the provided examples as a guide: 96 | - Positive examples show queries that should be classified under that label. 97 | - Negative examples show queries that should not be classified under that label. 98 | 4. Consider both the content and the intent of the query when matching to a label. 99 | 5. Choose the most appropriate label that matches the query's intent and content. 100 | 6. If the query doesn't clearly fit any label, choose the closest match based on similarity to the examples and description. 101 | 7. Provide your classification as a single word matching the chosen label's name. 102 | 8. Do not assume any specific task unless it's explicitly mentioned in the 'task' variable. 103 | """ 104 | ) 105 | ) 106 | return template.render(self.model_dump()) 107 | 108 | def get_user_query(self, query: str) -> str: 109 | return f"Correctly Classify:\n\n{query}" 110 | 111 | def get_labels(self) -> List[str]: 112 | return [label.name for label in self.labels] 113 | 114 | def set_client(self, client: Instructor): 115 | self._client = client 116 | 117 | def predict( 118 | self, query: str, model: str, response_model: Type[T], client: Instructor 119 | ): 120 | system_message = self.to_system_messages() 121 | user_query = self.get_user_query(query) 122 | return client.create( 123 | model=model, 124 | response_model=response_model, 125 | messages=[ 126 | {"role": "system", "content": system_message}, 127 | {"role": "user", "content": user_query}, 128 | ], 129 | validation_context={ 130 | "labels": self.get_labels(), 131 | }, 132 | ) 133 | 134 | async def apredict( 135 | self, query: str, model: str, response_model: Type[T], client: AsyncInstructor 136 | ): 137 | system_message = self.to_system_messages() 138 | user_query = self.get_user_query(query) 139 | return await client.create( 140 | messages=[ 141 | {"role": "system", "content": system_message}, 142 | {"role": "user", "content": user_query}, 143 | ], 144 | model=model, 145 | response_model=response_model, 146 | validation_context={ 147 | "labels": self.get_labels(), 148 | }, 149 | ) 150 | -------------------------------------------------------------------------------- /cohort_1/week2_question_classification/example_notebook/README.md: -------------------------------------------------------------------------------- 1 | This folder includes materials from week 2 of [Systematically Improving RAG Applications`](https://maven.com/applied-llms/rag-playbook). 2 | 3 | It includes two notebooks. The most important notebook is `analyze_clusters`. This notebook shows how to classify questions from your production app into topics/categories to inform where you focus your attention. You can repurpose this code to run on your own applications. 4 | 5 | Because you'll want data when first exploring this code, the notebook uses synthetic data that was generated in `make_prod_questions.ipynb`. This notebook will not be used in your production system. Instead, it is included only to show how we created some test data for the sake of example. 6 | 7 | `question_types.py` includes the some interface types (much of which is used in both notebooks.) 8 | -------------------------------------------------------------------------------- /cohort_1/week3_seperate_indices/README.md: -------------------------------------------------------------------------------- 1 | This folder shows how to add tools to extract new indices on your data. 2 | 3 | It continues the example of a hardware retailer building a system to answer user questions. We'll extract more data from the previous text sources as well as being able to support new data types (e.g. images). 4 | -------------------------------------------------------------------------------- /cohort_1/week4_routing/README.md: -------------------------------------------------------------------------------- 1 | This directory shows how to test whether we are retrieving the correct tools. It includes 2 | 3 | - `benchmark_tool_retrieval.ipynb`: Notebook showing the workflow 4 | - `utils.py`: Reusable/modifiable utility functions to calculate recall & precision of tools 5 | - `funcs_to_call.py`: Mocked objects showing the list of available functions. They are not implemented here, because we would not be testing the implementation anyway. We are just testing whether we can retrieve the correct tools. 6 | -------------------------------------------------------------------------------- /cohort_1/week4_routing/funcs_to_call.py: -------------------------------------------------------------------------------- 1 | from pydantic import BaseModel, Field 2 | from typing import Union 3 | 4 | 5 | class ShippingDateRequest(BaseModel): 6 | """Check when a product will be shipped""" 7 | 8 | call_name: str = "ShippingDateRequest" 9 | sku: str = Field(..., description="SKU of the product to check shipping date for") 10 | 11 | 12 | class ShippingCostRequest(BaseModel): 13 | """Check the cost of shipping a product""" 14 | 15 | sku: str = Field(..., description="SKU of the product to check shipping cost for") 16 | shipping_location: str = Field(..., description="Location to ship to") 17 | 18 | 19 | class ProductDimensionsRequest(BaseModel): 20 | """Check the dimensions of a product""" 21 | 22 | sku: str = Field(..., description="SKU of the product to check dimensions for") 23 | 24 | 25 | class PriceHistoryRequest(BaseModel): 26 | """Check the price history of a product (e.g. identifying historical price fluctuations)""" 27 | 28 | sku: str = Field(..., description="SKU of the product to check price history for") 29 | 30 | 31 | class ProductComparisonRequest(BaseModel): 32 | """Compare two products""" 33 | 34 | sku1: str = Field(..., description="SKU of the first product to compare") 35 | sku2: str = Field(..., description="SKU of the second product to compare") 36 | 37 | 38 | class LogDesiredFeatureRequest(BaseModel): 39 | """Record a user's desire for a certain product feature""" 40 | 41 | sku: str = Field(..., description="SKU of the product to log a desired feature for") 42 | user_id: str = Field(..., description="User ID to log the desired feature for") 43 | desired_feature: str = Field(..., description="Desired feature to log") 44 | 45 | 46 | class ExtractDataFromImageRequest(BaseModel): 47 | """Use our product images with multimodal llm to extract info about the product""" 48 | 49 | image_url: str = Field(..., description="URL of the image to examine") 50 | question: str = Field(..., description="Question to answer about the image") 51 | 52 | 53 | class ProductMaterialsRequest(BaseModel): 54 | """Check what materials a product is made of""" 55 | 56 | sku: str = Field(..., description="SKU of the product to check materials for") 57 | 58 | 59 | FunctionOption = Union[ 60 | ShippingDateRequest, 61 | ShippingCostRequest, 62 | ProductDimensionsRequest, 63 | PriceHistoryRequest, 64 | ProductComparisonRequest, 65 | LogDesiredFeatureRequest, 66 | ExtractDataFromImageRequest, 67 | ProductMaterialsRequest, 68 | ] 69 | -------------------------------------------------------------------------------- /cohort_1/week5/pyproject.toml: -------------------------------------------------------------------------------- 1 | [project] 2 | name = "week5" 3 | version = "0.1.0" 4 | description = "Add your description here" 5 | readme = "README.md" 6 | requires-python = ">=3.12" 7 | dependencies = [ 8 | "datasets>=3.1.0", 9 | "instructor>=1.7.0", 10 | "ipykernel>=6.29.5", 11 | "ipywidgets>=8.1.5", 12 | "matplotlib>=3.9.2", 13 | "openai>=1.55.2", 14 | "pillow>=11.0.0", 15 | ] 16 | -------------------------------------------------------------------------------- /cohort_1/week5_fine_tuning/README.md: -------------------------------------------------------------------------------- 1 | The files in this directory create a fine-tuned reranking model on cohere and then test recall on this fine-tuned model. 2 | 3 | You can compare it to the [results from week 1](https://github.com/567-labs/systematically-improving-rag/blob/main/week1_bootstrap_evals/metrics.ipynb) where we used a cohere reranker that had not been fine-tuned. 4 | 5 | The key files in this directory are: 6 | 7 | - `finetune_sbert.py`: Fine tune a sentence transformer cross-encoder. Run this before `eval_sbert.py`. 8 | - `eval_sbert.py`: Evaluate recall for both the base sentence transformer model and the fine-tuned model 9 | - `cohere_fine_tuning.ipynb`: Create a fine-tuned cohere model and test precision/recall 10 | -------------------------------------------------------------------------------- /cohort_1/week5_fine_tuning/eval_sbert.py: -------------------------------------------------------------------------------- 1 | import json 2 | import lancedb 3 | import numpy as np 4 | from pydantic import BaseModel 5 | from sentence_transformers.cross_encoder import CrossEncoder 6 | from typing import List, Tuple 7 | 8 | # Constants 9 | FIRST_STAGE_LIMIT = 50 10 | BASE_MODEL_PATH = "cross-encoder/stsb-distilroberta-base" 11 | FINE_TUNED_MODEL_PATH = "./fine_tuned_reranker" 12 | 13 | 14 | class EvalQuestion(BaseModel): 15 | question: str 16 | answer: str 17 | chunk_id: str 18 | question_with_context: str 19 | 20 | 21 | # Load data and setup 22 | with open("../week1_bootstrap_evals/synthetic_eval_dataset.json", "r") as f: 23 | synthetic_questions = json.load(f) 24 | eval_questions = [EvalQuestion(**q) for q in synthetic_questions] 25 | 26 | db = lancedb.connect("../week1_bootstrap_evals/lancedb") 27 | reviews_table = db.open_table("reviews") 28 | 29 | 30 | def score_question(eval_question: EvalQuestion, model: CrossEncoder) -> float: 31 | """ 32 | Score a question using the given model. 33 | 34 | Args: 35 | eval_question (EvalQuestion): The question to evaluate. 36 | model (CrossEncoder): The model to use for ranking. 37 | 38 | Returns: 39 | float: The rank of the desired result, or inf if not found. 40 | """ 41 | query = f"Answer the following question: {eval_question.question_with_context}\n." 42 | target_id = int(eval_question.chunk_id) 43 | first_stage = reviews_table.search(query).limit(FIRST_STAGE_LIMIT).to_pandas() 44 | first_stage_ids = first_stage.id.astype(int).values 45 | review_text = first_stage.review.values 46 | reranked_results = model.rank(query, review_text) 47 | is_right_result = lambda x: first_stage_ids[x["corpus_id"]] == target_id 48 | try: 49 | rank_of_desired_result = next( 50 | i + 1 for i, d in enumerate(reranked_results) if is_right_result(d) 51 | ) 52 | except StopIteration: 53 | return np.inf 54 | return rank_of_desired_result 55 | 56 | 57 | def mean_reciprocal_rank(ranks: List[float]) -> float: 58 | """ 59 | Calculate the Mean Reciprocal Rank (MRR) from a list of ranks. 60 | 61 | Args: 62 | ranks (List[float]): List of ranks for each query. 63 | 64 | Returns: 65 | float: The Mean Reciprocal Rank. 66 | """ 67 | reciprocal_ranks = [1 / rank if rank != np.inf else 0 for rank in ranks] 68 | return np.mean(reciprocal_ranks) 69 | 70 | 71 | def evaluate_model( 72 | model: CrossEncoder, model_name: str 73 | ) -> Tuple[List[float], float, float]: 74 | """ 75 | Evaluate a model on the evaluation questions. 76 | 77 | Args: 78 | model (CrossEncoder): The model to evaluate. 79 | model_name (str): The name of the model for printing results. 80 | 81 | Returns: 82 | Tuple[List[float], float, float]: Ranks, recall at 5, and recall at 10. 83 | """ 84 | ranks = [score_question(eval_question, model) for eval_question in eval_questions] 85 | recall_at_5 = np.mean([rank <= 5 for rank in ranks]) 86 | recall_at_10 = np.mean([rank <= 10 for rank in ranks]) 87 | mrr = mean_reciprocal_rank(ranks) 88 | 89 | print(f"{model_name} results:") 90 | print(f"Recall at 5: {recall_at_5}") 91 | print(f"Recall at 10: {recall_at_10}") 92 | print(f"Mean Reciprocal Rank: {mrr:.4f}") 93 | 94 | return ranks, recall_at_5, recall_at_10, mrr 95 | 96 | 97 | # Evaluate base model 98 | base_model = CrossEncoder(BASE_MODEL_PATH) 99 | base_ranks, base_recall_at_5, base_recall_at_10, base_mrr = evaluate_model( 100 | base_model, "Base model" 101 | ) 102 | 103 | # Evaluate fine-tuned model 104 | fine_tuned_model = CrossEncoder(FINE_TUNED_MODEL_PATH) 105 | fine_tuned_ranks, fine_tuned_recall_at_5, fine_tuned_recall_at_10, fine_tuned_mrr = ( 106 | evaluate_model(fine_tuned_model, "Fine-tuned model") 107 | ) 108 | -------------------------------------------------------------------------------- /cohort_1/week5_fine_tuning/finetune_sbert.py: -------------------------------------------------------------------------------- 1 | import json 2 | from sentence_transformers import CrossEncoder, InputExample 3 | from torch.utils.data import DataLoader 4 | import lancedb 5 | from random import randint 6 | from torch import nn 7 | from collections import deque 8 | 9 | db = lancedb.connect("../week1_bootstrap_evals/lancedb") 10 | reviews_df = db.open_table("reviews").to_pandas() 11 | reviews_df.id = reviews_df.id.astype(int) 12 | 13 | with open("./ft_dataset.jsonl", "r") as f: 14 | finetune_data = [json.loads(line) for line in f] 15 | 16 | neg_examples_per_q = 3 17 | 18 | prepared_data = [] 19 | for item in finetune_data: 20 | query = item["question_with_context"] 21 | chunk_id = int(item["chunk_id"]) 22 | 23 | # Fetch the relevant passage from the reviews table 24 | relevant_review = reviews_df.loc[chunk_id].review 25 | 26 | prepared_data.append( 27 | { 28 | "query": query, 29 | "positive_passage": relevant_review, 30 | "negative_passages": [], 31 | } 32 | ) 33 | 34 | # Add negative examples 35 | for _ in range(neg_examples_per_q): 36 | random_chunk_id = randint(0, reviews_df.id.max()) 37 | if random_chunk_id != chunk_id: 38 | random_passage = reviews_df.loc[random_chunk_id].review 39 | prepared_data[-1]["negative_passages"].append(random_passage) 40 | 41 | # Prepare training examples 42 | train_examples = [] 43 | for item in prepared_data: 44 | train_examples.append( 45 | InputExample(texts=[item["query"], item["positive_passage"]], label=1.0) 46 | ) 47 | for negative_passage in item["negative_passages"]: 48 | train_examples.append( 49 | InputExample(texts=[item["query"], negative_passage], label=0.0) 50 | ) 51 | 52 | print(f"Created {len(train_examples)} training examples") 53 | 54 | # Initialize the model 55 | model = CrossEncoder("cross-encoder/stsb-distilroberta-base", num_labels=1) 56 | train_dataloader = DataLoader(train_examples, shuffle=True, batch_size=16) 57 | 58 | # Train the model 59 | output_path = "./fine_tuned_reranker" 60 | num_epochs = 1 61 | print_loss_freq = 20 62 | 63 | 64 | # Use custom loss function to help track loss while using SBERT 65 | class CustomMSELoss(nn.Module): 66 | def __init__(self): 67 | super().__init__() 68 | self.mse = nn.MSELoss() 69 | # store losses to average when we print loss 70 | self.losses = deque(maxlen=print_loss_freq) 71 | self.batch_count = 0 72 | 73 | def forward(self, predictions, labels): 74 | loss = self.mse(predictions.view(-1), labels.view(-1)) 75 | self.losses.append(loss.item()) 76 | self.batch_count += 1 77 | 78 | if self.batch_count % print_loss_freq == 0: 79 | avg_loss = sum(self.losses) / len(self.losses) 80 | print( 81 | f"\nBatch {self.batch_count}, Average Loss (last {print_loss_freq} batches): {avg_loss:.4f}" 82 | ) 83 | 84 | return loss 85 | 86 | 87 | custom_loss = CustomMSELoss() 88 | 89 | model.fit( 90 | train_dataloader=train_dataloader, 91 | epochs=num_epochs, 92 | warmup_steps=10, 93 | optimizer_params={"lr": 2e-5}, 94 | show_progress_bar=True, 95 | loss_fct=custom_loss, 96 | ) 97 | 98 | model.save(output_path) 99 | print(f"Fine-tuning completed. Model saved to '{output_path}'.") 100 | -------------------------------------------------------------------------------- /cohort_2/.env.example: -------------------------------------------------------------------------------- 1 | OPENAI_API_KEY= 2 | COHERE_API_KEY= 3 | HF_TOKEN= 4 | 5 | # Sign up for either Braintrust and/or Logfire. We have notebooks that walk through the same content which use both platforms. Signing up for one is enough 6 | BRAINTRUST_API_KEY= 7 | LOGFIRE_TOKEN= 8 | LOGFIRE_READ_TOKEN= -------------------------------------------------------------------------------- /cohort_2/.gitignore: -------------------------------------------------------------------------------- 1 | .venv 2 | __pycache__ 3 | .ipynb_checkpoints 4 | **/lancedb/ 5 | .env -------------------------------------------------------------------------------- /cohort_2/convert.py: -------------------------------------------------------------------------------- 1 | import os 2 | from pathlib import Path 3 | from typing import List 4 | import typer 5 | from rich import print 6 | from rich.progress import track 7 | 8 | # Add nbformat as an optional import since it may not be installed yet 9 | try: 10 | import nbformat 11 | except ImportError: 12 | print( 13 | "[yellow]Warning: nbformat not installed. Please install with 'pip install nbformat'[/yellow]" 14 | ) 15 | exit(1) 16 | 17 | 18 | def get_week_and_name(notebook_path: Path) -> tuple[str, str]: 19 | """ 20 | Extract week number and notebook name from path. 21 | 22 | Args: 23 | notebook_path: Path object for the notebook 24 | 25 | Returns: 26 | Tuple of (week_number, notebook_name) 27 | """ 28 | parts = notebook_path.parts 29 | week = "unknown" 30 | for part in parts: 31 | if part.startswith("week"): 32 | week = part.replace("week", "") 33 | break 34 | 35 | name = notebook_path.stem 36 | return week, name 37 | 38 | 39 | def convert_notebook_to_md(notebook_path: str, output_path: str) -> None: 40 | """ 41 | Convert a Jupyter notebook to markdown format preserving code, outputs and markdown cells. 42 | 43 | Args: 44 | notebook_path: Path to the input notebook file 45 | output_path: Path where markdown file should be saved 46 | """ 47 | # Create md directory if it doesn't exist 48 | os.makedirs(os.path.dirname(output_path), exist_ok=True) 49 | 50 | # Read the notebook 51 | with open(notebook_path) as f: 52 | nb = nbformat.read(f, as_version=4) 53 | 54 | md_content = [] 55 | 56 | # Convert each cell 57 | for cell in nb.cells: 58 | if cell.cell_type == "markdown": 59 | md_content.append(cell.source + "\n") 60 | 61 | elif cell.cell_type == "code": 62 | # Add code block 63 | md_content.append(f"```python\n{cell.source}\n```\n") 64 | # Add outputs if present 65 | if cell.outputs: 66 | md_content.append("\n") 67 | for output in cell.outputs: 68 | if "text" in output: 69 | md_content.append("```\n" + output.text + "\n```\n") 70 | elif "data" in output: 71 | if "text/plain" in output.data: 72 | md_content.append( 73 | "```\n" + str(output.data["text/plain"]) + "\n```\n" 74 | ) 75 | md_content.append("\n") 76 | 77 | # Write markdown file 78 | with open(output_path, "w") as f: 79 | f.write("\n".join(md_content)) 80 | 81 | 82 | def find_notebooks(root_dir: str) -> List[Path]: 83 | """ 84 | Find all Jupyter notebooks in a directory and its subdirectories. 85 | 86 | Args: 87 | root_dir: Root directory to search for notebooks 88 | 89 | Returns: 90 | List of paths to notebook files 91 | """ 92 | notebooks = [] 93 | for path in Path(root_dir).rglob("*.ipynb"): 94 | if not ".ipynb_checkpoints" in str(path): 95 | notebooks.append(path) 96 | return notebooks 97 | 98 | 99 | app = typer.Typer(help="Convert Jupyter notebooks to markdown files") 100 | 101 | 102 | @app.command() 103 | def convert( 104 | directory: str = typer.Argument(".", help="Root directory to search for notebooks"), 105 | dry_run: bool = typer.Option( 106 | False, 107 | "--dry-run", 108 | "-d", 109 | help="Show which files would be converted without actually converting", 110 | ), 111 | ) -> None: 112 | """ 113 | Convert all Jupyter notebooks in a directory to markdown format. 114 | Preserves code cells, outputs, and markdown content. Saves all files in md/ folder. 115 | """ 116 | root_dir = os.path.abspath(directory) 117 | notebooks = find_notebooks(root_dir) 118 | 119 | if not notebooks: 120 | print("[yellow]No notebooks found in the specified directory[/yellow]") 121 | raise typer.Exit() 122 | 123 | print(f"[green]Found {len(notebooks)} notebooks to convert[/green]") 124 | 125 | if dry_run: 126 | for nb_path in notebooks: 127 | week, name = get_week_and_name(nb_path) 128 | md_path = Path("md") / f"week{week}-{name}.md" 129 | print(f"Would convert {nb_path} to {md_path}") 130 | return 131 | 132 | # Convert each notebook with progress bar 133 | for nb_path in track(notebooks, description="Converting notebooks..."): 134 | week, name = get_week_and_name(nb_path) 135 | md_path = Path("md") / f"week{week}-{name}.md" 136 | print(f"Converting {nb_path} to {md_path}") 137 | convert_notebook_to_md(str(nb_path), str(md_path)) 138 | 139 | print("[green]Conversion complete![/green]") 140 | 141 | 142 | if __name__ == "__main__": 143 | app() 144 | -------------------------------------------------------------------------------- /cohort_2/office-hours/README.md: -------------------------------------------------------------------------------- 1 | # Cohort 2 Office Hours Transcript Organization 2 | 3 | This directory contains organized transcripts from the Cohort 2 office hours sessions. 4 | 5 | ## Organization Structure 6 | 7 | Transcripts are organized into weekly folders based on session dates: 8 | 9 | - `week1/`: February 4 (Tuesday) and February 6 (Thursday), 2025 10 | - `week2/`: February 11 (Tuesday) and February 13 (Thursday), 2025 11 | - `week3/`: February 18 (Tuesday) and February 20 (Thursday), 2025 12 | - `week4/`: February 25 (Tuesday) and February 27 (Thursday), 2025 13 | - `week5/`: March 4 (Tuesday) and March 6 (Thursday), 2025 14 | - `week6/`: March 11 (Tuesday) and March 13 (Thursday), 2025 15 | - `other/`: Files from non-Tuesday/Thursday dates or files without date information 16 | 17 | ## Generating Summary Files Using Cursor 18 | 19 | You can use Cursor's AI capabilities to generate summary files from the transcript files. These summaries organize the content into a Q&A format for easier reference. 20 | 21 | ### Steps to Generate a Summary: 22 | 23 | 1. **Open Cursor and navigate to the repository** 24 | 25 | 2. **Open the transcript files for the week you want to summarize** 26 | - For example, to create a Week 1 summary, open the transcript files from the `week1/` folder 27 | 28 | 3. **Create a new file** 29 | - Create `week#-summary.md` in the office-hours directory (e.g., `week1-summary.md`) 30 | 31 | 4. **Use Cursor's AI composer (CTRL+K or CMD+K)** 32 | - In the composer, enter a prompt like: 33 | ``` 34 | @[transcript-file-1] @[transcript-file-2] @[transcript-file-3] 35 | 36 | Create a summary file that extracts all Q&A pairs from these transcripts in the same format as the existing summary files. 37 | ``` 38 | - For example: 39 | ``` 40 | @02-04-2025-1353-merged.txt @02-04-2025-1744-merged.txt @02-06-2025-1854-merged.txt 41 | 42 | Create a summary file that extracts all Q&A pairs from these transcripts in the same format as week3-summary.md 43 | ``` 44 | 45 | 5. **Review and edit the generated content** 46 | - Cursor may need multiple attempts to generate all content 47 | - You can ask it to "check if anything is missing and add that back" to ensure comprehensive coverage 48 | 49 | 6. **Save the file** 50 | - The completed summary will be saved in the format consistent with other summary files 51 | 52 | ### Tips for Better Summaries: 53 | 54 | - Reference an existing summary file (like `week3-summary.md`) as a format example 55 | - Be specific about including all questions and answers from the transcripts 56 | - If the summary seems incomplete, ask Cursor to review specific portions of the transcripts again 57 | - Organize questions thematically when possible for easier reference 58 | 59 | ## File Naming Convention 60 | 61 | Files are renamed using a consistent format: 62 | 63 | ``` 64 | MM-DD-YYYY-HHMM-type.ext 65 | ``` 66 | 67 | Where: 68 | - `MM-DD-YYYY`: Month, day, and year of the session 69 | - `HHMM`: Hour and minute of the session (24-hour format) 70 | - `type`: Type of file (session, merged, chat) 71 | - `ext`: File extension (.vtt, .txt, etc.) 72 | 73 | Examples: 74 | - `02-04-2025-1353-session.vtt`: Transcript from February 4, 2025 at 1:53 PM 75 | - `02-18-2025-1349-merged.txt`: Merged transcript from February 18, 2025 at 1:49 PM 76 | - `02-20-2025-1857-session.txt`: Transcript from February 20, 2025 at 6:57 PM 77 | 78 | ## Types of Files 79 | 80 | - **session**: Original recording transcripts (VTT or TXT format) 81 | - **chat**: Chat messages from the session 82 | - **merged**: Combined transcript of recording and chat 83 | 84 | ## Using the Transcript Organization Script 85 | 86 | The `move-files.py` script handles organizing and renaming transcript files. It: 87 | 88 | 1. Checks both current directory and Downloads folder for transcript files 89 | 2. Extracts date and time information from filenames 90 | 3. Determines which week each file belongs to 91 | 4. Renames files to the consistent naming format 92 | 5. Moves files to the appropriate week folder 93 | 6. Places non-Tuesday/Thursday files in the "other" folder 94 | 95 | To run the script: 96 | 97 | ```bash 98 | python3 move-files.py 99 | ``` 100 | 101 | The script will: 102 | - Show which files were found and where they were moved 103 | - Remove duplicate files 104 | - Print a summary of files in each week folder 105 | 106 | ## Adding New Transcripts 107 | 108 | When new transcripts are downloaded: 109 | 110 | 1. Save them to your Downloads folder 111 | 2. Run the `move-files.py` script 112 | 3. The script will automatically organize and rename all new transcript files 113 | 114 | The script handles various transcript file formats and naming patterns, including: 115 | - Files with "transcript" in the name 116 | - Files with "recording" in the name (ending with .vtt, .txt, .srt) 117 | - Files starting with "GMT" followed by a date (ending with .vtt, .txt, .srt) -------------------------------------------------------------------------------- /cohort_2/pyproject.toml: -------------------------------------------------------------------------------- 1 | [project] 2 | name = "systematically-improving-rag" 3 | version = "0.1.0" 4 | description = "Systematically Improve Your RAG System with our interactive notebooks!" 5 | readme = "README.md" 6 | requires-python = ">=3.9,<3.10" 7 | dependencies = [ 8 | "accelerate==1.3.0", 9 | "anthropic>=0.40.0", 10 | "bertopic>=0.16.4", 11 | "braintrust>=0.0.177", 12 | "bs4>=0.0.2", 13 | "cohere>=5.11.3", 14 | "datasets>=3.1.0", 15 | "faker>=33.1.0", 16 | "thefuzz==0.22.1", 17 | "ipykernel>=6.29.5", 18 | "ipywidgets>=8.1.5", 19 | "jsonref>=1.1.0", 20 | "lancedb>=0.16.0", 21 | "markdownify>=0.13.1", 22 | "matplotlib>=3.9.2", 23 | "modal>=0.65.38", 24 | "nbformat>=5.10.4", 25 | "open-clip-torch>=2.29.0", 26 | "openai>=1.57.0", 27 | "pandas>=2.2.3", 28 | "pillow>=11.0.0", 29 | "pydantic>=2.10.1", 30 | "python-levenshtein>=0.26.1", 31 | "pyyaml>=6.0.2", 32 | "requests>=2.32.3", 33 | "scikit-learn>=1.6.0", 34 | "instructor==1.7.2", 35 | "seaborn>=0.13.2", 36 | "sentence-transformers==3.1.1", 37 | "streamlit>=1.40.1", 38 | "streamlit-shortcuts>=0.1.9", 39 | "tantivy>=0.22.0", 40 | "transformers==4.42.2", 41 | "eval_type_backport==0.2.2", 42 | "pydantic-evals[logfire]>=0.1.3" 43 | "docling==2.30.0", 44 | "pdf2image==1.17.0" 45 | ] 46 | 47 | [build-system] 48 | requires = ["setuptools>=61.0"] 49 | build-backend = "setuptools.build_meta" 50 | -------------------------------------------------------------------------------- /cohort_2/requirements.txt: -------------------------------------------------------------------------------- 1 | accelerate==1.3.0 2 | anthropic>=0.40.0 3 | bertopic>=0.16.4 4 | braintrust>=0.0.177 5 | bs4>=0.0.2 6 | cohere>=5.11.3 7 | datasets>=3.1.0 8 | faker>=33.1.0 9 | fuzzywuzzy>=0.18.0 10 | ipykernel>=6.29.5 11 | ipywidgets>=8.1.5 12 | jsonref>=1.1.0 13 | lancedb>=0.16.0 14 | markdownify>=0.13.1 15 | matplotlib>=3.9.2 16 | modal>=0.65.38 17 | nbformat>=5.10.4 18 | open-clip-torch>=2.29.0 19 | openai>=1.57.0 20 | pandas>=2.2.3 21 | pillow>=11.0.0 22 | pydantic>=2.10.1 23 | python-levenshtein>=0.26.1 24 | pyyaml>=6.0.2 25 | requests>=2.32.3 26 | scikit-learn>=1.6.0 27 | instructor==1.7.2 28 | seaborn>=0.13.2 29 | sentence-transformers==3.1.1 30 | streamlit>=1.40.1 31 | streamlit-shortcuts>=0.1.9 32 | tantivy>=0.22.0 33 | transformers==4.45.2 34 | eval_type_backport==0.2.2 35 | docling==2.25.0 -------------------------------------------------------------------------------- /cohort_2/week0/README.md: -------------------------------------------------------------------------------- 1 | # Week 0: Foundation and Environment Setup 2 | 3 | ## Notebooks Overview 4 | 5 | ### 1. Using Jupyter Notebooks.ipynb 6 | 7 | This notebook introduces the fundamental tool you'll use throughout the course: Jupyter Notebooks. 8 | 9 | **What You'll Do:** 10 | 11 | - Run basic Python code in interactive cells 12 | - Execute shell commands directly within notebooks 13 | - Create and visualize pandas DataFrames with sample sales data 14 | - Learn about kernels and how to select the right one (using kernel.png as reference) 15 | - Use magic commands like `%%time` to measure performance 16 | - Configure the `autoreload` extension for smoother development 17 | 18 | This hands-on introduction ensures you're comfortable with the notebook environment before diving into more complex RAG concepts. 19 | 20 | ### 2. LanceDB.ipynb 21 | 22 | This notebook introduces vector databases with LanceDB, a crucial component for building RAG systems. 23 | 24 | **What You'll Do:** 25 | 26 | - Define schemas with Pydantic for consistent data structures 27 | - Create and populate LanceDB tables with document data 28 | - Implement three key search strategies: 29 | 1. Vector search using OpenAI embeddings 30 | 2. Full-text search with inverted indices via tantivy 31 | 3. Hybrid search combining both approaches 32 | - Apply Cohere reranking to improve result quality 33 | - Compare search results for different query types 34 | - Analyze when each search approach performs best 35 | 36 | You'll gain hands-on experience with the retrieval foundations that power all RAG applications. 37 | 38 | ### 3. Using Pydantic Evals.ipynb 39 | 40 | This notebook introduces Pydantic Evals, a framework for systematically evaluating AI systems and tracking results. 41 | 42 | **What You'll Do:** 43 | 44 | - Configure Logfire for tracking evaluation results 45 | - Create your first evaluation with test cases, datasets, and evaluators 46 | - Build custom evaluators to assess model outputs 47 | - Evaluate an LLM-based classification system for customer queries 48 | - Save and load evaluation datasets for consistent testing 49 | - Track performance metrics across different test cases 50 | 51 | You'll establish a foundation for systematically measuring model performance, which is essential for improving RAG systems throughout the course. 52 | 53 | ## Technical Requirements 54 | 55 | - Python 3.9 56 | - Required libraries: pandas, matplotlib, lancedb, openai, tantivy, cohere 57 | - API keys for OpenAI (embeddings) and Cohere (reranking) 58 | 59 | ## Why These Notebooks Matter 60 | 61 | These notebooks provide the essential tools and concepts you'll use throughout the course. The Jupyter environment will be your workspace for all experiments, while LanceDB introduces the vector search capabilities that form the backbone of RAG systems. Mastering these fundamentals prepares you for systematic RAG improvement in the following weeks. 62 | -------------------------------------------------------------------------------- /cohort_2/week0/assets/kernel.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/567-labs/systematically-improving-rag/32241e32e571bb7e34c257c143cc24f117f73f79/cohort_2/week0/assets/kernel.png -------------------------------------------------------------------------------- /cohort_2/week0/uv.md: -------------------------------------------------------------------------------- 1 | # Virtual Environments 2 | 3 | When working through the different notebooks in this course, we highly recommend using virtual environments to manage dependencies. This is because of the following reasons 4 | 5 | 1. **Isolation**: Virtual environments allow you to isolate depencencies easily. This is useful when you'd like to share your code with others. 6 | 2. **Easy Cleanup** : If you'd like to delete the environment, you can simply delete the directory. This is not the case when you've messed up your system Python installation. 7 | 8 | By using virtual environments, we can ensure that the dependencies are the same across different machines and help prevent a host of other issues. 9 | 10 | ## Why `uv` 11 | 12 | `uv` is a new package manager which is faster and more efficient than traditional tools like `pip`. It is also tightly integrated with virtual environments, making it easier to create and manage them. 13 | 14 | We recommend checking out their [docs here](https://docs.astral.sh/uv/) for the latest information. In this portion, we'll show a few common commands that you'll need to get started. 15 | 16 | With its powerful features, `uv` provides several key benefits: 17 | 18 | - **Lightning Fast Performance**: `uv` is 10-100x faster than pip, making dependency installation and management significantly more efficient 19 | - **All-in-One Solution**: Replaces multiple tools like pip, poetry, virtualenv and more with a single unified interface 20 | - **Space Efficient**: Uses a global cache to deduplicate dependencies across projects, saving disk space while maintaining isolation 21 | 22 | ## Installation 23 | 24 | For MacOS and Linux, you can install `uv` using the following command: 25 | 26 | ```bash 27 | curl -LsSf https://astral.sh/uv/install.sh | sh 28 | ``` 29 | 30 | For Windows, you can use the following command: 31 | 32 | ```bash 33 | powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex" 34 | ``` 35 | 36 | Once you've done so, you'll be able to use the `uv` command in your terminal. 37 | 38 | ## Creating a virtual environment 39 | 40 | To create a virtual environment, you can use the following command: 41 | 42 | ```bash 43 | uv venv 44 | ``` 45 | 46 | This will in turn create a new virtual environment in the `.venv` directory. We can also specify a specific python version to use for the environment. 47 | 48 | ```bash 49 | uv venv --python 3.10 50 | ``` 51 | 52 | If you'd like to also name your virtual environment something other than `.venv`, you can do so by specifying the name as follows: 53 | 54 | ```bash 55 | uv venv my-env 56 | ``` 57 | 58 | This will create a virtual environment called `my-env` in the current directory. 59 | 60 | ## Activating a virtual environment 61 | 62 | To activate a virtual environment, you can use the following command: 63 | 64 | ```bash 65 | source .venv/bin/activate 66 | ``` 67 | 68 | If you're using a terminal like `fish`, the command is slightly different. 69 | 70 | ```bash 71 | source .venv/bin/activate.fish 72 | ``` 73 | 74 | ## Installing Dependencies 75 | 76 | You can then install dependencies using the following command 77 | 78 | ```bash 79 | uv sync 80 | ``` 81 | 82 | This will install all of the dependencies specified in the `pyproject.toml` file. If you have a `requirements.txt` file, you can also use it to install dependencies. 83 | 84 | ```bash 85 | uv pip install -r requirements.txt 86 | ``` 87 | 88 | ## Deactivating a virtual environment 89 | 90 | To deactivate a virtual environment, you can use the following command: 91 | 92 | ```bash 93 | deactivate 94 | ``` 95 | 96 | This will deactivate the virtual environment and return you to your regular shell. This is useful when you'd like to switch between different virtual environments. 97 | 98 | ## Best Practices 99 | 100 | When working with virtual environments, we recommend the following best practices: 101 | 102 | 1. Always add your virtual environment to your gitignore file 103 | 2. Try to use a `pyproject.toml` file where possible to ensure that you have a consistent way of installing dependencies. This will result in `uv.lock` file that you should always commit to your repository. 104 | -------------------------------------------------------------------------------- /cohort_2/week1/README.md: -------------------------------------------------------------------------------- 1 | # Week 1: Building Your RAG Evaluation Foundation 2 | 3 | ## Notebooks Overview 4 | 5 | ### 1. synthetic_questions.ipynb 6 | 7 | This notebook focuses on creating diverse, realistic test questions to evaluate your RAG system. 8 | 9 | **What You'll Do:** 10 | 11 | - Load the Bird-Bench dataset of SQL queries as sample documents 12 | - Define Pydantic models (`Chunk`, `Question`, `ChunkEval`) for structured data 13 | - Generate diverse synthetic questions with varying complexity using GPT models 14 | - Implement asynchronous processing with rate limiting for efficient generation 15 | - Apply randomized constraints to ensure question diversity 16 | - Create a test dataset that challenges your retrieval system 17 | 18 | You'll leave with a robust set of synthetic questions that serve as your evaluation benchmark. 19 | 20 | ### 1. synthetic_questions_modal.ipynb 21 | 22 | This notebook introduces a practical approach to evaluating RAG systems through synthetic data generation. 23 | 24 | **What You'll Do**: 25 | 26 | - Learn why retrieval metrics are more efficient than content generation metrics 27 | - Set up an open-source model (Qwen-2.5-7B-Instruct) via Modal for cost-effective evaluation 28 | - Define Pydantic models for structured data output with instructor using vLLM 29 | - Load and analyze the Bird-Bench Text-2-SQL dataset for challenging examples 30 | - Create a comprehensive evaluation dataset using Pydantic-Evals 31 | 32 | You'll leave with both a set of synthetic questions for benchmarking and the knowledge to deploy and use open-source models for cost-effective RAG evaluation. 33 | 34 | ### 2. benchmark_retrieval.ipynb 35 | 36 | This notebook creates a systematic framework for comparing different retrieval strategies. 37 | 38 | **What You'll Do:** 39 | 40 | - Set up LanceDB with multiple embedding models for comparison 41 | - Implement key retrieval metrics (recall@k, MRR@k) 42 | - Create a parameterized retrieval function supporting different strategies 43 | - Benchmark multiple approaches: 44 | - Different embedding models 45 | - Vector vs. hybrid search 46 | - With and without reranking 47 | - Use Braintrust to track and compare experiment results 48 | - Run parallel evaluations with `concurrent.futures` 49 | 50 | You'll establish an objective way to measure which retrieval approaches work best. 51 | 52 | ### 3. visualise_results.ipynb 53 | 54 | This notebook introduces statistical validation to verify that improvements are significant. 55 | 56 | **What You'll Do:** 57 | 58 | - Implement bootstrapping to simulate multiple experimental runs 59 | - Calculate confidence intervals for different retrieval metrics 60 | - Visualize confidence intervals with error bars 61 | - Perform t-tests to determine statistical significance between approaches 62 | - Make data-driven decisions about which improvements to implement 63 | 64 | You'll gain the ability to distinguish real improvements from random variation. 65 | 66 | ## Technical Requirements 67 | 68 | - Required libraries: pandas, numpy, matplotlib, openai, instructor, lancedb, braintrust, scipy 69 | - Bird-Bench dataset from "567-labs/bird-rag" 70 | - OpenAI API access for question generation 71 | - Braintrust account for experiment tracking 72 | 73 | ## Why These Notebooks Matter 74 | 75 | Together, these notebooks provide a complete evaluation framework for RAG systems: 76 | 77 | 1. Generate challenging test questions 78 | 2. Measure retrieval performance objectively 79 | 3. Verify that improvements are statistically significant 80 | 81 | This approach ensures you invest in retrieval techniques that provide measurable value rather than randomly trying different approaches. The evaluation framework you build here will guide all future RAG improvements in this course. 82 | -------------------------------------------------------------------------------- /cohort_2/week1/modal_vllm.py: -------------------------------------------------------------------------------- 1 | import modal 2 | from pathlib import Path 3 | 4 | # Variables 5 | MODEL_DIR = Path("/models") 6 | MODEL_NAME = "Qwen/Qwen2.5-7B-Instruct" 7 | MODEL_REVISION = "a09a35458c702b33eeacc393d103063234e8bc28" 8 | 9 | # If you'd like to use the AWQ version of the model, uncomment the following two lines 10 | # MODEL_NAME = "Qwen/Qwen2.5-7B-Instruct-AWQ" 11 | # MODEL_REVISION = "b25037543e9394b818fdfca67ab2a00ecc7dd641" 12 | 13 | 14 | GPU_TYPE = "A100" 15 | N_GPU = 1 16 | MINUTES = 60 17 | VLLM_PORT = 8000 18 | API_KEY = "super-secret-key" 19 | 20 | 21 | # First we define an image 22 | vllm_image = ( 23 | modal.Image.debian_slim(python_version="3.12") 24 | .pip_install( 25 | "vllm==0.7.2", 26 | "huggingface_hub[hf_transfer]==0.26.2", 27 | "flashinfer-python==0.2.0.post2", # pinning, very unstable 28 | extra_index_url="https://flashinfer.ai/whl/cu124/torch2.5", 29 | ) 30 | .env({"HF_HUB_ENABLE_HF_TRANSFER": "1"}) # faster model transfers 31 | ) 32 | 33 | # Create the Modal dirs 34 | 35 | vllm_image = vllm_image.env({"VLLM_USE_V1": "1"}).env({"HF_HOME": str(MODEL_DIR)}) 36 | volume = modal.Volume.from_name("model-weights-vol", create_if_missing=True) 37 | vllm_volume = modal.Volume.from_name("vllm-cache-vol", create_if_missing=True) 38 | 39 | 40 | app = modal.App("vllm server") 41 | 42 | 43 | @app.function( 44 | volumes={MODEL_DIR: volume}, # "mount" the Volume, sharing it with your function 45 | image=vllm_image, # only download dependencies needed here 46 | ) 47 | def download_model(): 48 | from huggingface_hub import snapshot_download 49 | 50 | snapshot_download( 51 | repo_id=MODEL_NAME, local_dir=MODEL_DIR / MODEL_NAME, revision=MODEL_REVISION 52 | ) 53 | print(f"Model downloaded to {MODEL_DIR / MODEL_NAME}") 54 | 55 | 56 | @app.function( 57 | image=vllm_image, 58 | gpu=f"{GPU_TYPE}:{N_GPU}", 59 | scaledown_window=15 * MINUTES, # how long should we stay up with no requests? 60 | timeout=10 * MINUTES, # how long should we wait for container start? 61 | volumes={ 62 | # Link our previously used cache dir to the container 63 | "/root/.cache/huggingface": volume, 64 | # Set a cache dir for vLLM torch compile graphs 65 | "/root/.cache/vllm": vllm_volume, 66 | }, 67 | ) 68 | @modal.concurrent( 69 | max_inputs=100 70 | ) # how many requests can one replica handle? tune carefully! 71 | @modal.web_server(port=VLLM_PORT, startup_timeout=10 * MINUTES) 72 | def serve(): 73 | import subprocess 74 | 75 | cmd = [ 76 | "vllm", 77 | "serve", 78 | "--uvicorn-log-level=info", 79 | MODEL_NAME, 80 | "--revision", 81 | MODEL_REVISION, 82 | "--host", 83 | "0.0.0.0", 84 | "--port", 85 | str(VLLM_PORT), 86 | "--api-key", 87 | API_KEY, 88 | # Use your custom parser instead of the default hermes 89 | "--tool-call-parser", 90 | "hermes", 91 | "--enable-auto-tool-choice", 92 | "-O 0", # Remove this if you're looking to use the precompiled model weights that we did with precompile_model.remote() 93 | ] 94 | 95 | subprocess.Popen(" ".join(cmd), shell=True) 96 | 97 | 98 | # This helps us to precompile the model before we start the server 99 | @app.function( 100 | image=vllm_image, 101 | gpu=f"{GPU_TYPE}:{N_GPU}", 102 | volumes={ 103 | "/root/.cache/huggingface": volume, 104 | "/root/.cache/vllm": vllm_volume, 105 | }, 106 | timeout=10 * MINUTES, 107 | ) 108 | def precompile_model(): 109 | import subprocess 110 | import os 111 | 112 | # Run a quick inference to trigger compilation 113 | cmd = [ 114 | "python", 115 | "-c", 116 | "from vllm import LLM, SamplingParams; " 117 | f"llm = LLM(model='{MODEL_NAME}', revision='{MODEL_REVISION}'); " 118 | "sampling_params = SamplingParams(max_tokens=10); " 119 | "prompts = ['Hello world']; " 120 | "outputs = llm.generate(prompts, sampling_params)", 121 | ] 122 | 123 | subprocess.run(cmd, check=True) 124 | print("Model compilation completed and cached") 125 | 126 | # List the cache contents to confirm 127 | cache_path = "/root/.cache/vllm/torch_compile_cache" 128 | if os.path.exists(cache_path): 129 | print(f"Cache directory contents: {os.listdir(cache_path)}") 130 | print( 131 | f"Total cache size: {sum(os.path.getsize(os.path.join(cache_path, f)) for f in os.listdir(cache_path) if os.path.isfile(os.path.join(cache_path, f)))} bytes" 132 | ) 133 | else: 134 | print("Cache directory not found") 135 | 136 | vllm_volume.commit() 137 | 138 | 139 | ## Run modal run download.py to cache the download and compilation of the model first 140 | @app.local_entrypoint() 141 | def main(): 142 | download_model.remote() 143 | # Optionally you can precompile the model's weights. This yields a small speedup in production 144 | # precompile_model.remote() 145 | -------------------------------------------------------------------------------- /cohort_2/week2/README.md: -------------------------------------------------------------------------------- 1 | # Week 2: Fine-tuning Embedding Models for Retrieval 2 | 3 | ## Notebooks Overview 4 | 5 | ### 1. Synthetic-Transactions.ipynb 6 | This notebook focuses on generating and validating synthetic transaction data for fine-tuning embedding models. 7 | 8 | **What You'll Do:** 9 | - Create a Pydantic model (`Transaction`) to represent financial transaction data 10 | - Generate realistic transaction descriptions using GPT models with instructor for structure 11 | - Set up a manual review process using the included Streamlit app (`label.py`) 12 | - Establish LanceDB tables for vector search evaluation 13 | - Implement evaluation metrics (recall@k, MRR@k) 14 | - Create train/eval splits for model training 15 | - Visualize performance across different data splits 16 | 17 | You'll build a high-quality dataset specifically designed for embedding fine-tuning. 18 | 19 | ### 2. Finetune Cohere.ipynb 20 | This notebook demonstrates fine-tuning a Cohere re-ranker model for improved retrieval performance. 21 | 22 | **What You'll Do:** 23 | - Prepare training data with hard negatives for effective re-ranker training 24 | - Upload dataset to Cohere and initiate model fine-tuning 25 | - Evaluate the fine-tuned model against baselines using metrics from Week 1 26 | - Visualize performance improvements through comparison charts 27 | - Analyze where and why the fine-tuned model performs better 28 | 29 | You'll experience how managed services can simplify the fine-tuning process while delivering significant performance gains. 30 | 31 | ### 3. Open Source Models.ipynb 32 | This notebook explores fine-tuning open-source embedding models using sentence-transformers. 33 | 34 | **What You'll Do:** 35 | - Prepare datasets for triplet loss training with semi-hard negative mining 36 | - Configure and train a BAAI/bge-base-en model with SentenceTransformerTrainer 37 | - Use BatchSemiHardTripletLoss for effective training 38 | - Push the fine-tuned model to Hugging Face Hub for sharing 39 | - Conduct comprehensive evaluation comparing base vs. fine-tuned model performance 40 | - Analyze trade-offs between managed and open-source approaches 41 | 42 | You'll gain hands-on experience with open-source model fine-tuning techniques. 43 | 44 | ## Data Files and Assets 45 | - `categories.json`: Contains 24 transaction categories with sample transactions 46 | - `cleaned.jsonl`: Manually approved and labeled transaction examples 47 | - `hard_negatives.png` and `semi-hard-negative.png`: Illustrations of training techniques 48 | - `helpers.py`: Utility functions for calculating metrics and performing search 49 | - `label.py`: Streamlit application for manual transaction labeling 50 | 51 | ## Technical Requirements 52 | - Required libraries: sentence-transformers, lancedb, braintrust, pydantic, openai, cohere 53 | - Hugging Face token with write access for model hosting (open-source approach) 54 | - Cohere API key for managed service section 55 | - GPU recommended for open-source fine-tuning 56 | 57 | ## Why These Notebooks Matter 58 | These notebooks demonstrate two distinct approaches to improving embedding performance: 59 | 1. Managed services (Cohere): Simple API-based implementation with minimal engineering 60 | 2. Open-source (sentence-transformers): Complete control with self-hosting capabilities 61 | 62 | Both approaches show significant performance gains (15-30% in MRR and recall), highlighting that domain-specific fine-tuning is one of the most effective ways to improve RAG system performance. -------------------------------------------------------------------------------- /cohort_2/week2/data/hard_negatives.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/567-labs/systematically-improving-rag/32241e32e571bb7e34c257c143cc24f117f73f79/cohort_2/week2/data/hard_negatives.png -------------------------------------------------------------------------------- /cohort_2/week2/data/semi-hard-negative.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/567-labs/systematically-improving-rag/32241e32e571bb7e34c257c143cc24f117f73f79/cohort_2/week2/data/semi-hard-negative.png -------------------------------------------------------------------------------- /cohort_2/week2/helpers.py: -------------------------------------------------------------------------------- 1 | import itertools 2 | from lancedb.rerankers import Reranker 3 | from lancedb.table import Table 4 | import json 5 | 6 | 7 | def calculate_mrr(predictions: list[str], gt: list[str]): 8 | mrr = 0 9 | for label in gt: 10 | if label in predictions: 11 | mrr = max(mrr, 1 / (predictions.index(label) + 1)) 12 | return mrr 13 | 14 | 15 | def get_recall(predictions: list[str], gt: list[str]): 16 | return len([label for label in gt if label in predictions]) / len(gt) 17 | 18 | 19 | def get_metrics_at_k( 20 | metrics: list[str], 21 | sizes: list[int], 22 | ): 23 | metric_to_score_fn = { 24 | "mrr": calculate_mrr, 25 | "recall": get_recall, 26 | } 27 | 28 | for metric in metrics: 29 | if metric not in metric_to_score_fn: 30 | raise ValueError(f"Metric {metric} not supported") 31 | 32 | eval_metrics = [(metric, metric_to_score_fn[metric]) for metric in metrics] 33 | 34 | return { 35 | f"{metric_name}@{size}": lambda predictions, gt, m=metric_fn, s=size: ( 36 | lambda p, g: m(p[:s], g) 37 | )(predictions, gt) 38 | for (metric_name, metric_fn), size in itertools.product(eval_metrics, sizes) 39 | } 40 | 41 | 42 | def task(user_query: str, table: Table, reranker: Reranker, max_k: int): 43 | query = table.search(user_query, query_type="vector").limit(max_k) 44 | 45 | if reranker: 46 | query = query.rerank(reranker) 47 | 48 | return [item["text"] for item in query.to_list()] 49 | 50 | 51 | def get_dataset_split(file_path: str): 52 | with open(file_path, "r") as f: 53 | return [json.loads(line) for line in f] 54 | -------------------------------------------------------------------------------- /cohort_2/week4/README.md: -------------------------------------------------------------------------------- 1 | # Week 4: Topic Modeling and Query Classification 2 | 3 | ## Notebooks Overview 4 | 5 | ### 1. Generate Dataset.ipynb 6 | This notebook focuses on creating a realistic dataset of user queries based on Klarna's FAQ documents. 7 | 8 | **What You'll Do:** 9 | - Generate diverse synthetic questions using instructor and OpenAI 10 | - Create queries with varied personas (angry customer, confused user, etc.) 11 | - Implement intent variation (seeking information, reporting issues, etc.) 12 | - Calculate embedding similarity to ensure dataset diversity 13 | - Add citations linking questions to source documents 14 | - Store the output in structured JSONL format for further analysis 15 | 16 | You'll build a comprehensive dataset that simulates real user behavior with diverse intents and communication styles. 17 | 18 | ### 2. Topic Modelling.ipynb 19 | This notebook applies unsupervised learning to discover patterns in user queries. 20 | 21 | **What You'll Do:** 22 | - Configure and apply BERTopic for unsupervised topic discovery 23 | - Set up embedding model, UMAP, and HDBSCAN for effective clustering 24 | - Visualize topic similarities and relationships 25 | - Analyze satisfaction scores by topic to identify pain points 26 | - Create matrices showing relationship between topics and satisfaction 27 | - Identify the "danger zone" of high-volume, low-satisfaction query areas 28 | 29 | You'll gain insights into natural query patterns that can guide system improvements. 30 | 31 | ### 3. Classifier.ipynb 32 | This notebook develops a systematic approach to categorize and track queries over time. 33 | 34 | **What You'll Do:** 35 | - Define a classification taxonomy using YAML configuration 36 | - Implement a classifier using instructor and Pydantic for validation 37 | - Categorize queries by type, intent, and complexity 38 | - Test the classifier on sample queries 39 | - Examine different categorization strategies (multi-label vs. hierarchical) 40 | - Create a foundation for ongoing query monitoring 41 | 42 | You'll build a system to categorize incoming queries that provides actionable analytics. 43 | 44 | ## Data Files and Assets 45 | - `data/md/`: Klarna FAQ pages converted to markdown format 46 | - `data/questions.jsonl`: Base questions extracted from FAQ pages 47 | - `data/cleaned.jsonl`: Processed dataset with synthetic queries 48 | - `categories.yml`: Classification schema defining query types 49 | - `config.yml`: Configuration for query generation with personas/intents 50 | - `assets/matrix.png`: Visualization of topic-satisfaction relationship 51 | 52 | ## Technical Requirements 53 | - Required libraries: BERTopic, sentence-transformers, UMAP, HDBSCAN, instructor, pydantic, yaml 54 | - OpenAI API access for query generation and classification 55 | - Basic understanding of dimensionality reduction and clustering concepts 56 | 57 | ## Why These Notebooks Matter 58 | Together, these notebooks provide a systematic approach to understanding and improving RAG applications through query pattern analysis: 59 | 60 | 1. First, generate realistic test data that mimics actual user behavior 61 | 2. Then, discover natural patterns in queries through unsupervised learning 62 | 3. Finally, build a classifier that categorizes queries for ongoing monitoring 63 | 64 | This data-driven approach helps identify which query areas need improvement most urgently, allowing targeted enhancements rather than making random changes. -------------------------------------------------------------------------------- /cohort_2/week4/assets/matrix.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/567-labs/systematically-improving-rag/32241e32e571bb7e34c257c143cc24f117f73f79/cohort_2/week4/assets/matrix.png -------------------------------------------------------------------------------- /cohort_2/week4/categories.yml: -------------------------------------------------------------------------------- 1 | question_type: 2 | Payment Methods & Changes: 3 | - title: Payment Method Changes 4 | description: User needs to switch payment methods after making a purchase 5 | - title: Payment Options 6 | description: User is asking about available payment methods or how to set them up 7 | - title: Early Payoff 8 | description: User wants to pay their balance in full rather than continuing with installments 9 | 10 | Account & Settings: 11 | - title: Login Issues 12 | description: User is having trouble logging into their account 13 | - title: Account Management 14 | description: User is asking how to update or manage their account details 15 | 16 | Indices: 17 | - title: Shipping Information 18 | description: User is inquiring about shipping times, costs, or options 19 | - title: Payment Methods 20 | description: User has questions about accepted payment types or processing issues 21 | - title: Product Availability 22 | description: User is asking about stock levels or when an item will be restocked 23 | - title: Warranty Information 24 | description: User is asking about warranties on products 25 | - title: Technical Support 26 | description: User needs assistance with technical issues or troubleshooting 27 | 28 | Others: 29 | - title: Others 30 | description: If the query doesn't fit into any of the other categories 31 | -------------------------------------------------------------------------------- /cohort_2/week4/config.yml: -------------------------------------------------------------------------------- 1 | 2 | length: 3 | - short message that's got 1-2 spelling mistakes 4 | - medium message that's straight to the point 5 | - message with some useless context 6 | 7 | persona: 8 | - user that's not tech savvy 9 | - new user to klarna 10 | - frequent user of klarna 11 | - someone that's not tech savvy and wants to use klarna for the first time 12 | 13 | customer_intents: 14 | "Account & settings": 15 | "Login": 16 | - User is unable to log in to their account. 17 | - User forgot their password and needs to reset it. 18 | - User's account is locked, and they want access restored. 19 | - User is facing errors during the login process. 20 | "Manage account": 21 | - User wants to update their account details. 22 | - User cannot find the specific property they want to update (e.g., email, password). 23 | - User wants to close their account. 24 | - User is confused about where to manage their account settings. 25 | 26 | "Declined purchase": 27 | "Declined Purchase": 28 | - User wants to understand why their purchase was declined. 29 | - User is frustrated with a declined transaction and wants to resolve it. 30 | - User wants to retry a declined purchase but needs guidance. 31 | - User wants to know if a declined purchase affects their Klarna account status. 32 | 33 | "Delivery & returns": 34 | "Problem resolution": 35 | - User has not received their order and wants to know what to do next. 36 | - User received a damaged or incorrect item and wants assistance. 37 | - User is dissatisfied with the store's response and wants Klarna to intervene. 38 | "Returns": 39 | - User wants to initiate a return for an item purchased with Klarna. 40 | - User needs clarification on how return policies apply to Klarna purchases. 41 | - User wants to know what happens to their payment schedule after a return. 42 | "Cancellations": 43 | - User wants to cancel their order and needs to understand the process. 44 | - User is unsure if Klarna needs to be notified about the cancellation. 45 | - User's order was canceled by the store, and they want to know next steps. 46 | "Deliveries": 47 | - User is concerned about a delayed delivery. 48 | - User wants help tracking their order. 49 | - User did not receive their order and wants to report it. 50 | - User wants to know Klarna's policies regarding lost or delayed deliveries. 51 | 52 | "Fraud & security": 53 | "Data protection": 54 | - User is concerned about how their personal data is handled. 55 | - User wants to request deletion of their personal information. 56 | - User wants to know if Klarna shares their data with third parties. 57 | "Prevent fraud": 58 | - User is concerned about securing their Klarna account from potential fraud. 59 | "Report fraud": 60 | - User clicked on a link that was not legitimate and is panicking right now 61 | - User recieved a suspicious call claiming to be from Klarna and accidentally revealed their Klarna card details 62 | - User wants to report a fraudulent purchase or unauthorized transaction. 63 | 64 | "Payments": 65 | "Payment issues": 66 | - User is confused about why a payment isn't showing in their account. 67 | - User was charged incorrectly or twice for a single purchase. 68 | - User experienced a failed payment and wants to resolve the issue. 69 | "Make & manage payments": 70 | - User wants to manage their Klarna payment schedule. 71 | - User wants to pay off their balance early and needs guidance. 72 | - User wants to know what happens if they miss a payment deadline. 73 | - User wants to know if they can change their payment method 74 | 75 | "Products & services": 76 | "Klarna Card": 77 | - User wants to apply for a Klarna Card. 78 | - User lost or had their Klarna Card stolen and needs assistance. 79 | - User wants to understand the benefits and limitations of the Klarna Card. 80 | "Payment options": 81 | - User is unsure about the different payment options Klarna offers. 82 | - User wants to switch between payment plans (e.g., from installments to full payment). 83 | - User wants to know if interest or fees apply to their payment plan. 84 | "How to use Klarna": 85 | - User wants to understand how Klarna works for both online and in-store purchases. 86 | - User wants to find stores that accept Klarna. 87 | - User wants to know if Klarna can be used for recurring payments or subscriptions. 88 | "Klarna balance": 89 | - User wants to check their Klarna account balance. 90 | - User is confused about discrepancies in their balance. 91 | - User wants to know how refunds impact their Klarna balance. 92 | "One-time card": 93 | - User wants to create a Klarna one-time card. 94 | - User needs to cancel or adjust a one-time card. 95 | - User wants to know what happens to unused funds on a one-time card. 96 | 97 | "Refunds": 98 | "Manage refunds": 99 | - User is asking about refund status for their Nike purchase 100 | - User is asking if their Nike purchase is eligible for a refund 101 | - User wants refund instead of store credit 102 | - user wants to report nike for not responding to requests 103 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/can-i-change-my-return-after-its-been-reported.md: -------------------------------------------------------------------------------- 1 | # Can I change my return after it’s been reported? 2 | 3 | It’s not possible to change the information submitted after reporting the return to us. But don’t worry, you’ll only have to pay for the items that you keep. 4 | 5 | #### I want to return more/fewer items 6 | 7 | If you’ve changed your mind and you want to return more, or fewer, items than you originally reported, inform the store directly as only they can update your payment plan. Please check the store’s website for information on their return policy and instructions on how to make changes to your return. 8 | 9 | Once the store has registered your return, your payment plan will be adjusted accordingly, and we’ll notify you and send you an email with the updated payment plan. 10 | 11 | #### I want to cancel the return 12 | 13 | If you no longer want to return your order, you can cancel the reported return in the Klarna app by following the steps below. 14 | 15 | **To cancel your return:** 16 | 17 | 1. Go to [**Payments**](https://app.klarna.com/transactions/my-klarna/) 18 | 2. Select the purchase you want to update 19 | 3. Tap **Cancel return** 20 | 21 | If an outstanding amount is due after canceling the return, you can make a payment by [logging in.](https://app.klarna.com/to-do/to-pay/) 22 | 23 | *Good to know:* Once you’ve canceled your return, you will not be able to report a return for the same purchase again. 24 | 25 | #### I’ve accidentally canceled my return 26 | 27 | Once you’ve canceled your return, you will not be able to report a return to us again for the same purchase. If you have canceled your return in error, contact the store directly to adjust your payment plan. 28 | 29 | #### Related articles 30 | 31 | * [What’s the status of the investigation of my return?](https://www.klarna.com/us/customer-service/whats-the-status-of-the-investigation-of-my-return/) 32 | * [Why was the investigation of my return closed, even though my return hasn’t been registered yet?](https://www.klarna.com/us/customer-service/why-was-the-investigation-of-my-return-closed-even-though-my-return-hasnt-been-registered-yet/) 33 | * [How long does it take for my return to be registered?](https://www.klarna.com/us/customer-service/what-happens-when-i-report-a-return/) 34 | * [What happens when I report a return?](https://www.klarna.com/us/customer-service/what-happens-when-i-report-a-return/) 35 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/can-i-pay-in-store-using-klarna.md: -------------------------------------------------------------------------------- 1 | # Can I pay in-store using Klarna? 2 | 3 | To use Klarna in a physical store: 4 | 5 | * [Log in](https://app.klarna.com/login?auto-login&market=us&utm_campaign=customer-service-faq&utm_medium=referral&utm_source=klarna) to the Klarna app and tap **In\-store** from your home screen 6 | * Choose the store you want to shop at 7 | * Follow the prompts to create a digital card 8 | * Add the card to your digital wallet 9 | * When you’re ready to checkout, just access your Klarna in\-store digital card in your digital wallet; and tap your phone on the card reader to pay. 10 | 11 | *Good to know:* 12 | 13 | * If you attempt to shop with retailers that are not listed in the app, your purchase may be declined 14 | * When adding a payment card, ensure it has at least 25% of your order amount available. 15 | * Amex and prepaid cards are not accepted. 16 | * If you have an Android device you need to set up Google Pay. 17 | #### Related articles 18 | 19 | * [What is a One\-time card and how does it work?](https://www.klarna.com/us/customer-service/what-is-a-one-time-card-and-how-does-it-work/) 20 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/can-i-use-klarna-to-pay-for-subscriptions-and-on-demand-services.md: -------------------------------------------------------------------------------- 1 | # Can I use Klarna to pay for subscriptions and on-demand services? 2 | 3 | Yes, select Klarna in the checkout of partnering subscription or on\-demand services. You can choose either to pay in full by card or [Pay in 4\.](https://www.klarna.com/us/what-is-klarna/) 4 | 5 | 1. Tap **Klarna** in the service providers’ checkout 6 | 2. Select your preferred payment method 7 | 3. Confirm to activate **Klarna** as the payment method 8 | 9 | **Manage your subscriptions and on\-demand services in the app** 10 | 11 | [Log in](https://login.klarna.com/login/) for an overview of all your active services, payment settings and upcoming renewals. You can also navigate directly to the service provider to update your subscription plans: 12 | 13 | 1. Go to the **You** tab in the app 14 | 2. Tap **My Services** 15 | 3. Select the name of the provider and follow the prompts 16 | 17 | **How do I change or update my card?** 18 | 19 | 1. [Log in](https://login.klarna.com/login/) to the app 20 | 2. Go to **My Services** 21 | 3. Follow the prompts to change or add a new card 22 | 23 | *Good to know:* You can update your payment option on the partnered providers' website or app. 24 | 25 | 26 | 27 | 28 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/can-i-use-klarna-to-purchase-gift-cards.md: -------------------------------------------------------------------------------- 1 | # Can I use Klarna to purchase gift cards? 2 | 3 | Yes, you can purchase a gift card using our pay in full or pay later payment methods: 4 | 5 | 1. Use the search bar in the [app](https://app.klarna.com/login) to find your favorite stores offering gift cards 6 | 2. Select the gift card amount and payment method at checkout 7 | 8 | When shopping online, enter the gift card number and PIN in the gift card section in the store’s checkout. If you’re shopping in\-store, provide the gift card number and PIN to the cashier. 9 | 10 | #### Want to send your gift card to a friend? 11 | 12 | Once you’ve created a gift card, tap **Send to a friend** from the Gift card details page and follow the prompts to personalize, send or schedule it. You can gift the card either by email or text message. 13 | 14 | *Good to know:* 15 | 16 | * Gift cards are non\-refundable, which means your payment plan will continue as scheduled. 17 | * Our customer service agents are not able to cancel, refund, or adjust gift card amounts. 18 | * Stores typically issue store credit to refund returned gift card orders. 19 | * We recommend reading the store’s gift card Terms \& Conditions before placing an order. 20 | 21 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/does-klarna-charge-an-annual-fee.md: -------------------------------------------------------------------------------- 1 | # Does Klarna charge an annual fee? 2 | 3 | No, there is no annual fee for using [Klarna](https://login.klarna.com/login?market=GB&ui_locales=en-US#). 4 | 5 | Late payments and unpaid debts may however incur fees for our Pay later products and services. Read our [Terms \& Conditions](https://www.klarna.com/us/pay-with-klarna/) for detailed information about fees. 6 | 7 | #### Related Articles 8 | 9 | * [Products \& services](https://www.klarna.com/us/customer-service/csc/products-services/) 10 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/how-can-i-add-remove-or-change-my-card-or-bank-account.md: -------------------------------------------------------------------------------- 1 | # How can I add, remove or change my card or bank account? 2 | 3 | #### To change the card details 4 | 5 | 1. Go to [**Control center**](https://app.klarna.com/you/) 6 | 2. Select **Payment methods** 7 | 3. Tap **Add card** or select a stored card and tap **Make card primary** or **Remove your stored card** 8 | 9 | You‘ll see a temporary authorization hold when you add a new card, which will be released within 5\-7 business days. 10 | 11 | **Please note:**Payments for already placed purchases will be debited from the card used to place the purchase. To add a new card for existing purchases, you need to change the card on each purchase. 12 | 13 | #### To change the bank account details 14 | 15 | 1. Go to [**Settings**](https://app.klarna.com/you/) 16 | 2. Select **Payment methods** 17 | 3. Tap **Connect a bank account** to add an account or 18 | select the connected bank and tap **Change connected bank account** or **Remove bank account** 19 | 20 | *Please note:* Changing the bank account will cancel any currently scheduled payments. You can reschedule payments after you’ve added a new bank account. 21 | 22 | #### Can’t update your payment details? 23 | 24 | * Check you don’t have any processing payments 25 | * You can’t change the card within 3 days of a payment due date 26 | * Make sure the card does not expire within 2 months and is not a prepaid card 27 | * Ensure the bank is a local country bank 28 | #### Related articles 29 | 30 | * [How do I change the card for a purchase?](https://www.klarna.com/us/customer-service/how-do-i-change-the-card-for-a-purchase/) 31 | * [How can I pay?](https://www.klarna.com/us/customer-service/how-can-i-pay/) 32 | * [I have a payment due, but get an error message when I try to pay, what can I do?](https://www.klarna.com/us/customer-service/i-have-a-payment-due-but-get-an-error-message-when-i-try-to-pay-what-can-i-do/) 33 | 34 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/how-can-i-avoid-late-fees.md: -------------------------------------------------------------------------------- 1 | # How can I avoid late fees? 2 | 3 | To ensure you never miss a payment and avoid late fees: 4 | 5 | #### Enable payment notifications: 6 | 7 | 1. Go to [**Settings**](https://app.klarna.com/you/) 8 | 2. Navigate to **Notifications** 9 | 3. Find **Purchases and payments** 10 | 4. Tap the toggle to turn on **Payment due notifications** 11 | 12 | If you haven’t received your order, or have returned it to the store, report it to pause your invoice. 13 | 14 | #### Related articles 15 | 16 | * [How can I extend my due date?](https://www.klarna.com/us/customer-service/how-can-i-extend-my-due-date/) 17 | * [What happens if I can’t pay on time?](https://www.klarna.com/us/customer-service/what-happens-if-i-cant-pay-on-time/) 18 | * [I haven’t received my order. Do I have to pay?](https://www.klarna.com/us/customer-service/i-havent-received-my-order-do-i-have-to-pay/) 19 | 20 |   21 | 22 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/how-can-i-change-my-email-address.md: -------------------------------------------------------------------------------- 1 | # How can I change my email address? 2 | 3 | Chat with our Customer Service to change your email address for your active payment plan. 4 | We’ll email a verification code to your new email address to confirm the change. 5 | 6 | **What happens next?** 7 | When logging in with the new email address, you will now see the purchases connected to that email address in your Klarna account. Use the new email address to place future purchases and to keep and manage all your payments in one Klarna account. 8 | 9 | If you have an active membership or subscription update the store about your new email address. 10 | 11 | *Good to know:* 12 | 13 | * Use the updated email address when logging in for an overview of your purchases connected to that email address 14 | * The email address for paid purchases or processing purchases can’t be changed 15 | * The new email address will be used by Klarna when evaluating eligibility and to autofill when shopping with us 16 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/how-can-i-extend-my-due-date.md: -------------------------------------------------------------------------------- 1 | # How can I extend my due date? 2 | 3 | Log in to the app to extend your due date. If the option to extend your due date is available it will be offered: 4 | 5 | 1. Go to [**Payments**](https://app.klarna.com/to-do/to-pay/) 6 | 2. Select the order you would like to pay later 7 | 3. Tap **Options** and then tap **Extend due date** 8 | 9 | If you postpone your due date any upcoming payments for this order will be rescheduled to be withdrawn biweekly from the new payment date. You can see updated payment plans in the app by tapping **Show plan** on the order page. 10 | 11 | *Good to know:*You can extend your due date once per order. 12 | 13 | #### Financing 14 | 15 | Please note, it’s not possible to extend or change the due date of a Financing statement. 16 | 17 | #### Related articles 18 | 19 | * [What happens if I can’t pay on time?](https://www.klarna.com/us/customer-service/what-happens-if-i-cant-pay-on-time/) 20 | * [How do I return my order?](https://www.klarna.com/us/customer-service/how-do-i-return-my-order/) 21 | * [I haven’t received my order. Do I have to pay?](https://www.klarna.com/us/customer-service/i-havent-received-my-order-do-i-have-to-pay/) 22 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/how-can-i-get-approved-to-pay-with-klarna.md: -------------------------------------------------------------------------------- 1 | # How can I get approved to pay with Klarna? 2 | 3 | Follow the instructions in the pop\-up message which appeared at checkout before attempting to make another purchase, if you are not approved to pay with Klarna. 4 | 5 | Our automated approval decisions are based on the available customer data, primarily shared by credit bureaus, including information such as if you’ve paid off previous credits on time, or if you have too much outstanding debt elsewhere. 6 | 7 | #### I’ve used Klarna before but have been declined 8 | 9 | Each purchase attempt generates a new automated approval decision, which is based on [current credit data](https://www.klarna.com/us/customer-service/why-was-my-purchase-not-approved-with-klarna/), regardless of past approvals or rejections. A rejection does not negatively impact your credit score. 10 | 11 | *Good to know:* Our customer service agents do not have additional information about the automated decline reason, based on credit bureau decision data, and are not able to change or influence the decision. 12 | 13 | #### Can I make multiple purchases with Klarna? 14 | 15 | Yes. There is no set limit for how many purchases you can have with Klarna. 16 | 17 | #### Related articles 18 | 19 | * [Why was my purchase not approved with Klarna?](https://www.klarna.com/us/customer-service/why-was-my-purchase-not-approved-with-klarna/) 20 | * [Does Klarna perform a credit check on me and will it affect my credit score?](https://www.klarna.com/us/customer-service/does-klarna-affect-my-credit-score/) 21 | * [My purchase was denied, why do I still have a pending authorization hold on my card?](https://www.klarna.com/us/customer-service/my-purchase-was-denied-why-do-i-still-have-a-pending-authorization-hold-on-my-card/) 22 | * [Why can’t I pay with Klarna anymore?](https://www.klarna.com/us/customer-service/why-cant-i-pay-with-klarna-anymore/) 23 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/how-can-i-make-a-formal-complaint-to-klarna.md: -------------------------------------------------------------------------------- 1 | # How can I make a formal complaint to Klarna? 2 | 3 | If something hasn’t quite worked out the way it was supposed to, please contact our Customer Service. 4 | 5 | At Klarna, each of our customers is important to us, and we uphold your right to a fair, swift, and courteous service at all times. 6 | 7 | **To submit your complaint:** 8 | 9 | * Contact our [Customer Service](https://www.klarna.com/us/customer-service/#cs-contact-details/) 10 | * File your complaint with one of our trained Customer Service agents 11 | * Provide us with as much relevant information as possible (e.g., your statement reference), to ensure that we can help you as quickly and effectively as possible 12 | 13 | We deal with all complaints promptly, effectively, and in a positive manner and aim to resolve the matter during your initial contact with us. If this isn’t possible, we will keep you updated throughout our investigation, and we will endeavor to resolve your complaint as quickly as possible. 14 | 15 | Learn more about [Klarna’s complaints handling procedure](https://www.klarna.com/us/feedback-and-complaints/). 16 | 17 | *Please note:* If you are dissatisfied with the delivery or the return of your order, read our related [Delivery and Return articles](https://www.klarna.com/us/customer-service/csc/delivery-returns/) for more support. 18 | 19 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/how-can-i-merge-my-klarna-accounts.md: -------------------------------------------------------------------------------- 1 | # How can I merge my Klarna accounts? 2 | 3 | If you wish to manage all your payments in one Klarna account, chat with our Customer Service to see if it’s possible to connect your order to your preferred email address. 4 | 5 | *Good to know:* Always use the same email address when shopping with Klarna to avoid creating multiple Klarna accounts – it’ll make life easier next time you are looking for your purchase and payments details. 6 | 7 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/how-can-i-pay-for-my-klarna-card-purchases.md: -------------------------------------------------------------------------------- 1 | # How do I pay for my Klarna Card purchases? 2 | 3 | Your Klarna Card purchases are added to your monthly statement, emailed on the 26th of each month, and automatically withdrawn on the 16th of the following month.  4 | 5 | Payments can only be made via your connected bank account. We do not accept checks or debit or credit cards as payment for purchases made with the Klarna Card. 6 | 7 | [Log in to the app](https://app.klarna.com/login/) for an overview of all your Klarna Card purchases. 8 | 9 | #### Related articles 10 | 11 | * [Why was my Klarna Card purchase declined?](https://www.klarna.com/us/customer-service/why-was-my-klarna-card-purchase-declined/) 12 | * [How do I apply for a Klarna Card?](https://www.klarna.com/us/customer-service/how-do-i-apply-for-a-klarna-card/) 13 | * [When will I receive my Klarna Card refund?](https://www.klarna.com/us/customer-service/when-will-i-receive-my-klarna-card-refund/) 14 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/how-can-i-pay.md: -------------------------------------------------------------------------------- 1 | # How can I pay? 2 | 3 | Your payments are automatically withdrawn from your connected card or bank account according to the agreed payment schedule, but you can make early payments anytime you wish. 4 | 5 | #### To make a manual payment: 6 | 7 | 1. Go to [**Payments**](https://app.klarna.com/to-do/to-pay/) 8 | 2. Navigate to the order you would like to pay 9 | 3. Tap **Payment options** and follow the instructions 10 | 11 | *Good to know:* 12 | 13 | * Apple pay is also an available payment option in the app. 14 | * Prepaid cards are not accepted. We accept major debit and credit cards such as Visa, Discover, Maestro and Mastercard. 15 | * Credit cards are not accepted for purchases made with the Klarna Card. 16 | * You can’t cancel a payment while the payment is processing. 17 | * When paying your Financing statement be sure to pay the interest\-saver amount to maintain the promotional terms of your Financing purchases. 18 | #### Related articles 19 | 20 | * [Where can I find my payment plan?](https://www.klarna.com/us/customer-service/where-can-i-find-my-payment-plan/) 21 | * [When is my payment due?](https://www.klarna.com/us/customer-service/when-is-my-payment-due/) 22 | * [How can I add, remove or change my card or bank account?](https://www.klarna.com/us/customer-service/how-can-i-add-remove-or-change-my-card-or-bank-account/) 23 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/how-can-i-protect-myself-against-identity-theft.md: -------------------------------------------------------------------------------- 1 | # How can I protect myself against identity theft? 2 | 3 | #### Our tips to help you protect your personal details and stay safe: 4 | 5 | * Limit the personal details you share online 6 | * Don’t download files from an unknown source 7 | * Verify the legitimacy of unknown callers 8 | * Destroy unnecessary documents in your name (e.g. bills, bank statements) 9 | * Make sure that only you have access to your post 10 | * Report and immediately cancel lost/stolen cards or identification documents 11 | 12 | Read [Klarna’s fraud liability protection program](https://www.klarna.com/us/fraud-liability-protection/) to learn more about how we protect you from purchases or charges made without your authorization. 13 | 14 | #### Related articles 15 | 16 | * [How can I report a fraud attempt?](https://www.klarna.com/us/customer-service/how-can-i-report-a-fraud-attempt/) 17 | * [I don’t recognize a charge, what should I do?](https://www.klarna.com/us/customer-service/i-dont-recognize-a-charge-what-should-i-do/) 18 | * [What is phishing and how can I identify it?](https://www.klarna.com/us/customer-service/what-is-phishing-and-how-can-i-identify-it/) 19 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/how-can-i-report-a-fraud-attempt.md: -------------------------------------------------------------------------------- 1 | # How can I report a fraud attempt? 2 | 3 | If you are concerned that you may be a victim of fraud, you can immediately pause your payments by reporting the unauthorized purchase in the app: 4 | 5 | 1. Go to [**Payments**](https://app.klarna.com/transactions/my-klarna/) and select the unauthorized purchase 6 | 2. Select **Report a problem** and choose **I suspect identity theft** 7 | 3. Follow the prompts to submit your claim 8 | 4. Your payments will be paused while we investigate on your behalf 9 | 10 | **Note:** Pay attention to Klarna communications after your report since we might reach out if we need more information 11 | 12 | **Should I also report it to the authorities?** 13 | ​​Yes, if you believe an unauthorized purchase has been made in your name, you should report this first in the app and immediately after to the authorities. 14 | 15 | **Additional actions:** 16 | 17 | * Report the related card or transaction to your issuing bank 18 | * Provide us with your Police report case number as soon as you get it 19 | 20 | Read [Klarna’s fraud liability protection program](https://www.klarna.com/us/fraud-liability-protection/) to learn more about how we protect you from purchases or charges made without your authorization. 21 | 22 | #### Related articles 23 | 24 | * [I don’t recognize a charge, what should I do?](https://www.klarna.com/us/customer-service/i-dont-recognize-a-charge-what-should-i-do/) 25 | * [How can I protect myself against identity theft?](https://www.klarna.com/us/customer-service/how-can-i-protect-myself-against-identity-theft/) 26 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/how-can-i-request-the-deletion-of-my-personal-data.md: -------------------------------------------------------------------------------- 1 | # How can I request the deletion of my personal data? 2 | 3 | If you have decided to stop using Klarna, you can handle the deletion of personal information directly in the app. 4 | 5 | #### To delete your card information: 6 | 7 | 1. Go to [**Payment methods**](https://app.klarna.com/funding-sources/manage/) 8 | 2. Choose the card you want to delete 9 | 3. Tap **Remove your saved card** and choose **Remove connected card** 10 | #### To turn off your payment autofill information: 11 | 12 | 1. Go to [**Control center**](https://app.klarna.com/you/) and navigate to **Privacy \& Security** 13 | 2. Choose **Autofill at purchase** 14 | 3. Tap the toggle to turn off the autofill feature 15 | 16 | #### To stop receiving marketing emails from us: 17 | 18 | 1. Go to [**Control center**](https://app.klarna.com/you/) and navigate to **Notifications** 19 | 2. Find **Marketing emails** 20 | 3. Tap the toggle to turn off **Receive marketing emails** 21 | 22 | If you wish to limit the sharing of data with third parties, have all your personal information deleted or receive a data transcript, to to [Settings](https://app.klarna.com/login) in the app and navigate to **Privacy \& Security** or [contact us](https://app.klarna.com/support/) and submit a request. 23 | 24 | **Good to know:** As a financial institution, there is certain personal data we are required to store in order to stay compliant with banking laws and regulations. 25 | 26 | #### Related articles 27 | 28 | * [Is shopping with Klarna safe?](https://www.klarna.com/us/customer-service/is-shopping-with-klarna-safe/) 29 | * [I don’t recognize a charge, what should I do?](https://www.klarna.com/us/customer-service/i-dont-recognize-a-charge-what-should-i-do/) 30 | * [What is phishing and how can I identify it?](https://www.klarna.com/us/customer-service/what-is-phishing-and-how-can-i-identify-it/) 31 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/how-can-i-use-klarna-on-apple-pay.md: -------------------------------------------------------------------------------- 1 | # How can I use Klarna on Apple Pay? 2 | 3 | You can use Klarna’s payment options directly on Apple Pay with your Klarna Card, or in\-store card, when shopping online or in apps, on iPhone or iPad.  4 | 5 | 1. When checking out, choose Apple Pay and select your **Klarna Card** 6 | 2. If you do not have a [**Klarna Card**](https://www.klarna.com/us/klarna-card/), you can still use Klarna on Apple Pay by creating an in\-store card in the [**app**](https://www.klarna.com/us/klarna-app/) and adding it to your **Apple Wallet** 7 | 3. You can find your **Klarna Card** or **In\-store Card** by tapping **Other Cards \& Pay Later Options** on Apple Pay 8 | 4. Choose your preferred payment plan and confirm the purchase 9 | 10 | **Good to know:** 11 | 12 | * You’ll receive your order confirmation by email. 13 | * You can view and manage all your purchases in the Klarna app or on klarna.com. 14 | * In\-store purchases are not supported. 15 | 16 | **To add your Klarna Card or in\-store card to Apple Wallet** 17 | 18 | 1. In the Klarna [app,](https://login.klarna.com/na/lp/idp/oauth2/auth?client_id=639c2886-026e-452f-b5fc-096683d95b0e&redirect_uri=https%3A%2F%2Fapp.klarna.com%2Fauth%2Fcallback&nonce=a--D08YrPeT_vDFDm7ejog&response_type=code&scope=openid+offline_access+default&state=WcPBRKH5RNGO7Kd5Uib3zg&market=US&ui_locales=en-US&code_challenge_method=S256&code_challenge=HAUHwQBEMrgsnrOjXl93k2__oUAWTq5d_YxmOodcjA8&mode=login&funnel_id=c99d9292-7231-44d4-931c-c4aaea612e95&device_id=undefined&klarna_app_flavor=pink&klarna_app_version=24.34.23%2B7&audience=access) tap **Wallet** 19 | 2. Select your Klarna Card or in\-store card and tap **Add to Apple Wallet** 20 | 3. Fill\-in your card details and agree to the [T\&Cs](https://www.klarna.com/us/legal/) on the Apple Wallet intro screen 21 | 4. Select **Done** on the confirmation screen 22 | 23 | **Troubleshooting** 24 | 25 | * Be sure your device is updated to the latest iOS or iPadOS version. 26 | * Error message **Verification Not Available**: Wait 24 hours before trying again. 27 | * Error message **Card Not Added:** 28 | 1. Be sure your [Klarna Card is not temporarily frozen or blocked](https://www.klarna.com/uk/customer-service/why-was-my-klarna-card-purchase-declined/) 29 | 2. Check you entered the card details correctly 30 | 31 | **My  order using Klarna on Apple Pay didn’t go through** 32 | 33 | The reason is displayed in a pop\-up message, which appears at checkout, [if you are not approved to pay with Klarna.](https://www.klarna.com/us/customer-service/why-was-my-purchase-not-approved-with-klarna/) Be sure to follow the instructions in the pop\-up message before attempting to make another purchase. 34 | 35 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/how-do-i-add-money-into-my-klarna-balance.md: -------------------------------------------------------------------------------- 1 | # How do I add money into my Klarna balance? 2 | 3 | By logging in to the [app](https://app.klarna.com/login?auto-login&market=us&utm_campaign=header-login-link&utm_medium=referral&utm_source=klarna&pid=KlarnaCom&c=us.baseline.app-installs&af_adset=app-installs&af_ad=header-login-link). 4 | 5 | 1. Go to **Wallet** 6 | 2. Select **Klarna balance** 7 | 3. Tap **Add money** and follow the prompts to transfer funds from a US debit card in your name. 8 | 9 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/how-do-i-apply-for-a-klarna-card.md: -------------------------------------------------------------------------------- 1 | # How do I apply for a Klarna Card? 2 | 3 | Apply for your Klarna Card directly in the app or by logging in to our [website](https://login.klarna.com/login?market=GB&ui_locales=en-US#): 4 | 5 | 1. [Log in](https://login.klarna.com/login?market=GB&ui_locales=en-US#) to the app 6 | 2. Go to **Wallet** and tap **Klarna Card** 7 | 3. Choose **Join** [**waitlist**](https://www.klarna.com/us/klarna-card-waitlist/) 8 | 9 | To be eligible for the Klarna Card you must: 10 | 11 | * Be a US resident 12 | * Be at least 18 years old 13 | * Have previously used Klarna and paid on time 14 | * Have a good credit history 15 | 16 | *Please note:* 17 | 18 | * [A soft credit check will be performed](https://www.klarna.com/us/customer-service/klarna-perform-credit-check/) when applying for a Klarna Card. 19 | * The Klarna Card is not currently available in Puerto Rico. 20 | 21 | #### Related articles 22 | 23 | * [Why was I denied a Klarna Card?](https://www.klarna.com/us/customer-service/why-was-i-denied-a-klarna-card/) 24 | * [Does Klarna perform a credit check?](https://www.klarna.com/us/customer-service/klarna-perform-credit-check/) 25 | * [How do I pay for my Klarna Card purchases?](https://www.klarna.com/us/customer-service/how-do-i-pay-for-my-klarna-card-purchases/) 26 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/how-do-i-block-or-cancel-my-klarna-card.md: -------------------------------------------------------------------------------- 1 | # How do I block or cancel my Klarna Card? 2 | 3 | Temporarily block your Klarna Card by freezing it in the app. You can unfreeze your card in the app when you are ready to use your card again. 4 | 5 | #### Freeze card 6 | 7 | 1. Log in to the app 8 | 2. Tap [Klarna Card](https://app.klarna.com/login) 9 | 3. Choose **Freeze my card** 10 | #### Cancel card 11 | 12 | You can permanently cancel your Klarna Card in the app: 13 | 14 | 1. Log in to the app 15 | 2. Tap [Klarna Card](https://app.klarna.com/cards/) 16 | 3. Go to the **Settings** of your Klarna Card 17 | 4. Choose **Cancel my card** and select the reason why you want to cancel your Klarna Card to confirm 18 | #### Replace card 19 | 20 | To deactivate and replace your Klarna Card: 21 | 22 | 1. Log in to the app 23 | 2. Tap [Klarna Card](https://app.klarna.com/cards/) 24 | 3. Choose **Replace my card** 25 | 26 | *Good to know:* 27 | 28 | * If you’ve created a virtual Klarna Card, you must first cancel it in the app before canceling the physical card. 29 | * Purchases you placed with a canceled Klarna Card remain due for payment. 30 | * Cancel your Klarna Card before the 20th of the month to avoid being charged a fee for the following month. 31 | #### Related articles 32 | 33 | * [How do I apply for a Klarna Card?](https://www.klarna.com/us/customer-service/how-do-i-apply-for-a-klarna-card/) 34 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/how-do-i-cancel-my-order.md: -------------------------------------------------------------------------------- 1 | # How do I cancel my order? 2 | 3 | To cancel your order, follow these steps: 4 | 5 | 1. **Contact the store** 6 | Contact the store directly to cancel your order. Always follow the store’s cancellation policy. As the payment provider, we can’t cancel your order. 7 | 2. **Wait for the store’s confirmation** 8 | We will cancel your payment accordingly and refund any paid amount when the store confirms the cancellation. 9 | 3. **Check for updates in the Klarna app** 10 | You can check the status of your order anytime under [**Payments**](https://app.klarna.com/transactions/my-klarna/) in the Klarna app. 11 | 4. **Receive refund** 12 | [Refund processing times](https://www.klarna.com/us/customer-service/how-long-do-refunds-take/) vary and can take up to 14 days, depending on your financial institution. If 14 days have passed since the store confirmed your cancellation and you’re still missing your refund, please [chat](https://app.klarna.com/support/) with our Customer Service with your cancellation confirmation available. 13 | 14 | #### Related articles 15 | 16 | * [How long do refunds take?](https://www.klarna.com/us/customer-service/how-long-do-refunds-take/) 17 | * [My order was cancelled, why am I still being charged?](https://www.klarna.com/us/customer-service/my-order-was-cancelled-why-am-i-still-being-charged/) 18 | * [What should I do if the store doesn’t respond?](https://www.klarna.com/us/customer-service/what-should-i-do-if-the-store-doesnt-respond/) 19 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/how-do-i-change-my-billing-address.md: -------------------------------------------------------------------------------- 1 | # How do I change my billing address? 2 | 3 | Log in to change the billing address in the app. 4 | 5 | 1. Go to [**Control center**](https://app.klarna.com/you/) 6 | 2. Select **Account into and address** 7 | 3. Edit your billing address and **Confirm** 8 | 9 | **What happens next?** 10 | Please allow 48 hours for the change to be updated in our system. 11 | 12 | *Good to know:* Updating the address in your Klarna account won’t change the shipping address for your orders. Inform the store if the shipping address also needs to be updated 13 | 14 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/how-do-i-change-my-name.md: -------------------------------------------------------------------------------- 1 | # How do I change my name? 2 | 3 | Chat with our Customer Service with a government\-issued ID to hand, to change your name in your Klarna account. 4 | 5 | **What happens next?** 6 | It will take 48 hours for your name change to update on your Klarna account. Keep in mind, changes to your name will not be reflected on purchases you’ve previously placed. 7 | 8 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/how-do-i-change-the-card-for-a-purchase.md: -------------------------------------------------------------------------------- 1 | # How do I change the card for a purchase? 2 | 3 | Log in to the app to change the payment card used for a specific purchase. 4 | 5 | 1. Tap [**Payments**](https://app.klarna.com/to-do/to-pay/) 6 | 2. Select the order you would like to pay for with another card 7 | 3. Tap **Manage** and then **Change card** 8 | 4. Select your preferred card or tap **Pay with other card** and follow the prompts 9 | 10 | Please repeat the steps if you wish to change the payment card for another purchase. 11 | To change the card for your future purchases, you need to change the stored card in your Klarna account. 12 | 13 | *Good to know:* You‘ll see a temporary authorization hold when you add a new card, which will be released within 5\-7 business days. 14 | 15 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/how-do-i-close-my-klarna-account.md: -------------------------------------------------------------------------------- 1 | # How do I close my Klarna account? 2 | 3 | Settle any outstanding payments and then chat with Customer Service to close your Klarna account. 4 | 5 | **What happens next?** 6 | 7 | * You will not be able to log in to your Klarna account 8 | * You will not be able to view previous payments or purchases 9 | * If you requested your personal information be removed it is then permanently deleted, except for certain information that we’re legally required to retain, as outlined in our [Privacy Policy](/us/privacy/) 10 | #### Related articles 11 | 12 | * [How can I request the deletion of my personal data?](https://www.klarna.com/us/customer-service/how-can-i-request-the-deletion-of-my-personal-data/) 13 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/how-do-i-log-in.md: -------------------------------------------------------------------------------- 1 | # How do I log in? 2 | 3 | #### To log in to your Klarna account 4 | 5 | 1. Open the **Klarna app** or go to [**Log in**](https://app.klarna.com/login?auto-login&market=us&utm_campaign=customer-service-faq&utm_medium=referral&utm_source=klarna) 6 | 2. Enter the email address or phone number associated with your Klarna purchases 7 | 3. Tap **Confirm** and **Continue** 8 | 4. You will receive an email with a login link or a text with a 6\-digit verification code 9 | 5. Open the login link on the same device or insert the code into the app to log in 10 | 11 | *Good to know:* The first time you log in to your Klarna account you will be required to verify both your email and phone number. Then you can choose your preferred login method. 12 | 13 | #### To enable Face ID/Touch ID or a PIN code verification 14 | 15 | 1. Go to [**Control center**](https://app.klarna.com/you/) 16 | 2. Tap select **Face ID/Touch ID \& PIN** 17 | 3. Toggle on **Face ID/Touch ID** or **PIN** and complete the setup steps 18 | 19 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/how-do-i-return-an-order-purchased-with-a-gift-card.md: -------------------------------------------------------------------------------- 1 | # How do I return an order purchased with a gift card? 2 | 3 | Contact the store to let them know you want to return items purchased with your gift card or follow the store’s Return Policy. 4 | 5 | Stores typically issue store credit when refunding returned gift card orders. 6 | 7 | *Good to know:* 8 | 9 | * Gift cards are non\-refundable, which means your payment plan will continue as scheduled. 10 | * Our customer service agents are not able to cancel, refund, or adjust gift card amounts. 11 | #### Related articles 12 | 13 | * [I have a problem with the order purchased with my gift card, what should I do?](https://www.klarna.com/us/customer-service/i-have-a-problem-with-the-order-purchased-with-my-gift-card-what-should-i-do/) 14 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/how-do-i-return-my-order.md: -------------------------------------------------------------------------------- 1 | # How do I return my order? 2 | 3 | Follow the store’s return instructions and then let us know which items you have returned by reporting a return in the app: 4 | 5 | 1. Go to [**Payments**](https://app.klarna.com/transactions/my-klarna/) 6 | 2. Select the purchase, or the items, you wish to return 7 | 3. Tap **Report return** 8 | 9 | Heads up, if you later wish to return more items than you initially reported in the app, be sure to let the store know so your order details can be updated by the store. 10 | 11 | #### Save the tracking information 12 | 13 | Submit the information at any time in the [app](https://app.klarna.com/login) – follow the prompts to enter the tracking details \& the date you made the return. If after 21 days your return is not registered correctly by the store we can always support you if you provided the required returns information. 14 | 15 | And always be sure to keep copies of your correspondence or proof of your attempts to contact the store. 16 | 17 | #### Goods ordered online returned to the physical store 18 | 19 | If you have returned goods ordered online to the physical store keep proof of the return until your return has been registered. 20 | 21 | #### Check the the app for updates 22 | 23 | Once the store has adjusted your payment plan we will notify you, and send an email, with the new balance. [How your refund will be processed](https://www.klarna.com/us/customer-service/how-long-do-refunds-take/) depends on the payment method you used. If you have any questions regarding the status of your return or refund eligibility contact the store directly. 24 | 25 | *Good to know:* if you receive store credit, for example a gift card, your payments to Klarna are still due. Please note that Klarna manages your payment while the store is responsible for all other aspects concerning your order. 26 | 27 | #### Related articles 28 | 29 | * [What happens when I report a return?](https://www.klarna.com/us/customer-service/what-happens-when-i-report-a-return/) 30 | * [Can I change my return after it’s been reported?](https://www.klarna.com/us/customer-service/can-i-change-my-return-after-its-been-reported/) 31 | * [My return hasn’t been registered, what should I do?](https://www.klarna.com/us/customer-service/my-return-hasnt-been-registered-what-should-i-do/) 32 | * [How long do refunds take?](https://www.klarna.com/us/customer-service/how-long-do-refunds-take/) 33 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/how-do-i-use-my-klarna-balance.md: -------------------------------------------------------------------------------- 1 | # How do I use my Klarna balance? 2 | 3 | Log in to the [app](https://login.klarna.com/na/lp/idp/oauth2/auth?client_id=639c2886-026e-452f-b5fc-096683d95b0e&redirect_uri=https%3A%2F%2Fapp.klarna.com%2Fauth%2Fcallback&nonce=YBevdEe0IF2-sInml8dHQA&response_type=code&scope=openid+offline_access+default&state=SpJBkAcgIRPICZ8Yh2c2Zg&market=US&ui_locales=en-US&code_challenge_method=S256&code_challenge=2wCCL7az9jdBbHIEjW0U00dK2TKLQ8OFwQmtID0VFHQ&mode=login&funnel_id=21def097-09ac-4515-b783-946b3ef21d5b&device_id=undefined&klarna_app_flavor=pink&klarna_app_version=24.47.209%2B10&audience=access) to shop and pay for your orders or services placed using Klarna. 4 | 5 | 1\. Go to **Payments** 6 | 2\. Choose your Pay in 4 or Financing purchase 7 | 3\. Tap **Pay early** and select **Klarna balance** as your funding source 8 | 9 | You can also use your Klarna balance when the Pay Now option is offered in the store's checkout. Keeping in mind the amount on your balance needs to cover the full purchase price. 10 | 11 | **Good to know:** The funds loaded on your Klarna balance gift card are nonrefundable. 12 | 13 | 14 | 15 | **Related articles** 16 | 17 | * [How do I add money into my Klarna balance?](https://www.klarna.com/us/customer-service/how-do-i-add-money-into-my-klarna-balance/) 18 | 19 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/how-long-do-refunds-take.md: -------------------------------------------------------------------------------- 1 | # How long do refunds take? 2 | 3 | It can take up to 14 days for refunds to process and the balance of your connected payment method to update.  4 | 5 | * The refund process starts as soon as we receive [return](https://www.klarna.com/us/customer-service/csc/delivery-returns/returns/) confirmation from the store. 6 | * [Log in](https://app.klarna.com/login) to check the store has received your return or [canceled](https://www.klarna.com/us/customer-service/csc/delivery-returns/cancellations/) your order. 7 | 8 | You’ll receive an email and an [app](https://app.klarna.com/login/) notification with updated refund information. Any remaining payments will be canceled.  9 | 10 | **Did you return or cancel part of your order?** 11 | 12 | In this case, we will issue a partial refund: 13 | 14 | * If the partial refund is **less** than the remaining balance of your order, the amount still owed will be spread evenly across the remaining payments. 15 | * If the partial refund is **greater** than the remaining balance of your order, the difference will be credited to the original payment method. 16 | 17 | Note, for partial refunds on orders placed with the One\-time card Pay in 4 service, the service fee is still charged. 18 | 19 | **Still haven’t received your refund after 14 days?** 20 | 21 | Did you recently change banks or close your linked bank account? Have you canceled, temporarily frozen, blocked or changed your connected card? 22 | 23 | If so, [log in](https://app.klarna.com/login) to let us know \- your refund may have bounced back to Klarna, causing the delay. 24 | 25 | **Financing** 26 | 27 | * If the total amount owed is greater than the refund, the refund will be deducted from the total balance of your financing account. 28 | * If the total amount owed is less than the refund, the refund will be available as a positive balance in your Klarna account, allowing you to purchase again using your account at any participating online stores. 29 | 30 | **Good to know:** Should the store’s return policy only allow refunds in the form of store credit, you are still responsible for your payments to Klarna associated with that purchase. According to the store’s T\&Cs express shipping fees are not refundable. 31 | 32 | #### Related articles 33 | 34 | * [How do I cancel my order?](https://www.klarna.com/us/customer-service/how-do-i-cancel-my-order/) 35 | * [What happens when I report a return?](https://www.klarna.com/us/customer-service/what-happens-when-i-report-a-return/) 36 | * [My order was cancelled, why am I still being charged?](https://www.klarna.com/us/customer-service/my-order-was-cancelled-why-am-i-still-being-charged/) 37 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/how-long-does-it-take-for-my-problem-to-be-solved.md: -------------------------------------------------------------------------------- 1 | # How long does it take for my problem to be solved? 2 | 3 | The quickest way for you to resolve your problem is to find a solution together with the store. Please contact the store directly so they can update your payment plan accordingly. If you are unable to resolve your problem with the store after 21 days, we will ask you to provide us with more information so we can investigate further. 4 | 5 | Once Klarna starts the review, resolution time usually ranges between 20 – 50 days, hence we recommend resolving the issue together with the store even after you’ve reported your problem to us. 6 | 7 | #### Related articles 8 | 9 | * [What happens after I‘ve reported my problem?](https://www.klarna.com/us/customer-service/what-happens-after-ive-reported-my-problem/) 10 | * [What’s the status of the investigation into my problem?](https://www.klarna.com/us/customer-service/whats-the-status-of-the-investigation-into-my-problem/) 11 | * [My problem has been solved, what should I do?](https://www.klarna.com/us/customer-service/my-problem-has-been-solved-what-should-i-do/) 12 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/how-long-does-it-take-for-my-return-to-be-registered.md: -------------------------------------------------------------------------------- 1 | # How long does it take for my return to be registered? 2 | 3 | The store is responsible for registering your return. That’s why we recommend you reach out to them directly for updates on your return. If you have any questions about your return, check the store’s website for information on their return policy. 4 | 5 | If your return hasn’t been registered and your due date is approaching, let us know you’ve returned items to pause your payment plan: 6 | 7 | 1. Go to [**Payments**](https://app.klarna.com/transactions/my-klarna/) 8 | 2. Select the purchase you want to return 9 | 3. Tap **Report return** and follow the instruction to pause your payments 10 | 11 | Once your return has been registered by the store, your payment plan will be adjusted accordingly and we’ll notify you and send you an email with an updated payment plan. 12 | 13 | #### Waiting longer than expected for your return to be registered? 14 | 15 | Submit the returns information at any time in the [app](https://app.klarna.com/login) – follow the prompts to enter the tracking details \& the date you made the return. If after 21 days your return is not registered correctly by the store we can always support you if you provided the required returns information. 16 | 17 | It can take between 30 to 50 days for Klarna to reach a decision on the investigation, therefore we advise you to solve the issue with the store even after you’ve reported your return. And always be sure to keep copies of your correspondence or proof of your attempts to contact the store. 18 | 19 | #### Related articles 20 | 21 | * [How do I return my order?](https://www.klarna.com/us/customer-service/how-do-i-return-my-order/) 22 | * [Can I change my return after it’s been reported?](https://www.klarna.com/us/customer-service/can-i-change-my-return-after-its-been-reported/) 23 | * [My return hasn’t been registered, what should I do?](https://www.klarna.com/us/customer-service/my-return-hasnt-been-registered-what-should-i-do/) 24 | * [How long do refunds take?](https://www.klarna.com/us/customer-service/how-long-do-refunds-take/) 25 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/how-much-am-i-eligible-to-spend.md: -------------------------------------------------------------------------------- 1 | # How much am I eligible to spend? 2 | 3 | There is no predefined spending limit when using Klarna. Instead, a new automated approval decision about how much you can spend with us is made each time you pay with Klarna. 4 | 5 | #### To see your estimated amount to spend: 6 | 7 | 1. [Log in](https://app.klarna.com/login?auto-login&market=us&utm_campaign=customer-service-faq&utm_medium=referral&utm_source=klarna) to the app 8 | 2. Navigate to **Purchase power** 9 | 3. Follow the prompts to check the estimated amount you can spend on your next purchase 10 | 11 | Your Purchase power is an estimated amount based on factors such as your payment history with Klarna and your outstanding balance. The amount is not a guarantee and may change based on the store and payment option you choose. 12 | 13 | **Good to know:** If you are using the [Shopping extension on your browser](https://www.klarna.com/us/browser-extension/) you can see your estimated purchase power when creating a One\-time card. 14 | 15 | #### I want to increase my Purchase power 16 | 17 | A good payment history, always paying on time and making payments towards your outstanding purchases can increase your Purchase power over time. Our Customer Service can’t accept requests for spending limit increases. 18 | 19 | #### My Purchase power is $0 20 | 21 | If you’ve spent above your purchase power, then your purchase power will be $0\. Making payments towards your outstanding purchases can increase your Purchase Power. 22 | 23 | #### My Purchase power is not available 24 | 25 | If you have overdue or failed payments, then your Purchase power will not be available. To increase your chance of approval to shop with Klarna, and view your Purchase power in the app, settle your failed or overdue payments 26 | 27 | #### Related articles 28 | 29 | * [Why was my purchase not approved with Klarna?](https://www.klarna.com/us/customer-service/why-was-my-purchase-not-approved-with-klarna/) 30 | * [How can I get approved to pay with Klarna?](https://www.klarna.com/us/customer-service/why-was-my-purchase-not-approved-with-klarna/) 31 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/i-cant-log-in-what-can-i-do.md: -------------------------------------------------------------------------------- 1 | # I can’t log in, what can I do? 2 | 3 | If you’re unable to sign in to your account for any other reason, please call our Customer Service. 4 | 5 | #### I didn’t receive the login email 6 | 7 | Try the following if you didn’t receive the email with the verification code/login link: 8 | 9 | * Check your spam folder 10 | * Verify you are signing in with the same email address you used to place the order 11 | * Try entering another email address you might have used 12 | * Use your phone number to log in with the verification code instead 13 | #### The login link isn’t working 14 | 15 | If you received the email but the login link is not working, try the following: 16 | 17 | * Update the Klarna app to the latest version 18 | * Click the login link on the same device from which you intend to log in to the app 19 | * Use your phone number to log in with the verification code instead 20 | #### I didn’t receive the verification code 21 | 22 | If you didn’t receive the text with the verification code, try the following: 23 | 24 | * Check you entered the correct phone number 25 | * Try adding the country code to your phone number 26 | * Switch flight mode on and off then resend code 27 | #### I forgot my PIN code 28 | 29 | 1. Log in with your email instead 30 | 2. Go to [**Control center**](https://app.klarna.com/you/) 31 | 3. Scroll down and select **Face ID/Touch ID \& PIN** 32 | 4. Toggle **PIN** off and then back on 33 | 5. Complete the setup steps to sign in with your new PIN code 34 | 35 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/i-cant-open-a-klarna-balance-what-can-i-do.md: -------------------------------------------------------------------------------- 1 | # I can't open a Klarna balance, what can I do? 2 | 3 | Follow these troubleshooting steps before trying again to open a Klarna balance: 4 | 5 | **Identification verification failed** 6 | 7 | Ensure your full name, date of birth, SSN and address are entered correctly. 8 | 9 | **I received an error message** 10 | 11 | We would not be able to verify your data: 12 | 13 | * If your personal information is not yet updated after moving address 14 | * If you are below 18 15 | * If you already have an existing account 16 | * If you entered your full name, date of birth, SSN or address incorrectly 17 | 18 | 19 | **How to try again to open a Klarna balance** 20 | 21 | 1. Log in to the [app](https://login.klarna.com/na/lp/idp/oauth2/auth?client_id=639c2886-026e-452f-b5fc-096683d95b0e&redirect_uri=https%3A%2F%2Fapp.klarna.com%2Fauth%2Fcallback&nonce=sNZSAImpvWm-ILSUVCOxGA&response_type=code&scope=openid+offline_access+default&state=HE89_01IWsjlW1YMM5oWMg&market=US&ui_locales=en-US&code_challenge_method=S256&code_challenge=OAKrMno6WTR76PmcDkbaJcjigfOovJADGLBn6F_6PAE&mode=login&funnel_id=5f7fb9b8-03ef-4885-b542-83b8bfb8ca61&device_id=undefined&klarna_app_flavor=pink&klarna_app_version=24.37.26%2B11&audience=access) 22 | 2. Go to **Wallet** 23 | 3. Select **Klarna balance** 24 | 4. Follow the prompts to **Set up your Klarna balance** 25 | 26 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/i-dont-recognize-a-charge-what-should-i-do.md: -------------------------------------------------------------------------------- 1 | # I don’t recognize a charge, what should I do? 2 | 3 | If you receive an email or see a charge linked to Klarna that you don’t recognize, report the unauthorized charge to us. 4 | 5 | #### Before reporting a charge as unauthorized, first, check that: 6 | 7 | * You are the only person in your household who has used your account to shop 8 | * You did not select Klarna at the checkout on a store’s website without noticing 9 | * You did not receive an order confirmation or email from Klarna on or around the date of the purchase 10 | * It‘s not a charge for a previous order that has been billed separately 11 | * It’s not a possible authorization hold for a pending purchase 12 | 13 | If you still do not recognize this Klarna purchase and you believe you may be a victim of identity theft, report it to us directly in the app or by [contacting](https://app.klarna.com/support/) our customer service. 14 | 15 | #### Related articles 16 | 17 | * [How can I report a fraud attempt?](https://www.klarna.com/us/customer-service/how-can-i-report-a-fraud-attempt/) 18 | * [Why do I need to verify my details when paying?](https://www.klarna.com/us/customer-service/why-do-i-need-to-verify-my-details-when-paying/) 19 | * [How can I protect myself against identity theft?](https://www.klarna.com/us/customer-service/how-can-i-protect-myself-against-identity-theft/) 20 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/i-have-a-payment-due-but-get-an-error-message-when-i-try-to-pay-what-can-i-do.md: -------------------------------------------------------------------------------- 1 | # I have a payment due, but get an error message when I try to pay, what can I do? 2 | 3 | Follow these troubleshooting tips if your due payment failed when you tried to pay: 4 | 5 | #### Invalid card 6 | 7 | First, check if your connected card has expired or if it has been blocked or frozen. Log in to the app to update or change your connected card details: 8 | 9 | 1. Tap [**Purchases**](https://app.klarna.com/transactions/my-klarna/) 10 | 2. Select your order 11 | 3. Choose **Change card** 12 | #### Technical error 13 | 14 | If you encountered an error message when trying to make a payment in the app take a moment to: 15 | 16 | * Check you’re using the latest version of the app 17 | * Clear cookies in your browser 18 | * Be sure you’re not using a limited or restricted network (such as a work connection) 19 | * Check you have sufficient funds available to cover the payment 20 | #### Failed two\-factor authentication 21 | 22 | If your card issuer has requested you authenticate your identity for a Klarna payment, be sure to follow the prompts which redirect you to approve the payment. 23 | 24 | If you don’t follow the two\-factor authentication prompts you will receive an error message and be unable to complete the purchase – so try placing the order again but follow the prompts. 25 | 26 | #### Insufficient funds 27 | 28 | If we are unable to collect the payment on the scheduled date, we will try again. So please ensure your connected card is up to date and there are enough funds available for the scheduled payment. 29 | 30 | *Good to know:* If your payment fails, you’ll be sent an email with information on how to resume payment. 31 | 32 | #### Related articles 33 | 34 | * [When is my payment due?](https://www.klarna.com/us/customer-service/when-is-my-payment-due/) 35 | * [What happens if I can’t pay on time?](https://www.klarna.com/us/customer-service/what-happens-if-i-cant-pay-on-time/) 36 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/i-have-a-problem-with-the-order-purchased-with-my-gift-card-what-should-i-do.md: -------------------------------------------------------------------------------- 1 | # I have a problem with the order purchased with my gift card, what should I do? 2 | 3 | If you haven’t received the items purchased with your gift card, or there is something wrong with the order, contact the store. 4 | 5 | *Good to know:* 6 | 7 | * It is only the store that can answer questions about items purchased with a gift card. 8 | * Gift cards are non\-refundable, which means your payment plan will continue as scheduled. 9 | * Our customer service agents are not able to cancel, refund, or adjust gift card amounts. 10 | * Read the store’s gift card Terms \& Conditions 11 | #### My gift card does not work 12 | 13 | Let us know what the problem is by submitting the information in the app: 14 | 15 | 1. [Log in](https://app.klarna.com/login) to the app 16 | 2. Tap **Report a problem** to pause your upcoming payment 17 | 3. Keep an eye on the app for updates on the status of the investigation 18 | #### Can’t find your gift card in the app? 19 | 20 | 1. [Log in](https://app.klarna.com/login) the app 21 | 2. Select **Gift cards** from your **Wallet** 22 | 3. Tap **View all** and scroll to view your cards 23 | 24 | When shopping online, enter the gift card number and PIN in the gift card section in the store’s checkout. If you’re shopping in\-store, provide the gift card number and PIN to the cashier. 25 | 26 | #### Related articles 27 | 28 | * [What happens after I‘ve reported my problem?](https://www.klarna.com/us/customer-service/what-happens-after-ive-reported-my-problem/) 29 | * [How do I return an order purchased with a gift card?](https://www.klarna.com/us/customer-service/how-do-i-return-an-order-purchased-with-a-gift-card/) 30 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/i-havent-received-my-order-do-i-have-to-pay.md: -------------------------------------------------------------------------------- 1 | # I haven’t received my order. Do I have to pay? 2 | 3 | No, you do not need to pay for your order until you have received all your purchased items. 4 | 5 | If you haven’t yet received your order and the due date is approaching, we kindly ask you to do the following to avoid any fees: 6 | 7 | 1. **Track your delivery** 8 | Use the tracking number provided by the store for updates on your order’s estimated delivery date. 9 | 2. **Contact the store** 10 | If no shipping information is available or your delivery is delayed, please contact the store directly. Only the store is able to help you locate your delivery or potentially replace your purchase if it got lost in the mail. 11 | 3. **Let us know you haven’t received your order** 12 | While you’re trying to find a solution with the store, make sure that you don’t miss any payments by [logging in](https://app.klarna.com/login?auto-login&market=us&utm_campaign=customer-service-faq&utm_medium=referral&utm_source=klarna) and letting us know you are still waiting for your delivery. Choose the purchase you’re still waiting to be delivered and select **Report a problem**. 13 | 4. **Check the app for updates** 14 | As soon as you’ve reported your problem, your payment plan will be paused until you’ve found a solution with the store. Your payment plan will be updated accordingly when the store updates your order. 15 | 5. **Keep an eye on your email and notifications** 16 | If no updates have been made to your payment plan and you still haven’t received your order, you will be asked to provide additional information so we can support you and start an investigation. It’s important that you save all relevant documentation until the issue is resolved. 17 | 18 | *Good to know:* Please also follow the steps above if you’ve only received part of your order. 19 | 20 | #### Related articles 21 | 22 | * [When will my order be delivered?](https://www.klarna.com/us/customer-service/when-will-my-order-be-delivered/) 23 | * [What happens after I‘ve reported my problem?](https://www.klarna.com/us/customer-service/what-happens-after-ive-reported-my-problem/) 24 | * [How long does it take for my problem to be solved?](https://www.klarna.com/us/customer-service/how-long-does-it-take-for-my-problem-to-be-solved/) 25 | * [What should I do if the store doesn’t respond?](https://www.klarna.com/us/customer-service/what-should-i-do-if-the-store-doesnt-respond/) 26 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/if-the-store-only-confirms-part-of-my-order-will-my-payment-plan-be-adjusted.md: -------------------------------------------------------------------------------- 1 | # If the store only confirms part of my order will my payment plan be adjusted? 2 | 3 | Yes. If only part of your order has been shipped by the store, your payment plan is automatically adjusted before your 3rd payment is withdrawn. The adjusted payment plan then reflects the refunded amount of any missing items. 4 | 5 | If the missing items of your order are then shipped after your 3rd payment is withdrawn, the shipment is listed in the app as a separate order with a new payment plan. 6 | 7 | #### Did you use a One\-time card? 8 | 9 | Klarna will automatically adjust your payment plan after 14 days. But you can manually request the unused amount be refunded right away: 10 | 11 | 1. Go to [**Payments**](https://app.klarna.com/to-do/to-pay/) 12 | 2. Choose the incorrect payment plan 13 | 3. Tap **Refund unused card amount** 14 | 15 | *Good to know:* The One\-time card service fee amount will not change. 16 | 17 | 18 | #### Related articles 19 | 20 | * [What happens after I‘ve reported my problem?](https://www.klarna.com/us/customer-service/what-happens-after-ive-reported-my-problem/) 21 | * [How long does it take for my problem to be solved?](https://www.klarna.com/us/customer-service/how-long-does-it-take-for-my-problem-to-be-solved/) 22 | * [How long do refunds take?](https://www.klarna.com/us/customer-service/how-long-do-refunds-take/) 23 | 24 |   25 | 26 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/is-shopping-with-klarna-safe.md: -------------------------------------------------------------------------------- 1 | # Is shopping with Klarna safe? 2 | 3 | The highest security standards are used to protect your data and personal information. 4 | 5 | * All payment information is processed securely 6 | * No details are transferred to or held by the store 7 | * All transactions take place via secured connections following strict security protocols 8 | * You can remove your linked bank account or card at any time 9 | 10 | If you want to know more about the way we handle your personal information, please read our [Privacy Policy](https://cdn.klarna.com/1.0/shared/content/legal/terms/0/en_us/privacy). 11 | 12 | See our [Cookie Policy](https://www.klarna.com/us/cookies/) for information about how we use cookies and your ability to choose how to use them. 13 | 14 | #### Related articles 15 | 16 | * [Why do I need to verify my details when paying?](https://www.klarna.com/us/customer-service/why-do-i-need-to-verify-my-details-when-paying/) 17 | * [I don’t recognize a charge, what should I do?](https://www.klarna.com/us/customer-service/i-dont-recognize-a-charge-what-should-i-do/) 18 | * [What is phishing and how can I identify it?](https://www.klarna.com/us/customer-service/what-is-phishing-and-how-can-i-identify-it/) 19 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/klarna-perform-credit-check.md: -------------------------------------------------------------------------------- 1 | # Does Klarna perform a credit check and will this affect my credit score? 2 | 3 | Klarna evaluates your ability to pay by assessing available data to ensure responsible lending. 4 | 5 | When a credit check is performed, we verify your identity using the details you provided and we look at information from your credit report to understand your financial behavior and evaluate your creditworthiness. 6 | 7 | We will **not** perform a credit check when: 8 | 9 | * Signing up to use Klarna 10 | * Downloading the [app](https://www.klarna.com/us/klarna-app/) 11 | 12 | We will perform a soft credit check when you: 13 | 14 | * Use [Pay in 4](https://www.klarna.com/us/pay-in-4/) 15 | * Use [Pay in 30 days](https://cdn.klarna.com/1.0/shared/content/legal/terms/0/en_us/invoice?_gl=1*y50jvt*_gcl_aw*R0NMLjE2Mzk5OTg2NTIuQ2owS0NRaUE4SUNPQmhEbUFSSXNBRUdJNm8xcVUwX0oxM2VfTkRfemR0WW9va2tMaFFzdzNJTHN6clpadV9ab1hUNmpJX2tCRnhPQUNqc2FBa251RUFMd193Y0I.) 16 | * Use [Monthly financing](https://www.klarna.com/us/legal/) 17 | * Apply for a [Klarna Card](https://www.klarna.com/us/klarna-card/) 18 | 19 | Soft credit checks do not affect your credit score and will not be visible to other lenders. 20 | 21 | **Good to know**: If you make late payments or fail to pay, this could have a negative impact on your creditworthiness and prevent you from using our payment methods. Read more about [how these credit checks impact your credit score](https://www.klarna.com/us/customer-service/does-klarna-affect-my-credit-score/). 22 | 23 | Our third party credit checks are performed by [Transunion](https://www.transunion.com/). If you have questions regarding their services, your credit score, or the data provided as part of a credit check we advise you to reach out to them directly. 24 | 25 | These checks are performed in accordance with [Klarna’s Terms \& Conditions](https://www.klarna.com/us/legal/). 26 | 27 | #### Related article 28 | 29 | * [Does Klarna affect my credit score?](https://www.klarna.com/us/customer-service/does-klarna-affect-my-credit-score/) 30 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/my-order-was-cancelled-why-am-i-still-being-charged.md: -------------------------------------------------------------------------------- 1 | # My order was cancelled, why am I still being charged? 2 | 3 | If it appears that you are still being charged, even though the order is canceled, it is most likely due to the store not releasing the authorization hold. 4 | 5 | **What is an authorization hold?** 6 | 7 | * An authorization hold is an amount placed on your card to ensure enough funds are available for your first payment. 8 | * The hold is released by the store when the order is canceled. 9 | * Depending on your financial institution, the release will be reflected in your bank account between 1 to 7 business days. 10 | 11 | **What should I do?** 12 | If you are still being charged, or the authorization hold has not been released after a couple of days, we recommend [chatting](https://app.klarna.com/support/) with our Customer Service with your cancelation confirmation on hand. 13 | 14 | #### Related articles 15 | 16 | * [How do I cancel my order?](https://www.klarna.com/us/customer-service/how-do-i-cancel-my-order/) 17 | * [How long do refunds take?](https://www.klarna.com/us/customer-service/how-long-do-refunds-take/) 18 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/my-problem-has-been-solved-what-should-i-do.md: -------------------------------------------------------------------------------- 1 | # My problem has been solved, what should I do? 2 | 3 | If your problem was solved, you can easily resolve your issue in the Klarna app. If you report your problem as solved, it’s important to be aware that a problem can’t be reported again. Resolving your issue tells us that the problem has been resolved to your satisfaction. 4 | 5 | **To report your problem as solved:** 6 | 7 | 1. Go to [**Payments**](https://app.klarna.com/transactions/my-klarna/) 8 | 2. Choose the purchase you want to update 9 | 3. Select **Problem solved** at the bottom of the page 10 | 11 | We will automatically close the investigation and update your payment plan. If any payments are still outstanding for that purchase, you can resume your payment by [logging in](https://app.klarna.com/to-do/to-pay/). 12 | 13 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/my-purchase-was-denied-why-do-i-still-have-a-pending-authorization-hold-on-my-card.md: -------------------------------------------------------------------------------- 1 | # My purchase was denied, why do I still have a pending authorization hold on my card? 2 | 3 | The transaction on your payment card is an authorization hold, which is released within 24 hours if the Pay in 4 purchase is declined. 4 | 5 | *Good to know:* When the hold is released your account balance will update automatically. Depending on your financial institution, it can take between 1 to 7 business days for this to reflect on your account balance. 6 | 7 | In the case that you receive a payment plan for a failed or canceled purchase, it could be due to the store not canceling the order in their system. Contact the store to cancel the order. If the store is unable to cancel the order, [contact us](https://app.klarna.com/login) with the cancellation confirmation to hand. 8 | 9 | #### Related articles 10 | 11 | * [Why was my purchase not approved with Klarna?](https://www.klarna.com/us/customer-service/why-was-my-purchase-not-approved-with-klarna/) 12 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/my-return-hasnt-been-registered-what-should-i-do.md: -------------------------------------------------------------------------------- 1 | # My return hasn’t been registered, what should I do? 2 | 3 | Only the store knows the status of your return and can adjust the total amount of your purchase. Because of this, we advise you to contact the store directly. Once the store has registered your return, your payment plan will be adjusted and you will receive an email and app notification. Potential repayments will be processed within 14 days. How your refund will be processed depends on the payment method used. 4 | 5 | Submit the returns information at any time in the [app](https://app.klarna.com/login) – follow the prompts to enter the tracking details \& the date you made the return. If after 21 days your return is not registered correctly by the store we can always support you if you’ve provided the required returns information. 6 | 7 | And always be sure to keep copies of your correspondence or proof of your attempts to contact the store. 8 | 9 | #### Related articles 10 | 11 | * [How long do refunds take?](https://www.klarna.com/us/customer-service/how-long-do-refunds-take/) 12 | * [How do I return my order?](https://www.klarna.com/us/customer-service/how-do-i-return-my-order/) 13 | * [How long does it take for my return to be registered?](https://www.klarna.com/us/customer-service/how-long-does-it-take-for-my-return-to-be-registered/) 14 | * [What should I do if the store doesn’t respond?](https://www.klarna.com/us/customer-service/what-should-i-do-if-the-store-doesnt-respond/) 15 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/theres-something-wrong-with-my-order-do-i-have-to-pay.md: -------------------------------------------------------------------------------- 1 | # There’s something wrong with my order. Do I have to pay? 2 | 3 | No, you do not need to pay for items that you did not order, are faulty, or broken. Please contact the store to let them know you received the wrong order and follow their instructions. 4 | 5 | **Let us know if there’s something wrong with your order:** 6 | 7 | 1. **Go to**[**Payments**](https://app.klarna.com/transactions/my-klarna/)and select the order you want to cancel 8 | 2. Tap **Report a problem** 9 | 3. Select the option that suits you best and follow the instruction to pause your payments 10 | 11 | By doing this, your payment plan will be paused until you’ve resolved the issue with the store. You can [check the status](https://app.klarna.com/transactions/my-klarna) of your reported problem on the details page of your purchase. 12 | 13 | **Good to know:** Keep an eye on your email inbox as we may contact you after 21 days to request additional information. 14 | 15 | #### Related articles 16 | 17 | * [What happens after I‘ve reported my problem?](https://www.klarna.com/us/customer-service/what-happens-after-ive-reported-my-problem/) 18 | * [How long does it take for my problem to be solved?](https://www.klarna.com/us/customer-service/how-long-does-it-take-for-my-problem-to-be-solved/) 19 | * [How long do refunds take?](https://www.klarna.com/us/customer-service/how-long-do-refunds-take/) 20 | * [What should I do if the store doesn’t respond?](https://www.klarna.com/us/customer-service/what-should-i-do-if-the-store-doesnt-respond/) 21 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/what-happens-after-ive-reported-my-problem.md: -------------------------------------------------------------------------------- 1 | # What happens after I‘ve reported my problem? 2 | 3 | As soon as you’ve reported your problem your payment plan is paused. This gives you extra time to find a solution with the store. 4 | 5 | If after 21 days you’re unable to find a resolution with the store, you will receive a request from us for additional information regarding your issue. Ensure to keep all relevant documentation. We can only support you if you provide us with the necessary information upon request. 6 | 7 | **What happens next?** 8 | 9 | After reviewing the information provided, we’ll notify you of the decision as soon as the investigation gets closed. 10 | 11 | * **If the investigation is closed in your favor,** we’ll update your payment plan accordingly. In case you have paid too much we will refund any overpayments. 12 | * **If we close the investigation in the store’s favor,** your original payments will still apply, but the payment schedule will be updated accordingly. This means that any outstanding payments will be due. 13 | 14 | *Good to know:* The status of the investigation can be found on the [details page of your purchase](https://app.klarna.com/transactions/my-klarna/). If you’ve resolved the issue with the store after reporting your problem, you can easily update us by [logging in](https://app.klarna.com/). 15 | 16 | #### Related articles 17 | 18 | * [What’s the status of the investigation into my problem?](https://www.klarna.com/us/customer-service/whats-the-status-of-the-investigation-into-my-problem/) 19 | * [How long does it take for my problem to be solved?](https://www.klarna.com/us/customer-service/how-long-does-it-take-for-my-problem-to-be-solved/) 20 | * [Why was the investigation of my problem closed, even though my problem has not been solved yet?](https://www.klarna.com/us/customer-service/why-was-the-investigation-of-my-problem-closed-even-though-my-problem-has-not-been-solved-yet/) 21 | 22 |   23 | 24 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/what-happens-if-i-cant-pay-on-time.md: -------------------------------------------------------------------------------- 1 | # What happens if I can’t pay on time? 2 | 3 | If we are unable to collect the payment on the scheduled date, we will try again. If that payment attempt also fails the missed payment will be added to the amount of the next scheduled payment. Late fees may also apply. Please check the terms and conditions of your order for details. 4 | 5 | **Good to know**: Missed payments and unpaid debts are sent to debt collection. 6 | 7 | #### Need a few extra days? 8 | 9 | You can extend your next payment date once per order: 10 | 11 | 1. Go to [**Purchases**](https://app.klarna.com/to-do/to-pay/) 12 | 2. Select the order you would like to pay later 13 | 3. Tap **Manage payments** and then tap **Extend due date** 14 | 15 | **Good to know:** It is not possible to extend the due date for purchases placed with Financing. 16 | 17 | If you are struggling to make a payment or experiencing unexpected financial hardship [contact us](#cs-contact-details) \- we are here to help. 18 | 19 | #### Related articles 20 | 21 | * [How can I extend my due date?](https://www.klarna.com/us/customer-service/how-can-i-extend-my-due-date/) 22 | * [How do I return my order?](https://www.klarna.com/us/customer-service/how-do-i-return-my-order/) 23 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/what-happens-if-my-purchase-is-for-a-different-amount-than-the-one-time-card.md: -------------------------------------------------------------------------------- 1 | # What happens if my purchase is for a different amount than the One-time card? 2 | 3 | **In case of a higher purchase amount** 4 | You must cancel the One\-time card and then create a new One\-time card for the full purchase amount. 5 | 6 | **In case of a lower purchase amount** 7 | No worries – you’ll only be billed for what you end up using, including the service fee. When you are sure your whole order has been processed, [log in](https://app.klarna.com/login/) and select **Refund unused amount**. If you don’t use the option in the app, Klarna will automatically adjust the plan after 14 days. 8 | 9 | *Good to know:* The authorization hold placed on your payment card will be removed. This can take between 1 to 7 days depending on your bank. 10 | 11 | #### Related articles 12 | 13 | * [What is a One\-time card and how does it work?](https://www.klarna.com/us/customer-service/what-is-a-one-time-card-and-how-does-it-work/) 14 | * [What if I never use the One\-time card and want to cancel it?](https://www.klarna.com/us/customer-service/what-if-i-never-use-the-one-time-card-and-want-to-cancel-it/) 15 | * [Why was the One\-time card purchase declined?](https://www.klarna.com/us/customer-service/why-was-the-one-time-card-purchase-declined/) 16 | * [How long do refunds take?](https://www.klarna.com/us/customer-service/how-long-do-refunds-take/) 17 | 18 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/what-happens-when-i-report-a-return.md: -------------------------------------------------------------------------------- 1 | # What happens when I report a return? 2 | 3 | As soon as you’ve reported your return your invoice is paused. This allows time for the return to be registered by the store and your invoice to be updated accordingly. You will receive an email with updated payment information or you can always log in to the [app](https://app.klarna.com/login) for up\-to\-date payment details. 4 | 5 | Contact the store directly if your return is not registered correctly, as only they can adjust your remaining balance. Keep in mind that Klarna manages your payment while the store is responsible for all other aspects concerning your order. 6 | 7 | Submit the returns information at any time in the [app](https://app.klarna.com/login) – follow the prompts to enter the tracking details \& the date you made the return. If after 21 days your return is not registered correctly by the store we can always support you if you’ve provided the required returns information. 8 | 9 | And always be sure to keep copies of your correspondence or proof of your attempts to contact the store. 10 | 11 | **What happens next?** 12 | 13 | After reviewing the information provided, we’ll notify you of the decision as soon as the investigation gets closed. 14 | 15 | * **If the investigation is closed in your favor,** we’ll update your payment plan accordingly. In case you have paid too much we will refund any overpayments. 16 | * **If we close the investigation in the store’s favor,** your original payments will still apply, but the payment schedule will be updated accordingly. This means that any outstanding payments will be due. 17 | 18 | *Good to know:* The status of the investigation can be found on the [details page of your purchase](https://app.klarna.com/transactions/my-klarna/). If you’ve resolved the issue with the store after reporting your problem, you can easily update us by [logging in](https://app.klarna.com/). 19 | 20 | #### Related articles 21 | 22 | * [How long does it take for my return to be registered?](https://www.klarna.com/us/customer-service/how-long-does-it-take-for-my-return-to-be-registered/) 23 | * [Can I change my return after it’s been reported?](https://www.klarna.com/us/customer-service/can-i-change-my-return-after-its-been-reported/) 24 | * [My return hasn’t been registered, what 25 | should I do?](https://www.klarna.com/us/customer-service/my-return-hasnt-been-registered-what-should-i-do/) 26 | * [How long do refunds take?](https://www.klarna.com/us/customer-service/how-long-do-refunds-take/) 27 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/what-if-i-never-use-the-one-time-card-and-want-to-cancel-it.md: -------------------------------------------------------------------------------- 1 | # What if I never use the One-time card and want to cancel it? 2 | 3 | The One\-time card will expire if you don’t use it at the store’s checkout within 24 hours after it’s created. You can also cancel the One\-time card instantly if it’s no longer needed. When either of these actions occurs, we will cancel any payment plan you might have, including the service fee. 4 | 5 | **To cancel an unused One\-time card:** 6 | 7 | 1. Go to [**Wallet**](https://app.klarna.com/transactions/my-klarna/) 8 | 2. Select the card or payment plan you want to cancel 9 | 3. Tap **Cancel this card** 10 | 11 | **Good to know:**If you want to make another purchase, just create a new card and keep on shopping. 12 | 13 | #### Related articles 14 | 15 | * [What is a One\-time card and how does it work?](https://www.klarna.com/us/customer-service/what-is-a-one-time-card-and-how-does-it-work/) 16 | * [What happens if my purchase is for a different amount than the One\-time card?](https://www.klarna.com/us/customer-service/what-happens-if-my-purchase-is-for-a-different-amount-than-the-one-time-card/) 17 | * [Why was the One\-time card purchase declined?](https://www.klarna.com/us/customer-service/why-was-the-one-time-card-purchase-declined/) 18 | 19 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/what-is-a-one-time-card-and-how-does-it-work.md: -------------------------------------------------------------------------------- 1 | # What is a One-time card and how does it work? 2 | 3 | The One\-time card service enables you to use Klarna payment options at any VISA card accepting online store in the US. 4 | 5 | You can use the One\-time card service in the store’s checkout, like any regular credit card. Each One\-time card will follow a specific payment plan, which you can manage and pay in the app. 6 | 7 | Please be aware that an authorization hold might be placed on your payment card when you use the One\-time card. This ensures that enough funds are available on your credit or debit card. 8 | 9 | #### How to create a One\-time card 10 | 11 | 1. [**Log in**](https://login.klarna.com/login) **to shop with us** 12 | Choose Pay with Klarna or create a new One\-time card in the [app](https://login.klarna.com/login?market=GB&ui_locales=en-US#). 13 | 2. **Set the spending amount** 14 | Enter the exact amount, including shipping, that you plan to spend. After entering the amount, you will receive an overview of the agreement as confirmation. A flat service fee (or APR for longer payment plans) will be added to your purchase amount. 15 | 3. **Check out on the store’s site** 16 | When you’re ready to check out, your address details will be prefilled based on your information on file. If we don’t already have your information, you can enter your standard billing and shipping addresses. 17 | 4. **Pay with the One\-time card** 18 | When you are ready to pay, select the credit card option and enter the One\-time card information, as you would with any other card. Enter the card number, expiration date, and CVV code. 19 | 5. **Check the app or** [**log in**](https://login.klarna.com/login) **for updates** 20 | If you end up spending less than the amount on the One\-time card, you can change the spending amount by selecting **Refund unused amount** on the [purchase page](https://app.klarna.com/transactions/my-klarna/). Otherwise, the amount will automatically be adjusted after 14 days. 21 | 22 | *Good to know:* You can only use a One\-time card once and, if unused, the card expires after 24 hours. If you want to make another purchase, just create a new card and keep on shopping. 23 | 24 | #### Where can I use the One\-time card? 25 | 26 | At any VISA card\-accepting online store in the US. Just search for your favorite store in the search bar, create a One\-time card and use it to checkout. 27 | 28 | #### How do I pay for the purchases made with the One\-time card? 29 | 30 | When you create a One\-time card it’s connected to your personal debit or credit card, depending on the payment option you chose. Your first payment is debited 24 hours after the card is created and the following payments are automatically withdrawn according to the payment plan. You can view your payment plan, pay early and change your connected debit or credit card at any time in the app. 31 | 32 | #### When am I charged a service fee? 33 | 34 | You are charged a service fee when you use a One\-time card to make bi\-weekly payments for a purchase on a non\-partner retailer that doesn’t have Klarna as a payment method. In addition, you are charged a service fee to pay by creating a gift card at certain merchants. 35 | 36 | #### Related articles 37 | 38 | * [What happens if my purchase is for a different amount than the One\-time card?](https://www.klarna.com/us/customer-service/what-happens-if-my-purchase-is-for-a-different-amount-than-the-one-time-card/) 39 | * [What if I never use the One\-time card and want to cancel it?](https://www.klarna.com/us/customer-service/what-if-i-never-use-the-one-time-card-and-want-to-cancel-it/) 40 | * [Why was the One\-time card purchase declined?](https://www.klarna.com/us/customer-service/why-was-the-one-time-card-purchase-declined/) 41 | 42 |   43 | 44 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/what-is-financing-and-how-does-it-work.md: -------------------------------------------------------------------------------- 1 | # What is Financing and how does it work? 2 | 3 | Financing allows you to spread the cost of your purchase for up to 24 months. 4 | 5 | #### How to finance your purchase: 6 | 7 | 1. Select the Financing option in the store’s checkout 8 | 2. Choose your preferred timeline and agreed interest rate 9 | 3. Follow the steps to complete the application 10 | 4. Get an instant approval decision 11 | 5. Complete your purchase 12 | 13 | You’ll receive an email from us with your payment plan when the store ships your order. 14 | 15 | *Please note:* [A credit check will be performed](https://www.klarna.com/us/customer-service/klarna-perform-credit-check/) when you apply for our Financing options. 16 | 17 | #### When do I pay? 18 | 19 | Your first Financing payment is due one month after the store processes your order and then monthly thereafter on the same date each month until your statement is fully paid.  20 | 21 | [Log in](https://app.klarna.com/login) to view your payment plan and upcoming payments. 22 | 23 | **Good to know:** 24 | 25 | * Our Financing products are issued by WebBank (member FDIC) in partnership with Klarna. 26 | * Read our [Terms \& Conditions](https://www.klarna.com/us/legal/) for more detailed information about Financing and the Annual Percentage Rate (APR). 27 | 28 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/what-is-pay-later-in-30-days.md: -------------------------------------------------------------------------------- 1 | # What is ‘Pay later in 30 days’? 2 | 3 | Klarna’s ‘Pay in 30 days’ allows you to receive your order up front and get up to 30 days to pay without any interest and fees. In our app you’ll have the option to pay off the balance earlier or extend the due date to a later date for a fee if needed. 4 | 5 | If your order is eligible for ‘Pay in 30 days’, the option will be presented at the time of checkout. As soon as the store ships your order, you will receive an email from Klarna outlining your payments. Your purchase and payments will then be visible and manageable in your Klarna App. So make sure to download the [Klarna App](https://www.klarna.com/us/klarna-app/) to stay in full control and never miss a payment! 6 | 7 | *Good to know*: There is no fee to use ‘Pay in 30 days’! 8 | 9 | Find more information about your payment options with Klarna [here](https://www.klarna.com/us/what-is-klarna/). 10 | 11 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/what-is-phishing-and-how-can-i-identify-it.md: -------------------------------------------------------------------------------- 1 | # What is phishing and how can I identify it? 2 | 3 | Phishing is when someone initiates fraudulent communications that appear to come from a trusted source. It mainly happens through email (but it can happen through any channel) and the goal is to steal your credit card number, your password or other sensitive data to then use it, for example, to purchase goods or services. 4 | 5 | #### To spot a scam or phishing attack, ask yourself: 6 | 7 | * Is the sender unknown to you? 8 | * Is this contact you received unexpected? 9 | * Are there any spelling or grammar mistakes? 10 | * Are there unusual links or attachments? 11 | * Are you being asked for personal details and security information? 12 | * Is the sender rushing you to make a decision? 13 | 14 | If the answer is **Yes** to some or all of these questions, it might be a potential phishing attack. We encourage you to immediately interrupt communication (delete the email or hang up the phone) and possibly report the phishing attempt to the trusted source. 15 | 16 | #### Related articles 17 | 18 | * [How can I report a fraud attempt?](https://www.klarna.com/us/customer-service/how-can-i-report-a-fraud-attempt/) 19 | * [I don’t recognize a charge, what should I do?](https://www.klarna.com/us/customer-service/i-dont-recognize-a-charge-what-should-i-do/) 20 | * [How can I protect myself against identity theft?](https://www.klarna.com/us/customer-service/how-can-i-protect-myself-against-identity-theft/) 21 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/what-should-i-do-if-the-store-doesnt-respond.md: -------------------------------------------------------------------------------- 1 | # What should I do if the store doesn’t respond? 2 | 3 | If you have contacted the store to solve your problem but haven’t had any success, you can inform us about your issue by [reporting a problem](https://app.klarna.com/transactions/my-klarna/) for the order. 4 | 5 | As soon as you’ve reported your problem your payments are paused. This gives you extra time to find a solution with the store. Ensure to save all relevant documentation. For us to be able to support you, you must provide us with necessary information upon request. 6 | 7 | **Good to know:** Keep an eye on your email inbox as we may contact you after 21 days to request additional information. 8 | 9 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/whats-the-status-of-the-investigation-into-my-problem.md: -------------------------------------------------------------------------------- 1 | # What’s the status of the investigation into my problem? 2 | 3 | You can always check the status and outcome of the investigation in the app. 4 | 5 | 1. Go to [**Payments**](https://app.klarna.com/transactions/my-klarna/) 6 | 2. Select the purchase you want to check 7 | 3. Find the details regarding the investigation in the information box 8 | 9 | **Good to know:** Be sure to check your emails because after 21 days we will contact you for additional information. 10 | 11 | #### Related articles 12 | 13 | * [How long does it take for my problem to be solved?](https://www.klarna.com/us/customer-service/how-long-does-it-take-for-my-problem-to-be-solved/) 14 | * [Why was the investigation of my problem closed, even though my problem has not been solved yet?](https://www.klarna.com/us/customer-service/why-was-the-investigation-of-my-problem-closed-even-though-my-problem-has-not-been-solved-yet/) 15 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/whats-the-status-of-the-investigation-of-my-return.md: -------------------------------------------------------------------------------- 1 | # What’s the status of the investigation of my return? 2 | 3 | The current status or outcome of the investigation is always visible in the Klarna app: 4 | 5 | 1. Go to [**Payments**](https://app.klarna.com/transactions/my-klarna/) 6 | 2. Select the purchase you want to check 7 | 3. View any investigation updates in the information box 8 | 9 | *Good to know:* Submit the returns information at any time in the [app](https://app.klarna.com/login) – follow the prompts to enter the tracking details \& the date you made the return. If after 21 days your return is not registered correctly by the store we can always support you if you provided the required returns information. And always be sure to keep copies of your correspondence or proof of your attempts to contact the store. 10 | 11 | #### Related articles 12 | 13 | * [Can I change my return after it’s been reported?](https://www.klarna.com/us/customer-service/can-i-change-my-return-after-its-been-reported/) 14 | * [Why was the investigation of my return closed, even though my return hasn’t been registered yet?](https://www.klarna.com/us/customer-service/why-was-the-investigation-of-my-return-closed-even-though-my-return-hasnt-been-registered-yet/) 15 | * [What happens when I report a return?](https://www.klarna.com/us/customer-service/what-happens-when-i-report-a-return/) 16 | * [My problem has been solved, what should I do?](https://www.klarna.com/us/customer-service/my-problem-has-been-solved-what-should-i-do/) 17 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/when-is-my-payment-due.md: -------------------------------------------------------------------------------- 1 | # When is my payment due? 2 | 3 | The first payment is withdrawn when you check out on the store’s website, the remaining 3 payments are scheduled every 2 weeks. 4 | 5 | Log in to the app to check when your payment is due: 6 | 7 | * [Log in](https://app.klarna.com/login/) to the app 8 | * Tap **Payments** and select the order for payment information 9 | 10 | You will receive order confirmation by email once the store confirms your order and you will also receive payment reminders by email and in the app. 11 | 12 | *Good to know:* Download the app and enable notifications to ensure you never miss a payment date. 13 | 14 | #### Related articles 15 | 16 | * [What happens if I can’t pay on time?](https://www.klarna.com/us/customer-service/what-happens-if-i-cant-pay-on-time/) 17 | * [How can I extend my due date](https://www.klarna.com/us/customer-service/how-can-i-extend-my-due-date/) 18 | * [How do I return my order?](https://www.klarna.com/us/customer-service/how-do-i-return-my-order/) 19 | * [I haven’t received my order. Do I have to pay?](https://www.klarna.com/us/customer-service/i-havent-received-my-order-do-i-have-to-pay/) 20 | * [Where can I find my payment plan?](https://www.klarna.com/us/customer-service/where-can-i-find-my-payment-plan/) 21 | 22 |   23 | 24 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/when-will-i-receive-my-klarna-card-refund.md: -------------------------------------------------------------------------------- 1 | # When will I receive my Klarna Card refund? 2 | 3 | Your refund is processed as soon as we receive the refund or cancellation confirmation from the store. It can then take up to 14 days for the funds to clear and your connected bank account balance to update. 4 | 5 | *Good to know:* 6 | 7 | * The amount is refunded to your connected bank account. 8 | * If we are unable to match your refund to an order, the unmatched refund will be added to the next statement as a credit. 9 | 10 | #### Related articles 11 | 12 | * [How long do refunds take?](https://www.klarna.com/us/customer-service/how-long-do-refunds-take/) 13 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/when-will-my-klarna-card-be-delivered.md: -------------------------------------------------------------------------------- 1 | # When will my Klarna Card be delivered? 2 | 3 | Your virtual card will be available in the app as soon as you are approved. 4 | 5 | To use your virtual Klarna Card, add it to your digital wallet: 6 | 7 | 1. Go to **Klarna Card** 8 | 2. Tap Add to Apple Wallet or Add to Google Pay 9 | 3. Follow the prompts to add the card to your **Wallet** 10 | 11 | 12 | 13 | 14 | #### I’d like to order a physical card 15 | 16 | You will soon be able to order a physical card. Keep an eye for updates in the [app](https://www.klarna.com/us/klarna-app/).   17 | 18 | 19 | 20 | 21 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/when-will-my-order-be-delivered.md: -------------------------------------------------------------------------------- 1 | # When will my order be delivered? 2 | 3 | Deliveries are subject to the store’s delivery services, so please check the store’s website for information on their shipping policies. Use the tracking number provided by the store for updates on the estimated delivery date. 4 | 5 | When your order is processed, you’ll receive an email with the tracking and shipping details of your order. If you don’t receive any shipping confirmation, please contact the store to get updates on your delivery. 6 | 7 | *Good to know:* After you place an order on the store’s website, it may take a few days for the store to process your order. If you’d like to know the current status of your order, please contact the store directly. 8 | 9 | #### Are you still waiting for your order? 10 | 11 | If your order doesn’t arrive, please let us know before the payment deadline expires by [logging in](https://app.klarna.com/login?auto-login&market=us&utm_campaign=customer-service-faq&utm_medium=referral&utm_source=klarna) to the app. Choose the purchase you’re still waiting to be delivered and select **Report a problem**. As soon as you’ve reported your problem, your payment plan is paused. This gives you extra time to find a solution with the store. 12 | 13 | #### Tired of not having the shipping information for all your orders in one place? 14 | 15 | By using our **Email Connect** you can connect your Google Email to automatically view and track your deliveries directly in the Klarna app. 16 | 17 | #### Related articles 18 | 19 | * [I haven’t received my order. Do I have to pay?](https://www.klarna.com/us/customer-service/i-havent-received-my-order-do-i-have-to-pay/) 20 | * [What happens after I‘ve reported my problem?](https://www.klarna.com/us/customer-service/what-happens-after-ive-reported-my-problem/) 21 | * [What should I do if the store doesn’t respond?](https://www.klarna.com/us/customer-service/what-should-i-do-if-the-store-doesnt-respond/) 22 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/where-can-i-find-my-payment-plan.md: -------------------------------------------------------------------------------- 1 | # Where can I find my payment plan? 2 | 3 | **To view all your payment plans:** 4 | 5 | 1. Go to [**Payments**](https://app.klarna.com/to-do/to-pay/) to see your orders 6 | 2. Select an order to view your payment plan 7 | 8 | We’ll also send you an email with the payment plan when the store ships your order. If you don’t see an email from Klarna in your inbox, make sure to check your spam folder as well. 9 | 10 | **I still can’t find my payment plan** 11 | 12 | If you don’t see the order you are looking for in the app: 13 | 14 | * Verify you logged in with the same email address you used to place the order 15 | * Check you received the confirmation email from the store and log in with that linked email 16 | * [Chat](https://app.klarna.com/support/) with our customer service to change the email address connected to the order 17 | 18 | We recommend always using the same email address when shopping with Klarna. 19 | 20 | *Good to know:* If you place an order by signing up with **Apple ID** with a hidden email address you won’t be able to see the order in the app as it is linked to a random email address. Chat with our Customer Service to change the email address linked to the order. 21 | 22 | #### Related articles 23 | 24 | * [How can I pay?](https://www.klarna.com/us/customer-service/how-can-i-pay/) 25 | * [I haven’t received my order. Do I have to pay?](https://www.klarna.com/us/customer-service/i-havent-received-my-order-do-i-have-to-pay/) 26 | * [How can I change my email address?](https://www.klarna.com/us/customer-service/how-can-i-change-my-email-address/) 27 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/where-can-i-use-klarna.md: -------------------------------------------------------------------------------- 1 | # Where can I use Klarna? 2 | 3 | You can use Klarna at partnering physical shops and at any prepaid card\-accepting online store in the US. 4 | 5 | [Explore the stores](https://www.klarna.com/us/klarna-stores/) and shop with us directly from their websites or use the [app](https://login.klarna.com/login?market=GB&ui_locales=en-US#) to shop at any of your favorite stores and pay with Klarna. 6 | 7 | #### Can I use Klarna to buy anything? 8 | 9 | Almost! But there are some exceptions: 10 | 11 | * Utility bill or rent payments 12 | * Gift cards 13 | * Governmental bills \& fines 14 | * Medical care 15 | * Online gaming or gambling 16 | * Drugs or alcohol 17 | 18 | You can find more information in our [Terms \& Conditions](https://www.klarna.com/us/terms-of-use/). 19 | 20 | #### Related articles 21 | 22 | * [Can I pay in\-store using Klarna?](https://www.klarna.com/us/customer-service/can-i-pay-in-store-using-klarna/) 23 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/which-payment-methods-are-accepted-by-klarna.md: -------------------------------------------------------------------------------- 1 | # Which payment methods are accepted by Klarna? 2 | 3 | Klarna accepts major debit and credit cards such as Visa, Discover, Maestro and Mastercard. Prepaid cards are not accepted. 4 | 5 | Please note: 6 | 7 | * Capital One does not support Buy Now, Pay Later (BNPL) products. 8 | * We do not accept credit cards as a payment method for Financing or purchases made with the Klarna Card. 9 | 10 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/who-can-use-klarna.md: -------------------------------------------------------------------------------- 1 | # Who can use Klarna? 2 | 3 | To be eligible to use the [Klarna](https://login.klarna.com/login?market=GB&ui_locales=en-US#) you must: 4 | 5 | * Be a resident of  the United States or its territories 6 | * Be at least 18 7 | * Have a valid bank card/bank account 8 | * Have a positive credit history 9 | * Be able to receive verification codes via text 10 | 11 | #### Related articles 12 | 13 | * [How can I get approved to pay with Klarna?](https://www.klarna.com/us/customer-service/how-can-i-get-approved-to-pay-with-klarna/) 14 | * [Why was my purchase not approved with Klarna?](https://www.klarna.com/us/customer-service/why-was-my-purchase-not-approved-with-klarna/) 15 | * [Does Klarna affect my credit score?](https://www.klarna.com/us/customer-service/does-klarna-affect-my-credit-score/) 16 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/why-are-not-all-payment-methods-available-at-the-checkout.md: -------------------------------------------------------------------------------- 1 | # Why are not all payment methods available at the checkout? 2 | 3 | Klarna does not approve all orders for all our payment options and our aim is to support responsible, ethical, and sensible spending habits. Although our payment options are widely promoted they are not always universally available. We understand that it can be frustrating to not be able to choose the payment option you prefer. However, our approval process helps Klarna responsibly offer our services to our customers. Please understand that our Customer Service is not able to influence which payment options are available to you. 4 | 5 | There are a number of factors, including address details, cardholder details, amount of order, the online store, previous order history, that are taken into consideration to determine which payment methods are offered. Some stores that offer Klarna don’t offer all payment options. Even if the store offers your preferred payment options with Klarna, it may not be available for all products and purchases. At the time of your purchase, all available payment options will be presented at checkout. 6 | 7 | **Good to know**: Each purchase is a new decision so in the future, your preferred payment option might be available to you again. 8 | 9 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/why-cant-i-pay-with-klarna-anymore.md: -------------------------------------------------------------------------------- 1 | # Why can’t I pay with Klarna anymore? 2 | 3 | The decline reason is displayed in a pop\-up message, which appears at checkout, when you can’t use Klarna to pay. 4 | 5 | Follow the instructions in the pop\-up message before attempting to make another purchase. 6 | 7 | We understand it can be frustrating if you experience that not all or no Klarna payment options are available to you when making a purchase – especially having been approved in the past. Different factors are being assessed in real time to help our customers balance debt risk and stay on top of their spending. 8 | 9 | #### How the automated approval assessment works 10 | 11 | * For each order attempt, in addition to your Klarna payment history, [information from the credit bureaus](https://www.klarna.com/us/customer-service/why-was-my-purchase-not-approved-with-klarna/) is used to check your credit history, such as if you paid off other credits on time, or if you have too much outstanding debt elsewhere. 12 | * Every purchase attempt generates a new automated approval assessment based on available credit bureau decision data in real time, so one rejection does not mean that future purchases will always be declined. 13 | 14 | A declined purchase does not impact your [credit score.](https://www.klarna.com/us/customer-service/does-klarna-affect-my-credit-score/) 15 | 16 | *Good to know:* Our customer service agents do not have additional information about the automated decline reason, based on credit reference agency decision data, and are not able to change or influence the decision. 17 | 18 | #### Related articles 19 | 20 | * [How can I get approved to pay with Klarna?](https://www.klarna.com/us/customer-service/how-can-i-get-approved-to-pay-with-klarna/) 21 | * [Does Klarna affect my credit score?](https://www.klarna.com/us/customer-service/does-klarna-affect-my-credit-score/) 22 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/why-did-i-receive-a-reminder-after-returning-my-order.md: -------------------------------------------------------------------------------- 1 | # Why did I receive a reminder after returning my order? 2 | 3 | If the payment due date has passed but the return (or payment) has not been registered, you will receive a reminder. It can happen that the reminder was sent while your return was in transit or was still processing. 4 | 5 | If you have returned your order but still received a reminder, please let us know that you’re waiting for the return to be registered by reporting your return. 6 | 7 | **To report your return:** 8 | 9 | 1. Go to [**Payments**](https://app.klarna.com/transactions/my-klarna/) 10 | 2. Select the purchase you want to return 11 | 3. Tap **Report return** and follow the instruction to pause your invoice 12 | 13 | Once you have reported your return, your invoice will be paused until your return is registered. Your invoice will be adjusted when the store updates your order. 14 | 15 | [Log in](https://app.klarna.com/login/) to the app for any updates regarding your return. 16 | 17 | #### Related articles 18 | 19 | * [How long does it take for my return to be registered?](https://www.klarna.com/us/customer-service/how-long-does-it-take-for-my-return-to-be-registered/) 20 | * [How do I return my order?](https://www.klarna.com/us/customer-service/how-do-i-return-my-order/) 21 | * [Can I change my return after it’s been reported?](https://www.klarna.com/us/customer-service/can-i-change-my-return-after-its-been-reported/) 22 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/why-did-klarna-charge-me-twice.md: -------------------------------------------------------------------------------- 1 | # Why did Klarna charge me twice? 2 | 3 | You have not been charged twice, but you may temporarily see what appears to be two transactions for the same purchase on your account. 4 | 5 | The pending transaction you can see, also called an authorization hold, is the reservation of funds on your account to ensure you have sufficient funds reserved for the first installment of your Pay in 4 purchase. 6 | 7 | The pending hold is not debited from your account but it does temporarily reduce your available balance. The reservation of funds is released as soon as the processing of your order is complete. 8 | 9 | #### The pending hold has not been released 10 | 11 | There are several factors that affect how long the hold will appear on your account, such as if you placed the order in or outside of business hours, how long the store takes to process the order and your bank’s process. 12 | 13 | *Good to know:* Contact your bank or card issuer directly if after 7 business days the pending hold has not expired or been released. 14 | 15 | *Related articles* 16 | 17 | * [My purchase was denied, why do I still have a pending authorization hold on my card?](https://www.klarna.com/us/customer-service/my-purchase-was-denied-why-do-i-still-have-a-pending-authorization-hold-on-my-card/) 18 | * [What if I never use the One\-time card and want to cancel it?](https://www.klarna.com/us/customer-service/what-if-i-never-use-the-one-time-card-and-want-to-cancel-it/) 19 | * [My order was cancelled, why am I still being charged?](https://www.klarna.com/us/customer-service/my-order-was-cancelled-why-am-i-still-being-charged/) 20 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/why-do-i-need-to-verify-my-details-when-paying.md: -------------------------------------------------------------------------------- 1 | # Why do I need to verify my details when paying? 2 | 3 | We ask you to verify your details to ensure your information, and purchases associated with your email address, are shared securely. 4 | 5 | #### These are the ways by which we can verify your details: 6 | 7 | * We’ll send you a verification code by email to verify your email address 8 | * We’ll send you a verification code via SMS to verify your phone number 9 | * We’ll ask you to log into your bank to verify your identity 10 | * We will ask you to upload a photo ID together with a selfie 11 | 12 | #### What is PSD2? 13 | 14 | The Second EU Payment Services Directive (PSD2\) makes online payments more secure by requiring a stronger customer authentication procedure (SCA). Because of the directive, you need to confirm your payment with a second authentication step (two\-factor authentication). For example with a fingerprint in the online banking app or an additional verification code. 15 | 16 | #### Related articles 17 | 18 | * [Is shopping with Klarna safe?](https://www.klarna.com/us/customer-service/is-shopping-with-klarna-safe/) 19 | * [How can I request the deletion of my personal data?](https://www.klarna.com/us/customer-service/how-can-i-request-the-deletion-of-my-personal-data/) 20 | * [What is phishing and how can I identify it?](https://www.klarna.com/us/customer-service/what-is-phishing-and-how-can-i-identify-it/) 21 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/why-do-i-see-two-orders-in-the-klarna-app-when-i-only-placed-one.md: -------------------------------------------------------------------------------- 1 | # Why do I see two orders in the app when I only placed one? 2 | 3 | Orders can sometimes be split because the order was shipped separately. Each shipment is then listed in the app as a separate order. 4 | 5 | A separate payment plan is then generated for each shipment to ensure you are not charged for items that have not yet been shipped. 6 | 7 | The total number of payment plans generated for the split order will never exceed the original purchase amount of the order. 8 | 9 | But should you have been charged for something you have not purchased, or appear to be charged the wrong amount, let us know by reporting the problem: 10 | 11 | 1. Go to [**Payments**](https://app.klarna.com/transactions/my-klarna/) 12 | 2. Select the purchase in question 13 | 3. Tap **Report a problem** 14 | 15 | You will receive an email from us while your payments are paused. [Log in](https://app.klarna.com/login?auto-login&market=us&utm_campaign=customer-service-faq&utm_medium=referral&utm_source=klarna)  for payment information and updates about your reported problem. 16 | 17 | #### Related articles 18 | 19 | * [What happens after I’ve reported my problem?](https://www.klarna.com/us/customer-service/what-happens-after-ive-reported-my-problem/) 20 | * [How long does it take for my problem to be solved?](https://www.klarna.com/us/customer-service/how-long-does-it-take-for-my-problem-to-be-solved/) 21 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/why-is-my-debt-in-collection.md: -------------------------------------------------------------------------------- 1 | # Why is my debt in collection? 2 | 3 | If your payment is not registered by the last reminder due date, the debt is transferred to debt collection. 4 | 5 | #### Can I still pay Klarna directly? 6 | 7 | Yes, you are welcome to pay either directly to Klarna or the collection agency. Your outstanding balance will be updated in any case. 8 | 9 | #### To settle your balance through Klarna: 10 | 11 | 1. Go to [**Payments**](https://app.klarna.com/to-do/to-pay/) 12 | 2. Choose the purchase you would like to pay 13 | 3. Tap **Payment options** and follow the instructions 14 | 15 | Any questions regarding fees added for the collection of debt should be directed to the debt collection agency. You can find contact information for the collection agency printed on your debt collection letter. 16 | 17 | *Good to know:* If a payment is late, you will not be able to continue using Klarna until you’ve paid off your debt. Late payments might also be reported to the credit bureaus. 18 | 19 | #### Related articles 20 | 21 | * [What happens if I can’t pay on time?](https://www.klarna.com/us/customer-service/what-happens-if-i-cant-pay-on-time/) 22 | * [Why was I charged a late fee?](https://www.klarna.com/us/customer-service/why-was-i-charged-a-late-fee/) 23 | * [How can I avoid late fees?](https://www.klarna.com/us/customer-service/how-can-i-avoid-late-fees/) 24 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/why-was-i-charged-a-late-fee.md: -------------------------------------------------------------------------------- 1 | # Why was I charged a late fee? 2 | 3 | Late fees are added to your payment plan if: 4 | 5 | * The payment fails to be collected on the due date 6 | * The entire payment is missing for a purchase 7 | 8 | **Good to know:** If a payment is late, you will not be able to continue using Klarna until you’ve paid off your debt. 9 | 10 | #### Related articles 11 | 12 | * [How can I avoid late fees?](https://www.klarna.com/us/customer-service/how-can-i-avoid-late-fees/) 13 | * [How do I return my order?](https://www.klarna.com/us/customer-service/how-do-i-return-my-order/) 14 | * [I haven’t received my order. Do I have to pay?](https://www.klarna.com/us/customer-service/i-havent-received-my-order-do-i-have-to-pay/) 15 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/why-was-i-denied-a-klarna-card.md: -------------------------------------------------------------------------------- 1 | # Why was I denied a Klarna Card? 2 | 3 | There can be different reasons why you were denied when applying for a Klarna Card. 4 | 5 | To be eligible for a Klarna card, you must: 6 | 7 | * Be at least 18 years old 8 | * Must have paid with Klarna at least once before 9 | * Have a good credit history 10 | 11 | *Please note:* [A credit check will be performed](https://www.klarna.com/us/customer-service/klarna-perform-credit-check/) when applying for a Klarna Card. If your application is denied due to your current credit score, you’re welcome to apply again in the future. 12 | 13 | **What can I do?** 14 | A new version of the Klarna Card is coming. You can join the [waitlist](https://www.klarna.com/us/klarna-card-waitlist/) for updates. You'll receive a notification by email should you be eligible for the new version of the Klarna Card. 15 | 16 | #### Related articles 17 | 18 | * [How can I get approved to pay with Klarna?](https://www.klarna.com/us/customer-service/how-can-i-get-approved-to-pay-with-klarna/) 19 | * [How much am I eligible to spend?](https://www.klarna.com/us/customer-service/how-much-am-i-eligible-to-spend/) 20 | * [Does Klarna perform a credit check?](https://www.klarna.com/us/customer-service/klarna-perform-credit-check/) 21 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/why-was-my-klarna-card-purchase-declined.md: -------------------------------------------------------------------------------- 1 | # Why was my Klarna Card purchase declined? 2 | 3 | You can immediately see the reason why your Klarna Card purchase was declined either on the screen of the store’s card reader or on the pop\-up message which appeared at checkout. 4 | 5 | Follow the instructions in the pop\-up message before attempting to make another Klarna Card purchase.  6 | 7 | **How can I  get my Klarna Card purchase approved?** 8 | 9 | * **Be sure you haven’t exceeded your Purchase power** 10 | You can check your [Purchase power](https://app.klarna.com/purchase-power/) for the Klarna Card in our app 11 | * **Check you don’t have any unpaid invoices** 12 | Settle the amount due before making a new purchase with your Klarna Card. 13 | * **Avoid exceeding your daily spending limit** 14 | Manage spending responsibly. 15 | * **Your Klarna Card is temporarily frozen or blocked** 16 | Check you have, as required, connected a bank account to your Klarna Card then try to make the purchase again. 17 | 18 | *Good to know:* Your Klarna Card purchase might also have been declined because of the type of store at which you tried to use the card. While Klarna doesn’t monitor every store, we generally decline transactions for the following to promote financial wellness and best meet our regulatory, ethical, and risk standards: 19 | 20 | * Utility bills or rent payments 21 | * Food or delivery services 22 | * Gift card providers 23 | * Governmental agencies 24 | * Medical care 25 | * Online gaming or gambling 26 | * Selling of drugs or alcohol 27 | 28 | #### Related articles 29 | 30 | * [How much am I eligible to spend?](https://www.klarna.com/us/customer-service/how-much-am-i-eligible-to-spend/) 31 | * [How can I get approved to pay with Klarna?](https://www.klarna.com/us/customer-service/how-can-i-get-approved-to-pay-with-klarna/) 32 | * [How do I block or cancel my Klarna Card?](https://www.klarna.com/us/customer-service/how-do-i-block-or-cancel-my-klarna-card/) 33 | 34 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/why-was-my-purchase-not-approved-with-klarna.md: -------------------------------------------------------------------------------- 1 | # Why was my purchase not approved with Klarna? 2 | 3 | The reason is displayed in a pop\-up message, which appears at checkout, if you are not approved to pay with Klarna. Be sure to follow the instructions in the pop\-up message before attempting to make another purchase. 4 | 5 | We understand it can be frustrating if Klarna’s payment options are not available when making a purchase – especially having been approved in the past. Different factors are being assessed in real time to help our customers balance debt risk and stay on top of their spending. Rest assured, declined purchases do not impact your [credit score](https://www.klarna.com/us/customer-service/does-klarna-affect-my-credit-score/). 6 | 7 | *Good to know:* Common data points typically considered in automated approval decisions for Buy Now Pay Later (BNPL) services include: 8 | 9 | * Credit score 10 | * Income 11 | * Employment stability 12 | * Spending patterns 13 | * Debt\-to\-income ratio 14 | * Residential status 15 | * Identity and age verification 16 | * Credit history 17 | * Affordability assessment 18 | * Savings and investments 19 | * Outstanding credit 20 | 21 | Note, each purchase generates a new automated approval assessment so one rejection does not mean that your future purchases will also be declined. 22 | 23 | Our customer service agents do not have additional information about the automated decline reason, based on credit bureau decision data, and are not able to change or influence the decision. 24 | 25 | #### Related articles 26 | 27 | * [Does Klarna perform a credit check on me and will it affect my credit score?](https://www.klarna.com/us/customer-service/does-klarna-affect-my-credit-score/) 28 | * [How can I get approved to pay with Klarna?](https://www.klarna.com/us/customer-service/how-can-i-get-approved-to-pay-with-klarna/) 29 | * [My purchase was denied, why do I still have a pending authorization hold on my card?](https://www.klarna.com/us/customer-service/my-purchase-was-denied-why-do-i-still-have-a-pending-authorization-hold-on-my-card/) 30 | * [Why can’t I pay with Klarna anymore?](https://www.klarna.com/us/customer-service/why-cant-i-pay-with-klarna-anymore/) 31 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/why-was-the-first-installment-of-my-payment-plan-higher.md: -------------------------------------------------------------------------------- 1 | # Why was the first installment of my payment plan higher? 2 | 3 | If Pay in 4 equal payments is not available, you may be offered an alternative Pay in 4 payment option with a higher first installment, instead of 4 equal payments. 4 | 5 | This can happen because the order amount exceeded your purchase power, however, other credit factors are also assessed. 6 | 7 | Purchase power is the estimated amount available to spend using Klarna’s pay later products. Your spending limit and any outstanding balances are factored into the purchase power amount. 8 | 9 | *Good to know:* 10 | 11 | * Our Customer Service is not able to increase your spending limit nor has further information about the automated assessment. 12 | * When selecting 4 bi\-weekly payments as your payment plan on One\-time cards, there is a service fee charged alongside the first payment. 13 | #### What’s next? 14 | 15 | * If you would prefer a payment plan of 4 equal payments always ensure the total order amount is within your purchase power. 16 | * If the order amount exceeds either your spending limit, or the estimated purchase power, and you are offered an adjusted payment plan with a higher first installment amount – accept the offer in the checkout. 17 | #### Related articles 18 | 19 | * [Why was my One\-time card purchase declined?](https://www.klarna.com/us/customer-service/why-was-the-one-time-card-purchase-declined/) 20 | * [How much am I eligible to spend?](https://www.klarna.com/us/customer-service/how-much-am-i-eligible-to-spend/) 21 | * [Why did Klarna charge me twice?](https://www.klarna.com/us/customer-service/why-did-klarna-charge-me-twice/) 22 | 23 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/why-was-the-investigation-of-my-problem-closed-even-though-my-problem-has-not-been-solved-yet.md: -------------------------------------------------------------------------------- 1 | # Why was the investigation of my problem closed, even though my problem has not been solved yet? 2 | 3 | There can be different reasons why the investigation of your problem was closed, even though your problem isn’t solved. 4 | 5 | The most common reasons are: 6 | 7 | * **You didn’t provide valid information to support the investigation of your problem** 8 | If we contacted you for more information regarding your problem but didn’t receive valid information, the investigation will have been closed and your payments no longer paused. 9 | * **The store registered a refund** 10 | If the store registered a refund for your purchase the investigation of the problem will have closed automatically and your payments no longer paused. If the refund registered by the store is not correct, please contact the store to find a solution. 11 | * **The store informed us that the statement is correct and/or that the problem is invalid** 12 | If the store confirmed that the statement was correct or that the problem was deemed invalid, the investigation of the issue will have been closed and your payments no longer paused. 13 | * **You have resolved your issue** 14 | If you’ve resolved your issue in the Klarna app, the investigation of your problem will have been closed and your payments no longer paused. Please note that the option to report a problem won’t be available any longer. If your problem is not solved, please contact the store to find a solution. 15 | 16 | **Can I report a problem again?** 17 | It’s not possible to report a problem for the same purchase again, but we have extended your due date to give you more time to solve the issue together with the store. 18 | 19 | If the due date is approaching and you haven’t been able to solve the problem with the store, you can gather all the information regarding your problem and [chat](https://app.klarna.com/support) with our Customer Service. 20 | 21 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/why-was-the-investigation-of-my-return-closed-even-though-my-return-hasnt-been-registered-yet.md: -------------------------------------------------------------------------------- 1 | # Why was the investigation of my return closed, even though my return hasn’t been registered yet? 2 | 3 | There may be various reasons why the investigation into your return was closed, despite the return not being registered yet. 4 | 5 | The most common reasons are: 6 | 7 | * **Incomplete or invalid information was provided** 8 | If we contacted you for further information but didn’t receive the requested information, the investigation will have been closed and your payment plan no longer paused. 9 | * **The store registered your return** 10 | If the store registered your return the investigation would’ve closed automatically. If the return registered by the store is not correct, please contact the store to find a solution. 11 | * **The store informed us that the statement is correct and/or that the return is invalid** 12 | If the store confirmed that the payment was correct or the return was deemed invalid, the investigation will have been closed and your payment plan no longer paused. 13 | * **You have canceled your return** 14 | If you canceled your return in the Klarna app, the investigation would’ve been closed and your payment plan no longer paused. Please note, it won’t be possible to report a return twice for the same purchase. If your issue is not resolved, please contact the store to find a solution. 15 | #### Can I report a return again? 16 | 17 | No, It’s not possible to report a return for the same purchase twice, but we have extended your due date to give you more time to resolve the issue with the store. 18 | 19 | If the due date is approaching and you haven’t been able to come to an agreement with the store, collect all relevant information before [chatting](https://app.klarna.com/support/) with our Customer Service. 20 | 21 | #### Related articles 22 | 23 | * [What should I do if the store doesn’t respond?](https://www.klarna.com/us/customer-service/what-should-i-do-if-the-store-doesnt-respond/) 24 | -------------------------------------------------------------------------------- /cohort_2/week4/data/md/why-was-the-one-time-card-purchase-declined.md: -------------------------------------------------------------------------------- 1 | # Why was the One-time card purchase declined? 2 | 3 | The decline reason is displayed in a pop\-up message, which appears at checkout, if the One\-time card service is denied. 4 | 5 | Follow the instructions in the pop\-up message before attempting to make another purchase. 6 | 7 | We understand it can be frustrating if you experience that our One\-time card service is not available, especially having been approved in the past. 8 | 9 | #### Was the One\-time card service declined in the store’s checkout? 10 | 11 | Visit the store’s website for more information regarding their restrictions. 12 | 13 | While Klarna does not monitor every store searchable in the Klarna app, we generally decline transactions that don’t promote financial wellness or best meet our regulatory, ethical, and risk standards: 14 | 15 | * Bill or rent payments 16 | * Gift card providers 17 | * Governmental agencies 18 | * Medical care 19 | * Online gaming or gambling 20 | * Rideshare services 21 | * Subscription services 22 | * Drugs or alcohol 23 | * NFTs 24 | #### Related articles 25 | 26 | * [How can I get approved to pay with Klarna?](https://www.klarna.com/us/customer-service/how-can-i-get-approved-to-pay-with-klarna/) 27 | * [What happens if my purchase is for a different amount than the One\-time card?](https://www.klarna.com/us/customer-service/what-happens-if-my-purchase-is-for-a-different-amount-than-the-one-time-card/) 28 | * [What if I never use the One\-time card and want to cancel it?](https://www.klarna.com/us/customer-service/what-if-i-never-use-the-one-time-card-and-want-to-cancel-it/) 29 | * [What is a One\-time card and how does it work?](https://www.klarna.com/us/customer-service/what-is-a-one-time-card-and-how-does-it-work/) 30 | 31 | -------------------------------------------------------------------------------- /cohort_2/week5/README.md: -------------------------------------------------------------------------------- 1 | # Week 5: Structured Data and Metadata Enhancement 2 | 3 | ## Notebooks Overview 4 | 5 | ### 1. Generate Dataset.ipynb 6 | This notebook demonstrates how to extract structured metadata from product images and descriptions. 7 | 8 | **What You'll Do:** 9 | - Define a product taxonomy in YAML format (categories, subcategories, attributes) 10 | - Analyze clothing images from the irow/ClothingControlV2 dataset 11 | - Use multimodal LLMs (GPT-4o) to identify clothing items and their attributes 12 | - Extract structured product metadata (material, occasion, size, etc.) 13 | - Implement Pydantic validation to ensure metadata conforms to taxonomy 14 | - Upload the generated dataset to Hugging Face for sharing 15 | 16 | You'll learn how to consistently extract structured metadata that enhances retrieval capabilities. 17 | 18 | ### 2. Metadata Filtering.ipynb 19 | This notebook shows how to combine semantic search with structured metadata filtering. 20 | 21 | **What You'll Do:** 22 | - Ingest product data into LanceDB with embedded metadata 23 | - Implement query understanding to extract filtering criteria 24 | - Compare three retrieval approaches: 25 | 1. Pure semantic search 26 | 2. Pure metadata filtering 27 | 3. Combined approach (vector + metadata) 28 | - Evaluate performance using recall and MRR metrics 29 | - Demonstrate how metadata filtering handles queries semantic search struggles with 30 | - Improve product descriptions for better retrieval 31 | 32 | You'll see how structured metadata dramatically improves retrieval for queries with specific requirements. 33 | 34 | ### 3. Text-2-SQL.ipynb 35 | This notebook explores integrating database queries into RAG systems. 36 | 37 | **What You'll Do:** 38 | - Set up a SQLite database with product inventory and user data 39 | - Generate synthetic user profiles and order history with Faker 40 | - Create stock levels for products 41 | - Implement safe SQL generation using read-only connections 42 | - Build a system that determines when SQL queries are needed vs. retrieval 43 | - Combine retrieval results with database query results 44 | - Test the system with complex queries requiring both approaches 45 | 46 | You'll learn how to safely integrate SQL generation to handle queries that require database access. 47 | 48 | ### 4. PDF-Parser.ipynb 49 | This notebook focuses on extracting structured information from PDF documents with citation capabilities. 50 | 51 | **What You'll Do:** 52 | - Process PDF documents to extract text and layout information 53 | - Implement bounding box tracking for visual citations 54 | - Extract structured data from invoices and documents 55 | - Validate extracted information against predefined schemas 56 | - Create systems that can reference specific regions in source documents 57 | - Handle complex PDF structures with tables and forms 58 | 59 | You'll build a system that can extract structured information while maintaining citation integrity. 60 | 61 | ## Data Files and Assets 62 | - `taxonomy.yml`: Product categories, attributes, and valid values 63 | - `init.sql`: Database schema for product inventory and orders 64 | - `data/invoice.pdf`: Sample invoice for PDF parsing 65 | - `helpers.py`: Utility functions for taxonomy processing and validation 66 | 67 | ## Technical Requirements 68 | - Required libraries: instructor, pydantic, lancedb, openai, sqlite3, faker, poppler, docling 69 | - OpenAI API access (GPT-4o, GPT-4o-mini) 70 | - Basic understanding of SQL and databases 71 | - Familiarity with PDF document structure 72 | 73 | ## Why These Notebooks Matter 74 | These notebooks move beyond simple semantic search to address complex real-world queries: 75 | 76 | 1. Queries with specific attribute requirements ("red sneakers under $100") 77 | 2. Questions requiring database access ("How many orders did user X place last month?") 78 | 3. Document information extraction with citation needs 79 | 80 | By combining vector search with structured filtering, SQL access, and document parsing, you'll build RAG systems capable of handling sophisticated user needs that pure semantic search can't address alone. -------------------------------------------------------------------------------- /cohort_2/week5/data/documents/maybank-world-mastercard-tnc.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/567-labs/systematically-improving-rag/32241e32e571bb7e34c257c143cc24f117f73f79/cohort_2/week5/data/documents/maybank-world-mastercard-tnc.pdf -------------------------------------------------------------------------------- /cohort_2/week5/data/documents/mb-regional-golf-program-tnc.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/567-labs/systematically-improving-rag/32241e32e571bb7e34c257c143cc24f117f73f79/cohort_2/week5/data/documents/mb-regional-golf-program-tnc.pdf -------------------------------------------------------------------------------- /cohort_2/week5/data/documents/personal-credit-card-tnc.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/567-labs/systematically-improving-rag/32241e32e571bb7e34c257c143cc24f117f73f79/cohort_2/week5/data/documents/personal-credit-card-tnc.pdf -------------------------------------------------------------------------------- /cohort_2/week5/data/invoice.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/567-labs/systematically-improving-rag/32241e32e571bb7e34c257c143cc24f117f73f79/cohort_2/week5/data/invoice.pdf -------------------------------------------------------------------------------- /cohort_2/week5/helpers.py: -------------------------------------------------------------------------------- 1 | import itertools 2 | from lancedb.rerankers import Reranker 3 | from lancedb.table import Table 4 | 5 | 6 | def calculate_mrr(predictions: list[str], gt: list[str]): 7 | mrr = 0 8 | for label in gt: 9 | if label in predictions: 10 | mrr = max(mrr, 1 / (predictions.index(label) + 1)) 11 | return mrr 12 | 13 | 14 | def get_recall(predictions: list[str], gt: list[str]): 15 | return len([label for label in gt if label in predictions]) / len(gt) 16 | 17 | 18 | def get_metrics_at_k( 19 | metrics: list[str], 20 | sizes: list[int], 21 | ): 22 | metric_to_score_fn = { 23 | "mrr": calculate_mrr, 24 | "recall": get_recall, 25 | } 26 | 27 | for metric in metrics: 28 | if metric not in metric_to_score_fn: 29 | raise ValueError(f"Metric {metric} not supported") 30 | 31 | eval_metrics = [(metric, metric_to_score_fn[metric]) for metric in metrics] 32 | 33 | return { 34 | f"{metric_name}@{size}": lambda predictions, gt, m=metric_fn, s=size: ( 35 | lambda p, g: m(p[:s], g) 36 | )(predictions, gt) 37 | for (metric_name, metric_fn), size in itertools.product(eval_metrics, sizes) 38 | } 39 | 40 | 41 | def task(user_query: str, table: Table, reranker: Reranker, max_k: int): 42 | query = table.search(user_query, query_type="vector").limit(max_k) 43 | 44 | if reranker: 45 | query = query.rerank(reranker) 46 | 47 | return [item["id"] for item in query.to_list()] 48 | 49 | 50 | def process_taxonomy_file(yaml_path: str) -> dict: 51 | """ 52 | Process a taxonomy YAML file and extract taxonomy map, occasions, materials, and other relevant data. 53 | 54 | Args: 55 | yaml_path (str): Path to the taxonomy YAML file 56 | 57 | Returns: 58 | dict: Dictionary containing: 59 | - taxonomy_map: Nested dictionary mapping categories -> subcategories -> attributes 60 | - occasions: List of valid occasions 61 | - materials: List of valid materials 62 | - taxonomy: Raw taxonomy data 63 | """ 64 | import yaml 65 | 66 | # Load the YAML file 67 | with open(yaml_path, "r") as f: 68 | taxonomy = yaml.safe_load(f) 69 | 70 | # Create taxonomy map 71 | taxonomy_map = {} 72 | for category in taxonomy["categories"]: 73 | cat_name = category["name"] 74 | taxonomy_map[cat_name] = {} 75 | 76 | for subcategory in category["subcategories"]: 77 | subcat_name = subcategory["name"] 78 | taxonomy_map[cat_name][subcat_name] = {} 79 | 80 | # Add product types 81 | taxonomy_map[cat_name][subcat_name]["product_type"] = subcategory.get( 82 | "types", [] 83 | ) 84 | 85 | # Add attributes and their values 86 | taxonomy_map[cat_name][subcat_name]["attributes"] = {} 87 | for attr in subcategory.get("attributes", []): 88 | for attr_name, values in attr.items(): 89 | taxonomy_map[cat_name][subcat_name]["attributes"][attr_name] = ( 90 | values 91 | ) 92 | 93 | # Extract occasions and materials from common attributes 94 | occasions = [] 95 | materials = [] 96 | common_attributes = {} 97 | for attr in taxonomy["common_attributes"]: 98 | if "Occasion" in attr: 99 | occasions = attr["Occasion"] 100 | elif "Material" in attr: 101 | materials = attr["Material"] 102 | 103 | attr_name = list(attr.keys())[0] 104 | attr_values = attr[attr_name] 105 | common_attributes[attr_name] = attr_values 106 | 107 | return { 108 | "taxonomy_map": taxonomy_map, 109 | "occasions": occasions, 110 | "materials": materials, 111 | "common_attributes": common_attributes, 112 | "taxonomy": taxonomy["categories"], # Raw taxonomy data for reference 113 | } 114 | -------------------------------------------------------------------------------- /cohort_2/week5/init.sql: -------------------------------------------------------------------------------- 1 | -- Create stock table 2 | CREATE TABLE IF NOT EXISTS stock ( 3 | product_id INTEGER, 4 | size TEXT, 5 | color TEXT, 6 | quantity INTEGER 7 | ); 8 | 9 | -- Create orders table 10 | CREATE TABLE IF NOT EXISTS orders ( 11 | order_id INTEGER PRIMARY KEY AUTOINCREMENT, 12 | user_email TEXT, 13 | date DATETIME, 14 | total_amount DECIMAL(10,2), 15 | FOREIGN KEY (user_email) REFERENCES users(email) 16 | ); 17 | 18 | -- Create sales table (now with order_id and item details) 19 | CREATE TABLE IF NOT EXISTS sales ( 20 | sale_id INTEGER PRIMARY KEY AUTOINCREMENT, 21 | order_id INTEGER, 22 | product_id INTEGER, 23 | size TEXT, 24 | color TEXT, 25 | quantity INTEGER, 26 | price_per_unit DECIMAL(10,2), 27 | FOREIGN KEY (order_id) REFERENCES orders(order_id) 28 | ); 29 | 30 | -- Create users table 31 | CREATE TABLE IF NOT EXISTS users ( 32 | email TEXT PRIMARY KEY, 33 | name TEXT, 34 | gender TEXT, 35 | date_of_birth DATE, 36 | created_at DATETIME DEFAULT CURRENT_TIMESTAMP 37 | ); 38 | -------------------------------------------------------------------------------- /cohort_2/week6/README.md: -------------------------------------------------------------------------------- 1 | # Week 6: Tool Selection and Orchestration 2 | 3 | ## Notebooks Overview 4 | 5 | ### 1. Evaluate Tools.ipynb 6 | This notebook introduces evaluation metrics for measuring tool selection performance. 7 | 8 | **What You'll Do:** 9 | - Define precision and recall metrics specifically for tool selection 10 | - Create simulations of tool calls with various examples 11 | - Implement Pydantic models for structured tool definitions 12 | - Use the AsyncOpenAI client with instructor for tool generation 13 | - Compare sequential and parallel tool calling approaches 14 | - Evaluate when each approach performs better 15 | 16 | You'll establish objective ways to measure tool selection quality and learn when parallel execution is appropriate. 17 | 18 | ### 2. Generate Dataset.ipynb 19 | This notebook creates a synthetic dataset targeting common tool selection failure modes. 20 | 21 | **What You'll Do:** 22 | - Load available commands from `raw_commands.json` (70+ commands) 23 | - Generate diverse test cases that mimic real user behavior 24 | - Implement complex query generation with chain-of-thought reasoning 25 | - Create contrastive examples to generate more realistic queries 26 | - Analyze per-tool recall to identify specific weaknesses 27 | - Build a benchmarking dataset for systematic improvement 28 | 29 | You'll develop a comprehensive test suite that challenges tool selection capabilities. 30 | 31 | ### 3. Improving Performance.ipynb 32 | This notebook explores techniques to systematically improve tool selection accuracy. 33 | 34 | **What You'll Do:** 35 | - Implement system prompts with detailed user behavior context 36 | - Create few-shot examples demonstrating correct tool combinations 37 | - Measure performance improvements across different prompting strategies 38 | - Compare baseline, system prompt, and few-shot approaches 39 | - Analyze which types of queries improve the most with each approach 40 | - Achieve dramatic improvements in precision and recall 41 | 42 | You'll see how targeted improvements can substantially boost tool selection performance: 43 | - Baseline: 45% precision, 40% recall 44 | - With system prompt: 64% precision (+42%), 54% recall (+35%) 45 | - With system prompt + few-shot examples: 79% precision (+76%), 84% recall (+110%) 46 | 47 | ## Data Files and Assets 48 | - `raw_commands.json`: List of 70+ available commands with descriptions 49 | - `queries.jsonl`: Generated test queries with expected tool calls 50 | - `helpers.py`: Utility functions for evaluation and metrics calculation 51 | 52 | ## Technical Requirements 53 | - Required libraries: instructor, pydantic, openai.AsyncOpenAI, braintrust, pandas, asyncio 54 | - OpenAI API access for tool calling capabilities 55 | - Basic understanding of async programming concepts 56 | 57 | ## Why These Notebooks Matter 58 | Modern RAG systems increasingly rely on tool orchestration beyond simple retrieval. These notebooks apply the same systematic improvement methodology to tool selection: 59 | 60 | 1. First, establish objective metrics to measure performance 61 | 2. Then, generate targeted test cases to identify weaknesses 62 | 3. Finally, implement and measure specific improvements 63 | 64 | This approach demonstrates how simple prompting changes can dramatically improve tool selection, enabling RAG systems to coordinate multiple specialized capabilities rather than relying on retrieval alone. --------------------------------------------------------------------------------