├── .all-contributorsrc ├── .github ├── pull_request_template.md └── workflows │ └── publish.yml ├── .gitignore ├── CONTRIBUTING.md ├── README.md ├── book.toml └── src ├── README.md ├── SUMMARY.md ├── contribute-to-tidb ├── cherrypick-a-pr.md ├── code-style-and-quality-guide.md ├── committer-guide.md ├── community-guideline.md ├── contribute-code.md ├── introduction.md ├── issue-triage.md ├── make-a-proposal.md ├── miscellaneous-topics.md ├── release-notes-style-guide.md ├── report-an-issue.md ├── review-a-pr.md └── write-document.md ├── extending-tidb ├── add-a-function.md └── introduction.md ├── get-started ├── build-tidb-from-source.md ├── commit-code-and-submit-a-pull-request.md ├── debug-and-profile.md ├── install-golang.md ├── introduction.md ├── run-and-debug-integration-tests.md ├── setup-an-ide.md └── write-and-run-unit-tests.md ├── img ├── cbo-example-logicalPlan.png ├── cbo-explain-getBestTask.png ├── ddl-owner-detail-flow-chart.png ├── ddl-structure-flow-chart.png ├── dml-contention.png ├── dql-frame-diagram.png ├── dql-volcano.png ├── gc-leader-election.png ├── goland-run-tests.png ├── open-tidb-in-goland.png ├── plan-cache-execute.png ├── plan-cache-parameter.png ├── plan-cache-rebuilding.png ├── playground-attach-config.png ├── playground-attach-debug.png ├── raftStore.png ├── run-configs.png ├── sql-layer-architecture.png ├── stats-cmsketch.png ├── stats-histogram.png ├── tidb-architecture.png ├── transaction-architecture.png ├── transaction_scheduler.png ├── unistore-config.png ├── unistore-output.png ├── unit-test-config.png ├── unit-test-output.png ├── vscode-debug-binary.png ├── vscode-debug-config.png ├── vscode-go-to-edit.png ├── vscode-run-tests.png └── vscode_tide.png ├── project-management ├── introduction.md ├── release-train-model.md └── tidb-versioning.md ├── system-tables ├── introduction.md └── slow_query.md └── understand-tidb ├── 1pc.md ├── async-commit.md ├── cbo.md ├── ddl.md ├── dml.md ├── dql.md ├── execution.md ├── implementation-of-typical-operators.md ├── implementation-of-vectorized-execution.md ├── introduction.md ├── lock-resolver.md ├── memory-management-mechanism.md ├── mvcc-garbage-collection.md ├── optimistic-transaction.md ├── parallel-execution-framework.md ├── parser.md ├── pessimistic-transaction.md ├── plan-cache.md ├── planner.md ├── plugin.md ├── privilege.md ├── rbo.md ├── session.md ├── sql-plan-management.md ├── system-tables ├── information_schema │ ├── introduction.md │ └── slow_query.md └── introduction.md ├── table-statistics.md ├── the-lifecycle-of-a-statement.md ├── transaction-on-tikv.md └── transaction.md /.github/pull_request_template.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | ### What issue does this PR solve? 6 | 7 | 8 | 9 | - close #xxxx 10 | - to #xxxx 11 | 12 | ### What is changed: 13 | -------------------------------------------------------------------------------- /.github/workflows/publish.yml: -------------------------------------------------------------------------------- 1 | name: Publish 2 | 3 | on: 4 | push: 5 | branches: [master] 6 | 7 | jobs: 8 | deploy: 9 | runs-on: ubuntu-latest 10 | steps: 11 | - uses: actions/checkout@v3 12 | 13 | - name: Setup mdBook 14 | uses: peaceiris/actions-mdbook@v1 15 | with: 16 | mdbook-version: 'latest' 17 | 18 | - run: mdbook build 19 | 20 | - name: Deploy 21 | uses: peaceiris/actions-gh-pages@v3 22 | if: github.ref == 'refs/heads/master' 23 | with: 24 | github_token: ${{ secrets.GITHUB_TOKEN }} 25 | publish_dir: ./book 26 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | book 2 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing to TiDB Development Guide 2 | 3 | Welcome to contribute to the TiDB Development Guide! 4 | 5 | ## How to contribute 6 | 7 | This guide proposes to provide a one-stop wiki to all the development-related 8 | issues. Any changes that can enhance the objective are welcome. Otherwise, we 9 | may not accept the pull requests. 10 | 11 | 1. Pointing out the mistakes is important. Don't hesitate to file an issue to 12 | report mistakes, or make suggestions, if you don't have enough time to 13 | refine the content. 14 | 15 | 2. Improve the guide content, for example: 16 | - fix typos and adjust some statements to make them clearer 17 | - modify the contents if it's outdated 18 | - add new pages if something is missing 19 | - adjust the contents if the article structure is unreasonable 20 | 21 | 3. Reviewing the pull request 22 | 23 | Once the pull request receives 2 LGTM from the maintainers, it can be merged. 24 | 25 | ## How to add yourself to the contributors list 26 | 27 | At present, we only count content contribution and review contribution, once 28 | you have successfully contributed to this guide, you can nominate yourself as a 29 | contributor by the following two commands in the related pull request: 30 | 31 | 1. for content contributions: 32 | ```sh 33 | @all-contributors please add @your_github_id for content 34 | ``` 35 | 36 | 3. for pull request review contributions: 37 | ```sh 38 | @all-contributors please add @your_github_id for review 39 | ``` 40 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # TiDB Development Guide 2 | 3 | This repository contains the source of TiDB Development Guide. 4 | 5 | * The target audience of this guide is TiDB contributors, both new and experienced. 6 | * The objective of this guide is to help contributors become an expert of TiDB, who is familiar with its design and implementation and thus is able to use it fluently in the real world as well as develop TiDB itself deeply. 7 | 8 | ## Requirements 9 | 10 | Building the book requires [mdBook](https://github.com/rust-lang-nursery/mdBook). To get it: 11 | 12 | ```bash 13 | $ cargo install mdbook 14 | ``` 15 | 16 | ## Building 17 | 18 | To build the book, type: 19 | 20 | ```bash 21 | $ mdbook build 22 | ``` 23 | 24 | The output will be in the `book` subdirectory. To check it out, open it in 25 | your web browser. 26 | 27 | _Firefox:_ 28 | ```bash 29 | $ firefox book/index.html # Linux 30 | $ open -a "Firefox" book/index.html # OS X 31 | $ Start-Process "firefox.exe" .\book\index.html # Windows (PowerShell) 32 | $ start firefox.exe .\book\index.html # Windows (Cmd) 33 | ``` 34 | 35 | _Chrome:_ 36 | ```bash 37 | $ google-chrome book/index.html # Linux 38 | $ open -a "Google Chrome" book/index.html # OS X 39 | $ Start-Process "chrome.exe" .\book\index.html # Windows (PowerShell) 40 | $ start chrome.exe .\book\index.html # Windows (Cmd) 41 | ``` 42 | 43 | ## Contribute to this guide 44 | 45 | See [CONTRIBUTING](CONTRIBUTING.md) for details. 46 | -------------------------------------------------------------------------------- /book.toml: -------------------------------------------------------------------------------- 1 | [book] 2 | title = "TiDB Development Guide" 3 | 4 | [output.html] 5 | git-repository-url = "https://github.com/pingcap/tidb-dev-guide" 6 | -------------------------------------------------------------------------------- /src/SUMMARY.md: -------------------------------------------------------------------------------- 1 | # Summary 2 | 3 | [TiDB Development Guide](README.md) 4 | 5 | * [Get Started](get-started/introduction.md) 6 | * [Install Golang](get-started/install-golang.md) 7 | * [Get the code, build and run](get-started/build-tidb-from-source.md) 8 | * [Setup an IDE](get-started/setup-an-ide.md) 9 | * [Write and run unit tests](get-started/write-and-run-unit-tests.md) 10 | * [Debug and profile](get-started/debug-and-profile.md) 11 | * [Commit code and submit a pull request](get-started/commit-code-and-submit-a-pull-request.md) 12 | 13 | * [Contribute to TiDB](contribute-to-tidb/introduction.md) 14 | * [Community Guideline](contribute-to-tidb/community-guideline.md) 15 | * [Report an Issue](contribute-to-tidb/report-an-issue.md) 16 | * [Issue Triage](contribute-to-tidb/issue-triage.md) 17 | * [Contribute Code](contribute-to-tidb/contribute-code.md) 18 | * [Cherry-pick a Pull Request](contribute-to-tidb/cherrypick-a-pr.md) 19 | * [Review a Pull Request](contribute-to-tidb/review-a-pr.md) 20 | * [Make a Proposal](contribute-to-tidb/make-a-proposal.md) 21 | * [Code Style and Quality Guide](contribute-to-tidb/code-style-and-quality-guide.md) 22 | * [Write Document](contribute-to-tidb/write-document.md) 23 | * [Release Notes Language Style Guide](contribute-to-tidb/release-notes-style-guide.md) 24 | * [Committer Guide](contribute-to-tidb/committer-guide.md) 25 | * [Miscellaneous Topics](contribute-to-tidb/miscellaneous-topics.md) 26 | 27 | * [Understand TiDB](understand-tidb/introduction.md) 28 | * [The Lifecycle of a Statement](understand-tidb/the-lifecycle-of-a-statement.md) 29 | * [DDL](understand-tidb/ddl.md) 30 | * [DML](understand-tidb/dml.md) 31 | * [DQL](understand-tidb/dql.md) 32 | * [Parser](understand-tidb/parser.md) 33 | * [Planner](understand-tidb/planner.md) 34 | * [Table Statistics](understand-tidb/table-statistics.md) 35 | * [Rule-based Optimization](understand-tidb/rbo.md) 36 | * [Cost-based Optimization](understand-tidb/cbo.md) 37 | * [Plan Cache](understand-tidb/plan-cache.md) 38 | * [SQL Plan Management](understand-tidb/sql-plan-management.md) 39 | * [Execution](understand-tidb/execution.md) 40 | * [Parallel Execution Framework](understand-tidb/parallel-execution-framework.md) 41 | * [Implementation of Vectorized Execution](understand-tidb/implementation-of-vectorized-execution.md) 42 | * [Memory Management Mechanism](understand-tidb/memory-management-mechanism.md) 43 | * [Implementation of Typical Operators](understand-tidb/implementation-of-typical-operators.md) 44 | * [Transaction](understand-tidb/transaction.md) 45 | * [Transaction on TiKV](understand-tidb/transaction-on-tikv.md) 46 | * [Optimistic Transaction](understand-tidb/optimistic-transaction.md) 47 | * [Lock Resolver](understand-tidb/lock-resolver.md) 48 | * [Pessimistic Transaction](understand-tidb/pessimistic-transaction.md) 49 | * [Async Commit](understand-tidb/async-commit.md) 50 | * [1PC](understand-tidb/1pc.md) 51 | * [MVCC garbage collection](understand-tidb/mvcc-garbage-collection.md) 52 | * [Session](understand-tidb/session.md) 53 | * [Privilege](understand-tidb/privilege.md) 54 | * [Plugin](understand-tidb/plugin.md) 55 | * [System tables](understand-tidb/system-tables/introduction.md) 56 | * [information_schema](understand-tidb/system-tables/information_schema/introduction.md) 57 | * [slow_query](understand-tidb/system-tables/information_schema/slow_query.md) 58 | 59 | * [Project Management](project-management/introduction.md) 60 | * [Releases Train Model](project-management/release-train-model.md) 61 | * [TiDB Versioning](project-management/tidb-versioning.md) 62 | 63 | * [Extending TiDB](extending-tidb/introduction.md) 64 | * [Add a function](extending-tidb/add-a-function.md) -------------------------------------------------------------------------------- /src/contribute-to-tidb/cherrypick-a-pr.md: -------------------------------------------------------------------------------- 1 | # Cherry-pick a Pull Request 2 | 3 | TiDB uses [release train model](../project-management/release-train-model.md) and has multiple releases. Each release matches one git branch. For `type/bug` issues with `severity/critical` and `severity/major`, it is anticipated to be fixed on any [currently maintained releases](https://pingcap.github.io/tidb-dev-guide/project-management/release-train-model.html#current-maintained-releases) if affected. Contributors and reviewers are responsible to settle the affected versions once the bug is identified as `severity/critical` or `severity/major`. Cherry-pick pull requests shall be created to port the fix to affected branches after the original pull request merged. While creating cherry-pick pull requests, bots in TiDB community could help lighten your workload. 4 | 5 | ## What kind of pull requests need to cherry-pick? 6 | 7 | Because there are more and more releases of TiDB and limits of developer time, we are not going to cherry-pick every pull request. Currently, only problems with `severity/critical` and `severity/major` are candidates for cherry-pick. There problems shall be solved on all affected [maintained releases](https://pingcap.github.io/tidb-dev-guide/project-management/release-train-model.html#current-maintained-releases). Check [Issue Triage chapter](issue-triage.md) for severity identification. 8 | 9 | ## Create cherry-pick pull requests automatically 10 | 11 | Typically, TiDB repos use ti-chi-bot to help contributors create cherry-pick pull requests automatically. 12 | 13 | `ti-chi-bot` creates corresponding cherry-pick pull requests according to the `needs-cherry-pick-` on the original pull request once it's merged. If there is any failure or omission, contributors could run `/cherry-pick ` to trigger cherry-pick for a specific release. 14 | 15 | ## Create cherry-pick pull requests manually 16 | 17 | Contributors could also create cherry-pick pull requests manually if they want. [git cherry-pick](https://git-scm.com/docs/git-cherry-pick) is a good command for this. The requirements in [Contribute Code](contribute-code.md) also apply here. 18 | 19 | ## Pass triage complete check 20 | 21 | For pull requests, `check-issue-triage-complete` checker will first check whether the [corresponding issue](https://pingcap.github.io/tidb-dev-guide/contribute-to-tidb/contribute-code.html#referring-to-an-issue) has any `type/xx` label, if not, the checker fails. Then for issues with `type/bug` label, there must also exist a `severity/xx` label, otherwise, the checker fails. For `type/bug` issue with `severity/critical` or `severity/major` label, the checker checks if there is any `may-affects-x.y` label, which means the issue has not been diagnosed on all needed versions. If there is, the pull request is blocked and not able to be merged. So in order to merge a bugfix pull request into the target branch, every other effective version needs to first be diagnosed. TiDB maintainer will add these labels. 22 | 23 | ti-chi-bot will automatically trigger the checker to run on the associated PR by listening to the labeled/unlabeled event of `may-affects-x.y` labels on bug issues, contributors also could comment `/check-issue-triage-complete` or `/run-check-issue-triage-complete` like other checkers to rerun the checker manually and update the status. Once `check-issue-triage-complete` checker passes, ti-chi-bot will add `needs-cherry-pick-`/`needs-cherry-pick-` labels to pull requests according to the `affects-x.y` labels on the corresponding issues. 24 | 25 | In addition, if the checker fails, the robot will add the `do-not-merge/needs-triage-completed` label to the pull request at the same time, which will be used by other plugins like [tars](https://book.prow.tidb.io/#/en/plugins/tars). 26 | 27 | 28 | ## Review cherry-pick pull requests 29 | 30 | Cherry-pick pull requests obey the [same review rules](review-a-pr.md) as other pull requests. Besides the merge requirements as normal pull requests, cherry-pick pull requests are added `do-not-merge/cherry-pick-not-approved` label initially. To get it merged, it needs an additional `cherry-pick-approved` label from team *qa-release-merge*. 31 | 32 | ## Troubleshoot cherry-pick 33 | 34 | * If there is any error in the cherry-pick process, for example, the bot fails to create some cherry-pick pull requests. You could ask reviewers/committers/maintainers for help. 35 | * If there are conflicts in the cherry-pick pull requests. You must [resolve the conflicts](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/addressing-merge-conflicts/about-merge-conflicts) to get pull requests merged. Some ways can solve it: 36 | - Request privileges to the forked repo by sending `/cherry-pick-invite` comment in the cherry-pick pull request if you are a member of the orgnization. When you accepted the invitaion, you could directly push to the pull request branch. 37 | - Ask committers/maintainers to do that for you if you are not a member of the orgnization. 38 | - Manually create a new cherry-pick pull request for the branch. 39 | -------------------------------------------------------------------------------- /src/contribute-to-tidb/code-style-and-quality-guide.md: -------------------------------------------------------------------------------- 1 | # TiDB Code Style and Quality Guide 2 | 3 | This is an attempt to capture the code and quality standard that we want to maintain. 4 | 5 | ## The newtype pattern improves code quality 6 | 7 | We can create a new type using the `type` keyword. 8 | 9 | The newtype pattern is perhaps most often used in Golang to get around type restrictions rather than to try to create new ones. It is used to create different interface implementations for a type or to extend a builtin type or a type from an existing package with new methods. 10 | 11 | However, it is generally useful to improve code clarity by marking that data has gone through either a validation or a transformation. Using a different type can reduce error handling and prevent improper usage. 12 | 13 | ```go 14 | package main 15 | 16 | import ( 17 | "fmt" 18 | "strings" 19 | ) 20 | 21 | type Email string 22 | 23 | func newEmail(email string) (Email, error) { 24 | if !strings.Contains(email, "@") { 25 | return Email(""), fmt.Errorf("Expected @ in the email") 26 | } 27 | return Email(email), nil 28 | } 29 | 30 | func (email Email) Domain() string { 31 | return strings.Split(string(email), "@")[1] 32 | } 33 | 34 | func main() { 35 | ping, err := newEmail("go@pingcap.com") 36 | if err != nil { panic(err) } 37 | fmt.Println(ping.Domain()) 38 | } 39 | ``` 40 | 41 | ## When to use value or pointer receiver 42 | 43 | Because pointer receivers need to be used some of the time, Go programmers often use them all of the time. 44 | This is a typical outline of Go code: 45 | 46 | ``` go 47 | type struct S {} 48 | func NewStruct() *S 49 | func (s *S) structMethod() 50 | ``` 51 | 52 | Using pointers for the entire method set means we have to read the source code of every function to determine if it mutates the struct. Mutations are a source of error. This is particularly true in concurrent programs. We can contrast this with values: these are always concurrent safe. 53 | 54 | For code clarity and bug reduction a best practice is to default to using values and value receivers. 55 | However, pointer receivers are often required to satisfy an interface or for performance reasons, and this need overrides any default practice. 56 | 57 | However, performance can favor either approach. One might assume that pointers would always perform better because it avoids copying. However, the performance is roughly the same for small structs in micro benchmark. This is because the copying is cheap, inlining can often avoid copying anyways, and pointer indirection has its own small cost. In a larger program with a goal of predictable low latency the value approach can be more favorable because it avoids [heap allocation and any additional GC overhead](https://segment.com/blog/allocation-efficiency-in-high-performance-go-services/). 58 | 59 | As a rule of thumb is that when a struct has 10 or more words we should use pointer receivers. However, to actually know which is best for performance depends on how the struct is used in the program and must ultimately be determined by profiling. For example these are some factors that affect things: 60 | 61 | * method size: small inlineable methods favor value receivers. 62 | * Is the struct called repeatedly in a for loop? This favors pointer receivers. 63 | * What is the GC behavior of the rest of the program? GC pressure may favor value receivers. 64 | 65 | ## Parallel For-Loop 66 | 67 | There are two types of for loop on range: "with index" and "without index". Let's see an example for range with index. 68 | 69 | ```go 70 | func TestRangeWithIndex(t *testing.T) { 71 | rows := []struct{ index int }{{index: 0}, {index: 1}, {index: 2}} 72 | for _, row := range rows { 73 | row.index += 10 74 | } 75 | for i, row := range rows { 76 | require.Equal(t, i+10, row.index) 77 | } 78 | } 79 | ``` 80 | 81 | the output is: 82 | 83 | ``` 84 | Error Trace: version_test.go:39 85 | Error: Not equal: 86 | expected: 10 87 | actual : 0 88 | Test: TestShowRangeWithIndex 89 | ``` 90 | 91 | Test fails because when range with index, the loop iterator variable is the same instance of the variable with a clone of iteration target value. 92 | 93 | ### The same instance of the variable 94 | 95 | Since the the loop iterator variable is the same instance of the variable, it may result in tricky error with parallel for-loop. 96 | 97 | ```go 98 | done := make(chan bool) 99 | values := []string{"a", "b", "c"} 100 | for _, v := range values { 101 | go func() { 102 | fmt.Println(v) 103 | done <- true 104 | }() 105 | } 106 | for _ = range values { 107 | <-done 108 | } 109 | ``` 110 | 111 | You might expect to see `a`, `b`, `c` as the output, but you'll probably see instead is `c`, `c`, `c`. 112 | 113 | This is because each iteration of the loop uses the same instance of the variable `v`, so each closure shares that single variable. 114 | 115 | This is the same reason which result wrong test when use `t.Parallel()` with range, which is covered in [Parallel section of Write and run unit tests](../get-started/write-and-run-unit-tests.md#parallel) 116 | 117 | ### A clone of iteration target value 118 | 119 | Since the loop iterator variable is a clone of iteration target value, it may result in logic error. It can also lead to performance issue compared with none-index range loop or bare for loop. 120 | 121 | ```go 122 | type Item struct { 123 | id int 124 | value [1024]byte 125 | } 126 | 127 | func BenchmarkRangeIndexStruct(b *testing.B) { 128 | var items [1024]Item 129 | for i := 0; i < b.N; i++ { 130 | var tmp int 131 | for k := range items { 132 | tmp = items[k].id 133 | } 134 | _ = tmp 135 | } 136 | } 137 | 138 | func BenchmarkRangeStruct(b *testing.B) { 139 | var items [1024]Item 140 | for i := 0; i < b.N; i++ { 141 | var tmp int 142 | for _, item := range items { 143 | tmp = item.id 144 | } 145 | _ = tmp 146 | } 147 | } 148 | ``` 149 | 150 | ``` 151 | BenchmarkRangeIndexStruct-12 4875518 246.0 ns/op 152 | BenchmarkRangeStruct-12 16171 77523 ns/op 153 | ``` 154 | 155 | You can see range with index is much slower than range without index, since range with index use cloned value so have big performance decrease if cloned value use lots of memory. 156 | -------------------------------------------------------------------------------- /src/contribute-to-tidb/committer-guide.md: -------------------------------------------------------------------------------- 1 | # Committer Guide 2 | 3 | This is an evolving document to provide some helpful tips for committers. Most of them are lessons learned during development. We welcome every committer to contribute to this document. See the [TiDB Community Guideline](community-guideline.md) for an overview of the committership and the general development process. 4 | 5 | ## Community First 6 | 7 | The collective effort of the community moves the project forward and makes the project awesome for everyone. When we make a decision, it is always helpful to keep the community in mind. Here are some example questions that we can ask: 8 | 9 | * How can I encourage new contributors to get more involved in the project? 10 | * Can I help to save my fellow committers' time? 11 | * Have I enabled the rest of the community to participate the design proposals? 12 | 13 | ## Public Archive Principle 14 | 15 | While private channels such as face to face discussion are useful for development, they also create barriers for the broader community's participation. An open way of development suggests all decisions to be made in public channels, which are archived and accessible to everyone. As a result, any contributor can keep up with the development by watching the archives and join the development anytime. 16 | 17 | While this principle applies to every contributor, it is especially important for committers. Here are some example applications of this principle: 18 | 19 | * When getting a project-related question from a personal channel, encourage the person to open a public thread in the [TiDB Internals forum](https://internals.tidb.io/), so others in the community can benefit from the answer. 20 | * After an in-person discussion, send a summary to public channels (as an RFC or a discuss topic). 21 | 22 | ## Shepherd a Pull Request 23 | 24 | Here are some tips to shepherd a pull request. You can also take a look at [Review a Pull Request](review-a-pr.md). 25 | 26 | * Assign the PR to yourself, so that other committers know that the PR has already been tended to. 27 | * Make use of the status label to indicate the current status. 28 | * Check if a design document needs to be present. 29 | * If the contributor has not requested a reviewer, kindly ask the contributor to do so. If the PR comes from a new contributor, help the contributor to request reviewers and ask the contributor to do so next time. 30 | * Moderate the reviews, ask reviewers to approve explicitly. 31 | * Mark the PR as accepted and acknowledge the contributor/reviewers. 32 | * Merge the PR :) 33 | 34 | ## Time Management 35 | 36 | There are many things that a committer can do, such as moderating discussions, pull request reviews and code contributions. 37 | 38 | Working on an open source project can be rewarding, but also be a bit overwhelming sometimes. A little bit of time management might be helpful to alleviate the problem. For example, some committers have a "community day" in a week when they actively manage outstanding PRs, but watch the community less frequently in the rest of the time. 39 | 40 | Remember that your merit will never go away, so please take your time and pace when contributing to the project:) 41 | -------------------------------------------------------------------------------- /src/contribute-to-tidb/community-guideline.md: -------------------------------------------------------------------------------- 1 | # Community Guideline 2 | 3 | TiDB community aims to provide harassment-free, welcome and friendly experience for everyone. The first and most important thing for any participant in the community is be friendly and respectful to others. Improper behaviors will be warned and punished. 4 | 5 | We appreciate any contribution in any form to TiDB community. Thanks so much for your interest and enthusiasm on TiDB! 6 | 7 | ## Code of Conduct 8 | 9 | TiDB community refuses any kind of harmful behavior to the community or community members. Everyone should read our [Code of Conduct](https://github.com/pingcap/community/blob/master/CODE_OF_CONDUCT.md) and keep proper behavior while participating in the community. 10 | 11 | ## Governance 12 | 13 | TiDB development governs by two kind of groups: 14 | 15 | * [TOC](https://github.com/pingcap/community/tree/master/toc): TOC serves as the main bridge and channel for coordinating and information sharing across companies and organizations. It is the coordination center for solving problems in terms of resource mobilization, technical research and development direction in the current community and cooperative projects. 16 | * [teams](https://github.com/pingcap/community/tree/master/teams): Teams are persistent open groups that focus on a part of the TiDB projects. A team has its reviewer, committer and maintainer, and owns one or more repositories. Team level decision making comes from its maintainers. 17 | 18 | A typical promoted path for a TiDB developer is from user to reviewer, then committer and maintainer, finally maybe a TOC member. But gaining more roles doesn't mean you have any privilege over other community members or even any right to control them. Everyone in TiDB community are equal and share the responsibility to collaborate constructively with other contributors, building a friendly community. The roles are a natural reward for your substantial contribution in TiDB development and provide you more rights in the development workflow to enhance your efficiency. Meanwhile, they request some additional responsibilities from you: 19 | 20 | * Now that you are a member of team reviewers/committers/maintainers, you are representing the project and your fellow team members whenever you discuss TiDB with anyone. So please be a good person to defend the reputation of the team. 21 | * Committers/maintainers have the right to approve pull requests, so bear the additional responsibility of handling the consequences of accepting a change into the codebase or documentation. That includes reverting or fixing it if it causes problems as well as helping out the release manager in resolving any problems found during the pre-release testing cycle. While all contributors are free to help out with this part of the process, and it is most welcome when they do, the actual responsibility rests with the committers/maintainers that approved the change. 22 | * Reviewers/committers/maintainers also bear the primary responsibility for guiding contributors the right working procedure, like deciding when changes proposed on the issue tracker should be escalated to [internal.tidb.io](https://internals.tidb.io) for wider discussion, as well as suggesting the use of the [TiDB Design Documents](https://github.com/pingcap/tidb/tree/master/docs/design) process to manage the design and justification of complex changes, or changes with a potentially significant impact on end users. 23 | 24 | There should be no blockers to contribute with no roles. It's totally fine for you to reject the promotion offer if you don't want to take the additional responsibilities or for any other reason. Besides, except for code or documentation contribution, any kind of contribution for the community is highly appreciated. Let's grow TiDB ecosystem through our contributions of this community. 25 | -------------------------------------------------------------------------------- /src/contribute-to-tidb/contribute-code.md: -------------------------------------------------------------------------------- 1 | # Contribute Code 2 | 3 | TiDB is maintained, improved, and extended by code contributions. We welcome code contributions to TiDB. TiDB uses a workflow based on [pull requests](https://docs.github.com/en/github/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/about-pull-requests). 4 | 5 | ## Before contributing 6 | 7 | Contributing to TiDB does *not* start with opening a pull request. We expect contributors to reach out to us first to discuss the overall approach together. Without consensus with the TiDB committers, contributions might require substantial rework or will not be reviewed. So please [create a GitHub issue](report-an-issue.md), discuss under an existing issue, or create a topic on the [internal.tidb.io](https://internals.tidb.io) and reach consensus. 8 | 9 | For newcomers, you can check the [starter issues](https://github.com/pingcap/tidb/contribute), which are annotated with a "good first issue" label. These are issues suitable for new contributors to work with and won't take long to fix. But because the label is typically added at triage time it can turn out to be inaccurate, so do feel free to leave a comment if you think the classification no longer applies. 10 | 11 | To get your change merged you need to sign the [CLA](https://cla-assistant.io/pingcap/tidb) to grant PingCAP ownership of your code. 12 | 13 | ## Contributing process 14 | 15 | After a consensus is reached in issues, it's time to start the code contributing process: 16 | 17 | 1. Assign the issue to yourself via [/assign](https://prow.tidb.io/command-help?repo=pingcap%2Ftidb#assign). This lets other contributors know you are working on the issue so they won't make duplicate efforts. 18 | 2. Follow the [GitHub workflow](https://guides.github.com/introduction/flow/), commit code changes in your own git repository branch and open a pull request for code review. 19 | 3. Make sure the continuous integration checks on your pull request are green (i.e. successful). 20 | 4. Review and address [comments on your pull request](https://docs.github.com/en/github/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/commenting-on-a-pull-request). If your pull request becomes unmergeable, you need to [rebase your pull request](https://github.com/edx/edx-platform/wiki/How-to-Rebase-a-Pull-Request#perform-a-rebase) to keep it up to date. Since TiDB uses [squash and merge](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/configuring-pull-request-merges/about-merge-methods-on-github#squashing-your-merge-commits), simply merging master to catch up the change is also acceptable. 21 | 5. When your pull request gets enough approvals (the default number is 2) and all other requirements are met, it will be merged. 22 | 6. Handle regressions introduced by your change. Although committers bear the main responsibility to fix regressions, it's quite nice for you to handle it (reverting the change or sending fixes). 23 | 24 | Clear and kind communication is key to this process. 25 | 26 | ## Referring to an issue 27 | 28 | Code repositories in TiDB community require **ALL** the pull requests referring to its corresponding issues. In the pull request body, there **MUST** be one line starting with `Issue Number: ` and linking the relevant issues via the [keyword](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword), for example: 29 | 30 | If the pull request resolves the relevant issues, and you want GitHub to close these issues automatically after it merged into the default branch, you can use the syntax (`KEYWORD #ISSUE-NUMBER`) like this: 31 | 32 | ``` 33 | Issue Number: close #123 34 | ``` 35 | 36 | If the pull request links an issue but does not close it, you can use the keyword `ref` like this: 37 | 38 | ``` 39 | Issue Number: ref #456 40 | ``` 41 | 42 | Multiple issues should use full syntax for each issue and separate by a comma, like: 43 | 44 | ``` 45 | Issue Number: close #123, ref #456 46 | ``` 47 | 48 | For pull requests trying to close issues in a different repository, contributors need to first create an issue in the same repository and use this issue to track. 49 | 50 | If the pull request body does not provide the required content, the bot will add the `do-not-merge/needs-linked-issue` label to the pull request to prevent it from being merged. 51 | 52 | ## Writing tests 53 | 54 | One important thing when you make code contributions to TiDB is tests. Tests should be always considered as a part of your change. Any code changes that cause semantic changes or new function additions to TiDB should have corresponding test cases. And of course you can not break any existing test cases if they are still valid. It's recommended to [run tests](../get-started/write-and-run-unit-tests.md) on your local environment first to find obvious problems and fix them before opening the pull request. 55 | 56 | It's also highly appreciated if your pull request only contains test cases to increase test coverage of TiDB. Supplement test cases for existing modules is a good and easy way to become acquainted with existing code. 57 | 58 | ## Making good pull requests 59 | 60 | When creating a pull request for submission, there are several things that you should consider to help ensure that your pull request is accepted: 61 | 62 | * Does the contribution alter the behavior of features or components in a way that it may break previous users' programs and setups? If yes, there needs to be a discussion and agreement that this change is desirable. 63 | * Does the contribution conceptually fit well into TiDB? Is it too much of a special case such that it makes things more complicated for the common case, or bloats the abstractions/APIs? 64 | * Does the contribution make a big impact on TiDB's build time? 65 | * Does your contribution affect any documentation? If yes, you should add/change proper documentation. 66 | * If there are any new dependencies, are they under active maintenances? What are their licenses? 67 | 68 | ## Making good commits 69 | 70 | Each feature or bugfix should be addressed by a single pull request, and for each pull request there may be several commits. In particular: 71 | 72 | * Do *not* fix more than one issues in the same commit (except, of course, if one code change fixes all of them). 73 | * Do *not* do cosmetic changes to unrelated code in the same commit as some feature/bugfix. 74 | 75 | ## Waiting for review 76 | 77 | To begin with, please be patient! There are many more people submitting pull requests than there are people capable of reviewing your pull request. Getting your pull request reviewed requires a reviewer to have the spare time and motivation to look at your pull request. If your pull request has not received any notice from reviewers (i.e., no comment made) for some time, you can ping the reviewers and assignees, or take it to [internal.tidb.io](https://internals.tidb.io) for more attention. 78 | 79 | When someone does manage to find the time to look at your pull request, they will most likely make comments about how it can be improved (don't worry, even committers/maintainers have their pull requests sent back to them for changes). It is then expected that you update your pull request to address these comments, and the review process will thus iterate until a satisfactory solution has emerged. 80 | -------------------------------------------------------------------------------- /src/contribute-to-tidb/introduction.md: -------------------------------------------------------------------------------- 1 | # Contribute to TiDB 2 | 3 | TiDB is developed by an open and friendly community. Everybody is cordially welcome to join the community and contribute to TiDB. We value all forms of contributions, including, but not limited to: 4 | 5 | * Code reviewing of the existing patches 6 | * Documentation and usage examples 7 | * Community participation in forums and issues 8 | * Code readability and developer guide 9 | * We welcome contributions that add code comments and code refactor to improve readability 10 | * We also welcome contributions to docs to explain the design choices of the internal 11 | * Test cases to make the codebase more robust 12 | * Tutorials, blog posts, talks that promote the project 13 | 14 | Here are guidelines for contributing to various aspect of the project: 15 | 16 | * [Community Guideline](community-guideline.md) 17 | * [Report an Issue](report-an-issue.md) 18 | * [Issue Triage](issue-triage.md) 19 | * [Contribute Code](contribute-code.md) 20 | * [Cherry-pick a Pull Request](cherrypick-a-pr.md) 21 | * [Review a Pull Request](review-a-pr.md) 22 | * [Make a Proposal](make-a-proposal.md) 23 | * [Code Style and Quality Guide](code-style-and-quality-guide.md) 24 | * [Write Document](write-document.md) 25 | * [Release Notes Language Style Guide](contribute-to-tidb/release-notes-style-guide.md) 26 | * [Committer Guide](committer-guide.md) 27 | * [Miscellaneous Topics](miscellaneous-topics.md) 28 | 29 | Any other question? Reach out to the [TiDB Internals forum](https://internals.tidb.io/) to get help! 30 | -------------------------------------------------------------------------------- /src/contribute-to-tidb/issue-triage.md: -------------------------------------------------------------------------------- 1 | # Issue Triage 2 | 3 | TiDB uses an issue-centric workflow for development. Every problem, enhancement and feature starts with an issue. For bug issues, you need to perform some more triage operations on the issues. 4 | 5 | ## Diagnose issue severity 6 | 7 | The severity of a bug reflects the level of impact that the bug has on users when they use TiDB. The greater the impact, the higher severity the bug is. For higher severity bugs, we need to fix them faster. Although the impact of bugs can not be exhausted, they can be divided into four levels. 8 | 9 | ### Critical 10 | 11 | The bug affects critical functionality or critical data. It might cause huge losses to users and does not have a workaround. Some typical critical bugs are as follows: 12 | 13 | * Invalid query result (correctness issues) 14 | * TiDB returns incorrect results or results that are in the wrong order for a typical user-written query. 15 | * Bugs caused by type casts. 16 | * The parameters are not boundary value or invalid value, but the query result is not correct(except for overflow scenes). 17 | * Incorrect DDL and DML result 18 | * The data is not written to the disk, or wrong data is written. 19 | * Data and index are inconsistent. 20 | * Invalid features 21 | * Due to a regression, the feature can not work in its main workflow 22 | * Follower can not read follower. 23 | * SQL hint does not work. 24 | * SQL Plan 25 | * Cannot choose the best index. The difference between best plan and chosen plan is bigger than 200%. 26 | * DDL design 27 | * DDL process causes data accuracy issue. 28 | * Experimental feature 29 | * If the issue leads to another stable feature’s main workflow not work, and may occur on released version, the severity is critical. 30 | * If the issue leads to data loss, the severity is critical. 31 | * Exceptions 32 | * If the feature is clearly labeled as experimental, when it doesn’t work but doesn’t impact another stable feature’s main workflow or only impacts stable feature’s main workflow on master, the issue severity is major. 33 | * The feature has been deprecated and a viable workaround is available(at most major). 34 | * System stability 35 | * The system is unavailable for more than 5 minutes(if there are some system errors, the timing starts from failure recovery). 36 | * Tools cannot perform replication between upstream and downstream for more than 1 minute if there are no system errors. 37 | * TiDB cannot perform the upgrade operation. 38 | * TPS/QPS dropped 25% without system errors or rolling upgrades. 39 | * Unexpected TiKV core dump or TiDB panic(process crashed). 40 | * System resource leak, include but not limit to memory leak and goroutine leak. 41 | * System fails to recover from crash. 42 | * Security and compliance issues 43 | * CVSS score >= 9.0. 44 | * TiDB leaks secure information to log files, or prints customer data when set to be desensitized. 45 | * Backup or Recovery Issues 46 | * Failure to either backup or restore is always considered critical. 47 | * Incompatible Issues 48 | * Syntax/compatibility issue affecting default install of tier 1 application(i.e. Wordpress). 49 | * The DML result is incompatible with MySQL. 50 | * CI test case fail 51 | * Test cases which lead to CI failure and could always be reproduced. 52 | * Bug location information 53 | * Key information is missing in ERROR level log. 54 | * No data is reported in monitor. 55 | 56 | ### Major 57 | 58 | The bug affects major functionality. Some typical critical bugs are as follow: 59 | 60 | * Invalid query result 61 | * The query gets the wrong result caused by overflow. 62 | * The query gets the wrong result in the corner case. 63 | * For boundary value, the processing logic in TiDB is inconsistent with MySQL. 64 | * Inconsistent data precision. 65 | * Incorrect DML or DDL result 66 | * Extra or wrong data is written to TiDB with a DML in a corner case. 67 | * Invalid features 68 | * The corner case of the main workflow of the feature does not work. 69 | * The feature is experimental, but a main workflow does not work. 70 | * Incompatible issue of view functionality. 71 | * SQL Plan 72 | * Choose sub-optimal plan. The difference between best plan and chosen plan is bigger than 100% and less than 200% 73 | * System stability 74 | * TiDB panics but process does not exit. 75 | * Less important security and compliance issues 76 | * CVSS score >= 7.0 77 | * Issues that affects critical functionality or critical data but rare to reproduce(can’t be reproduced in one week, and have no clear reproduce steps) 78 | * CI test cases fail 79 | * Test case is not stable. 80 | * Bug location information 81 | * Key information is missing in WARN level log. 82 | * Data is not accurate in monitor. 83 | 84 | ### Moderate 85 | 86 | * SQL Plan 87 | * Cannot get the best plan due to invalid statistics. 88 | * Documentation issues 89 | * The bugs were caused by invalid parameters which rarely occurred in the product environment. 90 | * Security issues 91 | * CVSS score >= 4.0 92 | * Incompatible issues occurred on boundary value 93 | * Bug location information 94 | * Key information is missing in DEBUG/INFO level log. 95 | 96 | ### Minor 97 | 98 | The bug does not affect functionality or data. It does not even need a workaround. It does not impact productivity or efficiency. It is merely an inconvenience. For example: 99 | 100 | * Invalid notification 101 | * Minor compatibility issues 102 | * Error message or error code does not match MySQL. 103 | * Issues caused by invalid parameters or abnormal cases. 104 | 105 | ### Not a bug 106 | 107 | The following issues look like bugs but actually not. They should not be labeled `type/bug` and instead be only labeled `type/compatibility`: 108 | 109 | * Behavior is different from MySQL, but could be argued as correct. 110 | * Behavior is different from MySQL, but MySQL behavior differs between 5.7 and 8.0. 111 | 112 | ## Identify issue affected releases 113 | 114 | For `type/bug` issues, when they are created and identified as `severity/critical` or `severity/major`, the ti-chi-bot will assign a list of `may-affects-x.y` labels to the issue. For example, currently if we have version 5.0, 5.1, 5.2, 5.3, 4.0 and the in-sprint 5.4, when a `type/bug` issue is created and added label `severity/critical` or `severity/major`, the ti-chi-bot will add label `may-affects-4.0`, `may-affects-5.0`, `may-affects-5.1`, `may-affects-5.2`, and `may-affects-5.3`. These labels mean that whether the bug affects these release versions are not yet determined, and is awaiting being triaged. You could check [currently maintained releases list](https://pingcap.github.io/tidb-dev-guide/project-management/release-train-model.html#current-maintained-releases) for all releases. 115 | 116 | When a version is triaged, the triager needs to remove the corresponding `may-affects-x.y` label. If the version is affected, the triager needs to add a corresponding `affects-x.y` label to the issue and in the meanwhile the `may-affects-x.y` label can be automatically removed by the ti-chi-bot, otherwise the triager can simply remove the `may-affects-x.y` label. So when a issue has a label `may-affects-x.y`, this means the issue has not been diagnosed on version x.y. When a issue has a label `affects-x.y`, this means the issue has been diagnosed on version x.y and identified affected. When both the two labels are missing, this means the issue has been diagnosed on version x.y but the version is not affected. 117 | 118 | The status of the affection of a certain issue can be then determined by the combination of the existence of the corresponding `may-affects-x.y` and `affects-x.y` labels on the issue, see the table below for a clearer illustration. 119 | 120 | | may-affects-x.y | affects-x.y | status | 121 | |:--------------:|:-----------:|:-------------------------------------------------------------:| 122 | | YES | NO | version x.y has not been diagnosed | 123 | | NO | NO | version x.y has been diagnosed and identified as not affected | 124 | | NO | YES | version x.y has been diagnosed and identified as affected | 125 | | YES | YES | invalid status | 126 | -------------------------------------------------------------------------------- /src/contribute-to-tidb/make-a-proposal.md: -------------------------------------------------------------------------------- 1 | # Make a Proposal 2 | 3 | This page defines the best practices procedure for making a proposal in TiDB projects. This text is based on the content of [TiDB Design Document](https://github.com/pingcap/tidb/blob/7f4f5c02364b6578da561ec14f409a39ddf954a5/docs/design/README.md). 4 | 5 | ## Motivation 6 | 7 | Many changes, including bug fixes and documentation improvements can be implemented and reviewed via the normal GitHub pull request workflow. 8 | 9 | Some changes though are "substantial", and we ask that these be put through a bit of a design process and produce a consensus among the TiDB community. 10 | 11 | The process described in this page is intended to provide a consistent and controlled path for new features to enter the TiDB projects, so that all stakeholders can be confident about the direction the projects is evolving in. 12 | 13 | ## Who should initiate the design document? 14 | 15 | Everyone is encouraged to initiate a design document, but before doing it, please make sure you have an intention of getting the work done to implement it. 16 | 17 | ## Before creating a design document 18 | 19 | A hastily-proposed design document can hurt its chances of acceptance. Low-quality proposals, proposals for previously-rejected features, or those that don't fit into the near-term roadmap, may be quickly rejected, which can be demotivating for the unprepared contributor. Laying some groundwork ahead of the design document can make the process smoother. 20 | 21 | Although there is no single way to prepare for submitting a design document, it is generally a good idea to pursue feedback from other project developers beforehand, to ascertain that the design document may be desirable; having a consistent impact on the project requires concerted effort toward consensus-building. 22 | 23 | The most common preparations for writing and submitting a draft of design document is on the [TiDB Internals forum](https://internals.tidb.io/). 24 | 25 | ## What is the process? 26 | 27 | 1. Create an issue describing the problem, goal and solution. 28 | 2. Get responses from other contributors to see if the proposal is generally acceptable. 29 | 3. Create a pull request with a design document based on the [template](https://github.com/pingcap/tidb/blob/7f4f5c02364b6578da561ec14f409a39ddf954a5/docs/design/TEMPLATE.md) as `YYYY-MM-DD-my-feature.md`. 30 | 4. Discussion takes place, and the text is revised in response. 31 | 5. The design document is accepted or rejected when at least two committers reach consensus and no objection from the committer. 32 | 6. If accepted, create a [tracking issue](https://github.com/pingcap/tidb/issues/new?assignees=&labels=type%2Fenhancement&template=development-task.md) for the design document or convert one from a previous discuss issue. The tracking issue basically tracks subtasks and progress. And refer the tracking issue in the design document replacing placeholder in the template. 33 | 7. Merge the pull request of design. 34 | 8. Start the implementation. 35 | 36 | Please refer to the tracking issue from subtasks to track the progress. 37 | 38 | An example that almost fits into this model is the proposal "Support global index for partition table", without following the latest template. 39 | 40 | * [tracking issue](https://github.com/pingcap/tidb/issues/18032) 41 | * [pull request of design document](https://github.com/pingcap/tidb/pull/18982) 42 | -------------------------------------------------------------------------------- /src/contribute-to-tidb/miscellaneous-topics.md: -------------------------------------------------------------------------------- 1 | # Miscellaneous Topics 2 | 3 | ## Communication channels 4 | 5 | - [TiDB Internals Forum](https://internals.tidb.io): TiDB hosts a Discourse instance for TiDB development discussions. It has many separate categories for different topics. You can discuss anything about TiDB development and community in the forum. 6 | 7 | ## Related projects 8 | 9 | - [TiKV](https://github.com/tikv/tikv): TiKV is an open-source, distributed, and transactional key-value database. It is used by TiDB as the storage layer. 10 | - [Talent Plan](https://university.pingcap.com/talent-plan/): Talent Plan is an open source training program initiated by PingCAP. It aims to create and combine some open source learning materials for people interested in open source, distributed systems, Rust, Golang, and other infrastructure knowledges. 11 | 12 | ## Community events 13 | -------------------------------------------------------------------------------- /src/contribute-to-tidb/release-notes-style-guide.md: -------------------------------------------------------------------------------- 1 | # Release Notes Language Style Guide 2 | 3 | A concise release note can clearly and accurately deliver to users how your PR can make a difference. Your release note written in a PR will be presented in [docs.pingcap.com](https://docs.pingcap.com/tidb/stable/release-notes) as a part of the TiDB documentation. 4 | 5 | This release notes language style guide briefly explains what a quality release note looks like, provides examples, and aims to help you write quality release notes. 6 | 7 | ## What a quality release note looks like 8 | 9 | A high-quality release note has the following merits: 10 | 11 | - Clear in type 12 | - Adequate and clear in meaning 13 | - User perspective 14 | 15 | A release note with a distinguishable type can help users quickly identify the nature or goal of your PR change. Other teams will also benefit from it. 16 | 17 | Depending on what your PR changes, you can refer to one of the following release note types: 18 | 19 | - Compatibility change 20 | - Bug fix 21 | - Improvement or Feature enhancement 22 | 23 | ## Compatibility change 24 | 25 | A compatibility change note means: 26 | 27 | - Your PR adds, removes, or modifies one or more configuration items or system variables. 28 | - Your PR modifies the default value of a configuration item or system variable. 29 | 30 | For this type of note, you should clearly and adequately state the following aspects: 31 | 32 | - The previous code behavior, configuration item, or default value. 33 | - The new code behavior, configuration item, or default value since the new version. 34 | 35 | Note that the object of the change should be user-perceivable. If the changed configuration item or system variable is not supposed to be exposed to users, do not include it in your release notes. 36 | 37 | **Examples:** 38 | 39 | | **Not recommended** | **Clear in type** | **Adequate and clear in meaning** | **User perspective** | Recommended | 40 | | ------------------------------------------------------------ | ----------------- | --------------------------------- | -------------------- | ------------------------------------------------------------ | 41 | | copr: cast invalid utf8 string to real bug fix | ❌ | ❌ | ❌ | Previously, when TiDB converts an illegal UTF-8 string to a Real type, an error is reported directly. From now on, TiDB will process the conversion according to the legal UTF-8 prefix in the string. | 42 | | sink: fix kafka max message size inaccurate issue | ❌ | ❌ | ❌ | Change the default value of Kafka Sink max-message-bytes from 512 MB to 1 MB to prevent TiCDC from sending too large messages to Kafka clusters | 43 | | cdc/sink: adjust kafka initialization logic | ❌ | ❌ | ❌ | Change the default value of Kafka Sink partition-num to 3 so that TiCDC distributes messages across Kafka partitions more evenly | 44 | | cmd: hide --sort-dir in changefeed command. (deprecated warning exists) | ✅ | ❌ | ❌ | Deprecate `--sort-dir` in the `cdc cli changefeed` command. Instead, users can set `--sort-dir` in the `cdc server` command. | 45 | 46 | ## Bug fix 47 | 48 | A bug fix note means that your PR fixes an existing bug or issue. This type of notes start with "Fix" followed by "the issue/bug". 49 | 50 | Write your note clearly and adequately so that your target readers can get the main point of your bug fix. The bug or issue must be directly perceivable to the users, and you can refer to the associated GitHub issues. 51 | 52 | In addition, it is recommended to highlight the bug trigger condition or the workaround if there is any. 53 | 54 | **Examples:** 55 | 56 | | **Not recommended** | **Clear in type** | **Adequate and clear in meaning** | **User perspective** | Recommended | 57 | | ------------------------------------------------------------ | ----------------- | --------------------------------- | -------------------- | ------------------------------------------------------------ | 58 | | lock_resolver: avoid pessimistic transactions using resolveLocksForWrite | ❌ | ✅ | ❌ | Fix the issue that committing pessimistic transactions might cause write conflict | 59 | | retry when meeting stablish conn fails | ❌ | ❌ | ❌ | Fix the issue of unexpected results when TiFlash fails to establish MPP connections | 60 | | Fix the issue that greatest(datetime) union null returns empty string | ✅ | ❌ | ✅ | Fix the issue that the query result might be wrong when NULL is in the UNION subquery | 61 | | copr: make CM Sketch built with the same encoding as what TiDB assumes | ❌ | ❌ | ❌ | Fix the issue of potential wrong analyzed statistics when `tidb_analyze_version` is set to `1` | 62 | 63 | ## Improvement 64 | 65 | An improvement note means that your PR improves stability or performance of the product, or enhances an existing feature. In addition to describing what your PR has changed, you should also mention how users can benefit from it. 66 | 67 | This type of release note consists of two parts: what you have changed + the benefit of your change. This type of release notes often starts with "support", "increase", "improve", "optimize", etc. 68 | 69 | **Examples:** 70 | 71 | | **Not recommended** | **Clear in type** | **Adequate and clear in meaning** | **User perspective** | Recommended | 72 | | ------------------------------------------------------------ | ----------------- | --------------------------------- | -------------------- | ------------------------------------------------------------ | 73 | | Not use the stale read request's `start_ts` to update `max_ts` to avoid commit request keep retrying | ✅ | ✅ | ❌ | Improve commit performance in some edge cases | 74 | | Restore many small tables would be faster. | ✅ | ❌ | ❌ | Split and scatter Regions concurrently to improve restore speed | 75 | | server: stop status server early when gracefully shutdown | ✅ | ❌ | ❌ | Shut down the status server first to make sure that the client can correctly check the shutdown status | 76 | | Better err msg when PD endpoint missing certificate | ✅ | ❌ | ✅ | Improve the error message when connecting to a TLS protected PD endpoint without a certificate | 77 | -------------------------------------------------------------------------------- /src/contribute-to-tidb/report-an-issue.md: -------------------------------------------------------------------------------- 1 | # Report an Issue 2 | 3 | If you think you have found an issue in TiDB, you can report it to the [issue tracker](https://github.com/pingcap/tidb/issues). If you would like to report issues to TiDB documents or this development guide, they are in separate GitHub repositories, so you need to file issues to corresponding issue tracker, [TiDB document issue tracker](https://github.com/pingcap/docs/issues) and [TiDB Development Guide issue tracker](https://github.com/pingcap/tidb-dev-guide/issues). Read [Write Document](write-document.md) for more details. 4 | 5 | ## Checking if an issue already exists 6 | 7 | The first step before filing an issue report is to see whether the problem has already been reported. You can [use the search bar to search existing issues](https://docs.github.com/en/github/administering-a-repository/finding-information-in-a-repository/using-search-to-filter-issues-and-pull-requests). This doesn't always work, and sometimes it's hard to know what to search for, so consider this extra credit. We won't mind if you accidentally file a duplicate report. Don't blame yourself if your issue is closed as duplicated. We highly recommend if you are not sure about anything of your issue report, you can turn to [internal.tidb.io](https://internals.tidb.io) for a wider audience and ask for discussion or help. 8 | 9 | ## Filing an issue 10 | 11 | If the problem you're reporting is not already in the issue tracker, you can [open a GitHub issue](https://docs.github.com/en/issues/tracking-your-work-with-issues/creating-an-issue) with your GitHub account. TiDB uses issue template for different kinds of issues. Issue templates are a bundle of questions to collect necessary information about the problem to make it easy for other contributors to participate. For example, a bug report issue template consists of four questions: 12 | 13 | * Minimal reproduce step. 14 | * What did you expect to see? 15 | * What did you see instead? 16 | * What is your TiDB version? 17 | 18 | Answering these questions give the details about your problem so other contributors or TiDB users could pick up your issue more easily. 19 | 20 | As previous section shows, duplicated issues should be reduced. To help others who encountered the problem find your issue, except for problem details answered in the issue template, a descriptive title which contains information that might be unique to it also helps. This can be the components your issue belongs to or database features used in your issue, the conditions that trigger the bug, or part of the error message if there is any. 21 | 22 | ## Making good issues 23 | 24 | Except for a good title and detailed issue message, you can also add suitable labels to your issue via [/label](https://prow.tidb.io/command-help?repo=pingcap%2Ftidb#type), especially which component the issue belongs to and which versions the issue affects. Many committers and contributors only focus on certain subsystems of TiDB. Setting the appropriate component is important for getting their attention. Some issues might affect multiple releases. You can query [Issue Triage chapter](issue-triage.md) for more information about what need to do with such issues. 25 | 26 | If you are able to, you should take more considerations on your issue: 27 | 28 | * Does the feature fit well into TiDB's architecture? Will it scale and keep TiDB flexible for the future, or will the feature restrict TiDB in the future? 29 | * Is the feature a significant new addition (rather than an improvement to an existing part)? If yes, will the community commit to maintaining this feature? 30 | * Does this feature align well with currently ongoing efforts? 31 | * Does the feature produce additional value for TiDB users or developers? Or does it introduce the risk of regression without adding relevant user or developer benefit? 32 | 33 | Deep thoughts could help the issue proceed faster and help build your own reputation in the community. 34 | 35 | ## Understanding the issue's progress and status 36 | 37 | Once your issue is created, other contributors might take part in. You need to discuss with them, provide more information they might want to know, address their comments to reach consensus and make the progress proceeds. But please realize there are always more pending issues than contributors are able to handle, and especially TiDB community is a global one, contributors reside all over the world and they might already be very busy with their own work and life. Please be patient! If your issue gets stale for some time, it's okay to ping other participants, or take it to [internal.tidb.io](https://internals.tidb.io) for more attention. 38 | 39 | ## Disagreement with a resolution on the issue tracker 40 | 41 | As humans, we will have differences of opinions from time to time. First and foremost, please be respectful that care, thought, and volunteer time went into the resolution. 42 | 43 | With this in mind, take some time to consider any comments made in association with the resolution of the issue. On reflection, the resolution steps may seem more reasonable than you initially thought. 44 | 45 | If you still feel the resolution is incorrect, then raise a thoughtful question on [internal.tidb.io](https://internals.tidb.io). Further argument and disrespectful discourse on [internal.tidb.io](https://internals.tidb.io) after a consensus has been reached amongst the committers is unlikely to win any converts. 46 | 47 | ## Reporting security vulnerabilities 48 | 49 | Security issues are not suitable to report in public early, so different tracker strategy is used. Please refer to the [dedicated process](https://github.com/pingcap/tidb/security/policy). 50 | -------------------------------------------------------------------------------- /src/contribute-to-tidb/review-a-pr.md: -------------------------------------------------------------------------------- 1 | # Review a Pull Request 2 | 3 | TiDB values any [code review](https://en.wikipedia.org/wiki/Code_review). One of the bottlenecks in the TiDB development process is the lack of code reviews. If you browse the issue tracker, you will see that numerous issues have a fix, but cannot be merged into the main source code repository, because no one has reviewed the proposed solution. Reviewing a pull request can be just as informative as providing a pull request and it will allow you to give constructive comments on another developer's work. It is a common misconception that in order to be useful, a code review has to be perfect. This is not the case at all! It is helpful to just test the pull request and/or play around with the code and leave comments in the pull request. 4 | 5 | ## Principles of the code review 6 | 7 | * Technical facts and data overrule opinions and personal preferences. 8 | * Software design is about trade-offs, and there is no silver bullet. 9 | 10 | Everyone comes from different technical backgrounds with different knowledge. They have their own personal preferences. It is important that the code review is not based on biased opinions. 11 | 12 | Sometimes, making choices of accepting or rejecting a pull request can be tricky as in the following situations: 13 | 14 | * Suppose that a pull request contains special optimization that can improve the overall performance by 30%. However, the pull request introduces a totally different code path, and every subsequent feature must consider it. 15 | * Suppose that a pull request is to fix a critical bug, but the change in the pull request is risky to introduce other bugs. 16 | 17 | If a pull request under your review is in these tricky situations, what is the right choice, accepting the pull request or rejecting it? The answer is always "it depends." Software design is more like a kind of art than technology. It is about aesthetics and your taste of the code. There are always trade-offs, and often there's no perfect solution. 18 | 19 | ## Triaging pull requests 20 | 21 | Some pull request authors may not be familiar with TiDB, TiDB development workflow or TiDB community. They don't know what labels should be added to the pull requests and which experts could be asked for review. If you are able to, it would be great for you to triage the pull requests, adding suitable labels to the pull requests, asking corresponding experts to review the pull requests. These actions could help more contributors notice the pull requests and make quick responses. 22 | 23 | ## Checking pull requests 24 | 25 | There are some basic aspects to check when you review a pull request: 26 | 27 | * **Concentration**. One pull request should only do one thing. No matter how small it is, the change does exactly one thing and gets it right. Don't mix other changes into it. 28 | * **Tests**. A pull request should be test covered, whether the tests are unit tests, integration tests, or end-to-end tests. Tests should be sufficient, correct and don't slow down the CI pipeline largely. 29 | * **Functionality**. The pull request should implement what the author intends to do and fit well in the existing code base, resolve a real problem for TiDB users. To get the author's intention and the pull request design, you could follow the discussions in the corresponding GitHub issue or [internal.tidb.io](https://internals.tidb.io) topic. 30 | * **Style**. Code in the pull request should follow common programming style. For Go and Rust, there are built-in tools with the compiler toolchain. However, sometimes the existing code is inconsistent with the style guide, you should maintain consistency with the existing code or file a new issue to fix the existing code style first. 31 | * **Documentation**. If a pull request changes how users build, test, interact with, or release code, you must check whether it also updates the related documentation such as READMEs and any generated reference docs. Similarly, if a pull request deletes or deprecates code, you must check whether or not the corresponding documentation should also be deleted. 32 | * **Performance**. If you find the pull request may affect performance, you could ask the author to provide a benchmark result. 33 | 34 | ## Writing code review comments 35 | 36 | When you review a pull request, there are several rules and suggestions you should take to [write better comments](https://docs.github.com/en/github/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/commenting-on-a-pull-request): 37 | 38 | * **Be respectful to pull request authors and other reviewers**. Code review is a part of your community activities. You should follow the community requirements. 39 | * **Asking questions instead of making statements**. The wording of the review comments is very important. To provide review comments that are constructive rather than critical, you can try asking questions rather than making statements. 40 | * **Offer sincere praise**. Good reviewers focus not only on what is wrong with the code but also on good practices in the code. As a reviewer, you are recommended to offer your encouragement and appreciation to the authors for their good practices in the code. In terms of mentoring, telling the authors what they did is right is even more valuable than telling them what they did is wrong. 41 | * **Provide additional details and context of your review process**. Instead of simply "approving" the pull request. If you test the pull request, report the result and your test environment details. If you request changes, try to suggest how. 42 | 43 | ## Accepting pull requests 44 | 45 | Once you think the pull request is ready, you can [approve](https://docs.github.com/en/github/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/approving-a-pull-request-with-required-reviews) it, commenting with `/lgtm` is also valid. 46 | 47 | In the TiDB community, most repositories require two approvals before a pull request can be accepted. A few repositories require a different number of approvals, but two approvals are the default setting. After the required `lgtm` count is met, `lgtm` label will be added. 48 | Finally committer can [/approve](https://prow.tidb.io/command-help?repo=pingcap%2Ftidb#merge) the pull request, some special scopes need `/approve` by the scope approvers(define in `OWNERS` files). 49 | -------------------------------------------------------------------------------- /src/extending-tidb/add-a-function.md: -------------------------------------------------------------------------------- 1 | # Add a function 2 | 3 | To add a builtin function to TiDB the best practice is to look at MySQL first and try to implement the function in such a way that it is commpatible. Avoid adding functions that are already deprecated in MySQL or that might soon be deprecrated. 4 | 5 | Here we will implement a `HELLO()` function that has one argument that is a string. For this you need [a clone of the pingcap/tidb repository](../get-started/build-tidb-from-source.md#clone) 6 | 7 | ``` 8 | sql> SELECT HELLO("world"); 9 | ERROR: 1305 (42000): FUNCTION test.hello does not exist 10 | ``` 11 | 12 | The first step is to define the name of the function in `parser/ast/functions.go`: 13 | ```go 14 | // List scalar function names. 15 | const ( 16 | ... 17 | Hello = "hello" 18 | ) 19 | ``` 20 | 21 | This links `ast.Hello` with "hello". Note that the lookup for the function is done with the lowercase name, so always use the lowercase name, otherwise it won't find the function. 22 | 23 | The next step is to modify `expression/builtin.go` 24 | 25 | ```go 26 | var funcs = map[string]functionClass{ 27 | ... 28 | ast.Hello: &helloFunctionClass{baseFunctionClass{ast.Hello, 1, 1}}, 29 | } 30 | ``` 31 | 32 | Now we need to define `helloFunctionClass`. We will do this in `expression/builtin_string.go`. The functions are organised in multiple files, pick the one that fits best. 33 | 34 | ```go 35 | var ( 36 | ... 37 | _ functionClass = &helloFunctionClass{} 38 | ) 39 | ... 40 | var ( 41 | _ builtinFunc = &builtinHelloSig{} 42 | ) 43 | ... 44 | type helloFunctionClass struct { 45 | baseFunctionClass 46 | } 47 | 48 | func (c *helloFunctionClass) getFunction(ctx BuildContext, args []Expression) (builtinFunc, error) { 49 | if err := c.verifyArgs(args); err != nil { 50 | return nil, err 51 | } 52 | bf, err := newBaseBuiltinFuncWithTp(ctx, c.funcName, args, types.ETString, types.ETString) 53 | if err != nil { 54 | return nil, err 55 | } 56 | sig := &builtinHelloSig{bf} 57 | return sig, nil 58 | } 59 | 60 | type builtinHelloSig struct { 61 | baseBuiltinFunc 62 | } 63 | 64 | func (b *builtinHelloSig) Clone() builtinFunc { 65 | newSig := &builtinHelloSig{} 66 | newSig.cloneFrom(&b.baseBuiltinFunc) 67 | return newSig 68 | } 69 | 70 | func (b *builtinHelloSig) evalString(ctx EvalContext, row chunk.Row) (name string, isNull bool, err error) { 71 | name, isNull, err = b.args[0].EvalString(ctx, row) 72 | if isNull || err != nil { 73 | return name, isNull, err 74 | } 75 | return "hello " + name, false, nil 76 | } 77 | ``` 78 | 79 | The `getFunction()` method can return different functions depending on the type and number of arguments. This example always returns the same function that has one string argument and returns a string. 80 | 81 | Here `evalString()` gets called for every row. If the function returns an integer you have to use `evalInt` and there are also functions for Decimal, Real, Time and JSON. 82 | 83 | Now you need to [build TiDB](../get-started/build-tidb-from-source.md#build) again and try the newly added function. 84 | 85 | The final result: 86 | 87 | ``` 88 | sql> SELECT HELLO("world"); 89 | +----------------+ 90 | | HELLO("world") | 91 | +----------------+ 92 | | hello world | 93 | +----------------+ 94 | 1 row in set (0.0007 sec) 95 | ``` 96 | 97 | To show the function with multiple rows: 98 | 99 | ``` 100 | sql> WITH names AS (SELECT "Europe" AS "name" UNION ALL SELECT "America" UNION ALL SELECT "China") 101 | -> SELECT HELLO(name) FROM names; 102 | +---------------+ 103 | | HELLO(name) | 104 | +---------------+ 105 | | hello Europe | 106 | | hello America | 107 | | hello China | 108 | +---------------+ 109 | 3 rows in set (0.0008 sec) 110 | ``` 111 | 112 | For [testing](../get-started/write-and-run-unit-tests.md) have a look at `expression/builtin_string_test.go`. -------------------------------------------------------------------------------- /src/extending-tidb/introduction.md: -------------------------------------------------------------------------------- 1 | # Extending TiDB 2 | -------------------------------------------------------------------------------- /src/get-started/build-tidb-from-source.md: -------------------------------------------------------------------------------- 1 | # Get the code, build, and run 2 | 3 | ## Prerequisites 4 | 5 | * `git`: The TiDB source code is hosted on GitHub as a git repository. To work with the git repository, please [install `git`](https://git-scm.com/downloads). 6 | * `go`: TiDB is a Go project. Therefore, you need a working Go environment to build it. See the previous [Install Golang](install-golang.md) section to prepare the environment. 7 | * `gcc`: `gcc` command is required to use `cgo` while building. To install `gcc`, search for appropriate install guide for your OS. 8 | * `mysql` client (optional): After building TiDB from source, you can use the official [MySQL client](https://dev.mysql.com/downloads/mysql/) to connect to TiDB. It is not required if you want to build TiDB only. 9 | 10 | > **Note:** 11 | > 12 | > TiDB could compile and run on Windows 10. However, it is not expected to be deployed on Windows, where you might encounter many compatibility problems. To have a better experience, we recommend you [install WSL2](https://docs.microsoft.com/en-us/windows/wsl/install-win10) first. 13 | 14 | ## Clone 15 | 16 | Clone the source code to your development machine: 17 | 18 | ```bash 19 | git clone https://github.com/pingcap/tidb.git 20 | ``` 21 | 22 | ## Build 23 | 24 | Build TiDB from the source code: 25 | 26 | ```bash 27 | cd tidb 28 | make 29 | ``` 30 | 31 | ## Run 32 | 33 | Now that you have the `tidb-server` binary under the `bin` directory, execute it for a TiDB server instance: 34 | 35 | ```bash 36 | ./bin/tidb-server 37 | ``` 38 | 39 | This starts the TiDB server listening on port 4000 with embedded `unistore`. 40 | 41 | ## Connect 42 | 43 | You can use the official MySQL client to connect to TiDB: 44 | 45 | ```bash 46 | mysql -h 127.0.0.1 -P 4000 -u root -D test --prompt="tidb> " --comments 47 | ``` 48 | 49 | where 50 | 51 | * `-h 127.0.0.1` sets the Host to local host loopback interface 52 | * `-P 4000` uses port 4000 53 | * `-u root` connects as root user (`-p` not given; the development build has no password for root.) 54 | * `-D test` uses the Schema/Database test 55 | * `--prompt "tidb> "` sets the prompt to distinguish it from a connection to MySQL 56 | * `--comments` preserves comments like `/*T![clustered_index NONCLUSTERED */` instead of stripping them when sending the query to the server. 57 | 58 | If you encounter any problems during your journey, do not hesitate to reach out on the [TiDB Internals forum](https://internals.tidb.io/). 59 | -------------------------------------------------------------------------------- /src/get-started/commit-code-and-submit-a-pull-request.md: -------------------------------------------------------------------------------- 1 | # Commit the code and submit a pull request 2 | 3 | The TiDB project uses [Git](https://git-scm.com/) to manage its source code. To contribute to the project, you need to get familiar with Git features so that your changes can be incorporated into the codebase. 4 | 5 | This section addresses some of the most common questions and problems that new contributors might face. This section also covers some Git basics; however if you find that the content is a little difficult to understand, we recommend that you first read the following introductions to Git: 6 | 7 | * The "Beginner" and "Getting Started" sections of [this tutorial](https://www.atlassian.com/git/tutorials) from Atlassian 8 | * [Documentation](https://docs.github.com/en/github/getting-started-with-github/set-up-git) and [guides](https://guides.github.com/introduction/git-handbook/) for beginners from Github 9 | * A more in-depth [book](https://git-scm.com/book/en/v2/) from Git 10 | 11 | ## Prerequisites 12 | 13 | Before you create a pull request, make sure that you've installed Git, forked [pingcap/tidb](https://github.com/pingcap/tidb), and cloned the upstream repo to your PC. The following instructions use the command line interface to interact with Git; there are also several GUIs and IDE integrations that can interact with Git too. 14 | 15 | If you've cloned the upstream repo, you can reference it using `origin` in your local repo. Next, you need to set up a remote for the repo your forked using the following command: 16 | 17 | ```bash 18 | git remote add dev https://github.com/your_github_id/tidb.git 19 | ``` 20 | 21 | You can check the remote setting using the following command: 22 | 23 | ```bash 24 | git remote -v 25 | # dev https://github.com/username/tidb.git (fetch) 26 | # dev https://github.com/username/tidb.git (push) 27 | # origin https://github.com/pingcap/tidb.git (fetch) 28 | # origin https://github.com/pingcap/tidb.git (push) 29 | ``` 30 | 31 | ## Standard Process 32 | 33 | The following is a normal procedure that you're likely to use for the most common minor changes and PRs: 34 | 35 | 1. Ensure that you're making your changes on top of master and get the latest changes: 36 | 37 | ```bash 38 | git checkout master 39 | git pull master 40 | ``` 41 | 42 | 2. Create a new branch for your changes: 43 | 44 | ```bash 45 | git checkout -b my-changes 46 | ``` 47 | 48 | 3. Make some changes to the repo and test them. 49 | 50 | If the repo is buiding with [Bazel](https://bazel.build/) tool, you should update the bazel files(*.bazel, DEPS.bzl) also. 51 | 52 | 4. Commit your changes and push them to your `dev` remote repository: 53 | 54 | ```bash 55 | # stage files you created/changed/deleted 56 | git add path/to/changed/file.go path/to/another/changed/file.go 57 | 58 | # commit changes staged, make sure the commit message is meaningful and readable 59 | git commit -s -m "pkg, pkg2, pkg3: what's changed" 60 | 61 | # optionally use `git status` to check if the change set is correct 62 | # git status 63 | 64 | # push the change to your `dev` remote repository 65 | git push --set-upstream dev my-changes 66 | ``` 67 | 68 | 5. Make a PR from your fork to the master branch of pingcap/tidb. For more information on how to make a PR, see [Making a Pull Request](https://guides.github.com/activities/forking/#making-a-pull-request) in GitHub Guides. 69 | 70 | When making a PR, look at the [PR template](https://raw.githubusercontent.com/pingcap/tidb/master/.github/pull_request_template.md) and follow the commit message format, PR title format, and checklists. 71 | 72 | After you create a PR, if your reviewer requests code changes, the procedure for making those changes is similar to that of making a PR, with some steps skipped: 73 | 74 | 1. Switch to the branch that is the head and get the latest changes: 75 | 76 | ```bash 77 | git checkout my-changes 78 | git pull 79 | ``` 80 | 81 | 2. Make, stage, and commit your additional changes just like before. 82 | 3. Push those changes to your fork: 83 | 84 | ```bash 85 | git push 86 | ``` 87 | 88 | If your reviewer requests for changes with GitHub suggestion, you can commit the suggestion from the webpage. GitHub provides [documentation](https://docs.github.com/en/github/collaborating-with-issues-and-pull-requests/reviewing-changes-in-pull-requests/incorporating-feedback-in-your-pull-request#applying-suggested-changes) for this case. 89 | 90 | ## Conflicts 91 | 92 | When you edit your code locally, you are making changes to the version of pingcap/tidb that existed when you created your feature branch. As such, when you submit your PR it is possible that some of the changes that have been made to pingcap/tidb since then conflict with the changes you've made. 93 | 94 | When this happens, you need to resolve the conflicts before your changes can be merged. First, get a local copy of the conflicting changes: checkout your local master branch with `git checkout master`, then `git pull master` to update it with the most recent changes. 95 | 96 | ### Rebasing 97 | 98 | You're now ready to start the rebasing process. Checkout the branch with your changes and execute `git rebase master`. 99 | 100 | When you rebase a branch on master, all the changes on your branch are reapplied to the most recent version of master. In other words, Git tries to pretend that the changes you made to the old version of master were instead made to the new version of master. During this process, you should expect to encounter at least one "rebase conflict." This happens when Git's attempt to reapply the changes fails because your changes conflict with other changes that have been made. You can tell that this happened because you'll see lines in the output that look like: 101 | 102 | ```text 103 | CONFLICT (content): Merge conflict in file.go 104 | ``` 105 | 106 | When you open these files, you'll see sections of the form 107 | 108 | ```text 109 | <<<<<<< HEAD 110 | Original code 111 | ======= 112 | Your code 113 | >>>>>>> 8fbf656... Commit fixes 12345 114 | ``` 115 | 116 | This represents the lines in the file that Git could not figure out how to rebase. The section between `<<<<<<< HEAD` and `=======` has the code from master, while the other side has your version of the code. You'll need to decide how to deal with the conflict. You may want to keep your changes, keep the changes on master, or combine the two. 117 | 118 | Generally, resolving the conflict consists of two steps: First, fix the particular conflict. Edit the file to make the changes you want and remove the `<<<<<<<`, `=======`, and `>>>>>>>` lines in the process. Second, check the surrounding code. If there was a conflict, it's likely there are some logical errors lying around too! 119 | 120 | Once you're all done fixing the conflicts, you need to stage the files that had conflicts in them via git add. Afterwards, run `git rebase --continue` to let Git know that you've resolved the conflicts and it should finish the rebase. 121 | 122 | Once the rebase has succeeded, you'll want to update the associated branch on your fork with `git push --force-with-lease`. 123 | 124 | ## Advanced rebasing 125 | 126 | If your branch contains multiple consecutive rewrites of the same code, or if the rebase conflicts are extremely severe, you can use `git rebase --interactive master` to gain more control over the process. This allows you to choose to skip commits, edit the commits that you do not skip, change the order in which they are applied, or "squash" them into each other. 127 | 128 | Alternatively, you can sacrifice the commit history like this: 129 | 130 | ```bash 131 | # squash all the changes into one commit so you only have to worry about conflicts once 132 | git rebase -i $(git merge-base master HEAD) # and squash all changes along the way 133 | git rebase master 134 | # fix all merge conflicts 135 | git rebase --continue 136 | ``` 137 | 138 | Squashing commits into each other causes them to be merged into a single commit. Both the upside and downside of this is that it simplifies the history. On the one hand, you lose track of the steps in which changes were made, but the history becomes easier to work with. 139 | 140 | You also may want to squash together just the last few commits, possibly because they only represent "fixups" and not real changes. For example, `git rebase --interactive HEAD~2` allows you to edit the two commits only. 141 | 142 | ## Setting pre-commit 143 | 144 | We use [pre-commit](https://pre-commit.com/) to check the code style before committing. To install pre-commit, run: 145 | 146 | ```bash 147 | # Using pip: 148 | pip install pre-commit 149 | 150 | # Using homebrew: 151 | brew install pre-commit 152 | ``` 153 | 154 | After the installation is successful, run ```pre-commit install``` in the project root directory to enable git's pre-commit. -------------------------------------------------------------------------------- /src/get-started/debug-and-profile.md: -------------------------------------------------------------------------------- 1 | # Debug and profile 2 | 3 | In this section, you will learn: 4 | 5 | * How to debug TiDB 6 | * How to pause the execution at any line of code to inspect values and stacks 7 | * How to profile TiDB to catch a performance bottleneck 8 | 9 | ## Use delve for debugging 10 | 11 | [Delve](https://github.com/go-delve/delve) is a debugger for the Go programming language. It provides a command-line debugging experience similar to the GNU Project debugger (GDB), but it is much more Go native than GDB itself. 12 | 13 | ### Install delve 14 | 15 | To install delve, see the [installation guide](https://github.com/go-delve/delve/tree/master/Documentation/installation). After the installation, depending on how you set your environment variables, you will have an executable file named `dlv` in either `$GOPATH/bin` or `$HOME/go/bin`. You can then run the following command to verify the installation: 16 | 17 | ``` 18 | $ dlv version 19 | Delve Debugger 20 | Version: 1.5.0 21 | Build: $Id: ca5318932770ca063fc9885b4764c30bfaf8a199 $ 22 | ``` 23 | 24 | ### Attach delve to a running TiDB process 25 | 26 | Once you get the TiDB server running, you can attach the delve debugger. 27 | 28 | For example, you can build and run a standalone TiDB server by running the following commands in the root directory of the source code: 29 | 30 | ```bash 31 | make server 32 | ./bin/tidb-server 33 | ``` 34 | 35 | You can then start a new shell and use `ps` or `pgrep` to find the PID of the tidb server process you just started: 36 | 37 | ```bash 38 | pgrep tidb-server 39 | # OUTPUT: 40 | # 1394942 41 | ``` 42 | 43 | If the output lists multiple PIDs, it indicates that you might have multiple TiDB servers running at the same time. To determine the PID of the tidb server you are planning to debug, you can use commands such as `ps $PID`, where `$PID` is the PID you are trying to know more about: 44 | 45 | ```bash 46 | ps 1394942 47 | # OUTPUT: 48 | # PID TTY STAT TIME COMMAND 49 | # 1394942 pts/11 SNl 0:02 ./bin/tidb-server 50 | ``` 51 | 52 | Once you get the PID, you can attach delve to it by running the following command: 53 | 54 | ```bash 55 | dlv attach 1394942 56 | ``` 57 | 58 | You might get error messages of the kernel security setting as follows: 59 | 60 | ``` 61 | Could not attach to pid 1394942: this could be caused by a kernel security setting, try writing "0" to /proc/sys/kernel/yama/ptrace_scope 62 | ``` 63 | 64 | To resolve the error, follow the instructions provided in the error message and execute the following command as the root user to override the kernel security setting: 65 | 66 | ```bash 67 | echo 0 > /proc/sys/kernel/yama/ptrace_scope 68 | ``` 69 | 70 | Then retry attaching delve onto the PID, and it should work. 71 | 72 | If you've worked with GDB, the delve debugging interface will look familiar to you. It is an interactive dialogue that allows you to interact with the execution of the tidb server attached on. To learn more about delve, you can type help into the dialogue and read the `help` messages. 73 | 74 | ### Use delve for debugging 75 | 76 | After attaching delve to the running TiDB server process, you can now set breakpoints. TiDB server will pause execution at the breakpoints you specify. 77 | 78 | To create a breakpoint, you can write: 79 | 80 | ``` 81 | break [name] 82 | ``` 83 | 84 | where `[name]` is the name for the breakpoint, and `` is the position of a line of code in the source code. Note the name is optional. 85 | 86 | For example, the following command creates a breakpoint at the `Next` function of `HashJoinExec`. (The line number can be subject to change due to the modification of the source code). 87 | 88 | ```bash 89 | dlv debug tidb-server/main.go 90 | # OUTPUT: 91 | # Type 'help' for list of commands. 92 | # (dlv) break executor/join.go:653 93 | # Breakpoint 1 (enabled) set at 0x36752d8 for github.com/pingcap/tidb/executor.(*HashJoinExec).Next() ./executor/join.go:653 94 | # (dlv) 95 | ``` 96 | 97 | Once the execution is paused, the context of the execution is fully preserved. You are free to inspect the values of different variables, print the calling stack, and even jump between different goroutines. Once you finish the inspection, you can resume the execution by stepping into the next line of code or continue the execution until the next breakpoint is encountered. 98 | 99 | Typically, when you use a debugger, you need to take the following steps: 100 | 101 | 1. Locate the code and set a breakpoint. 102 | 2. Prepare data so that the execution will get through the breakpoint, and pause at the specified breakpoint as expected. 103 | 3. Inspect values and follow the execution step by step. 104 | 105 | ### Using delve to debug a test case 106 | 107 | If a test case fails, you can also use delve to debug it. Get the name of the test case, go to the corresponding package directory, and then run the following command to start a debugging session that will stop at the entry of the test: 108 | 109 | ``` 110 | dlv test -- -run TestName 111 | ``` 112 | 113 | ### Understand how TiDB works through debugging 114 | 115 | Besides debugging problems, you can also use the debugger to understand how TiDB works through tracking the execution step by step. 116 | 117 | To understand TiDB internals, it's critical that you understand certain functions. To better understand how TiDB works, you can pause the execution of these TiDB functions, and then run TiDB step by step. 118 | 119 | For example: 120 | 121 | 1. [`executor/compiler.go:Compile`](https://github.com/pingcap/tidb/blob/5c95062cc34d6d37e2e921f9bddba6205b43ee3a/executor/compiler.go#L48) is where each SQL statement is compiled and optimized. 122 | 2. [`planner/planner.go:Optimize`](https://github.com/pingcap/tidb/blob/5c95062cc34d6d37e2e921f9bddba6205b43ee3a/planner/optimize.go#L80) is where the SQL optimization starts. 123 | 3. [`executor/adapter.go:ExecStmt.Exec`](https://github.com/pingcap/tidb/blob/5c95062cc34d6d37e2e921f9bddba6205b43ee3a/executor/adapter.go#L312) is where the SQL plan turns into executor and where the SQL execution starts. 124 | 4. Each executor's `Open`, `Next`, and `Close` function marks the volcano-style execution logic. 125 | 126 | When you are reading the TiDB source code, you are strongly encouraged to set a breakpoint and use the debugger to trace the execution whenever you are confused or uncertain about the code. 127 | 128 | ## Using `pprof` for profiling 129 | 130 | For any database system, performance is always important. If you want to know where the performance bottleneck is, you can use a powerful Go profiling tool called `pprof`. 131 | 132 | ### Gather runtime profiling information through HTTP end points 133 | 134 | Usually, when TiDB server is running, it exposes a profiling end point through HTTP at `http://127.0.0.1:10080/debug/pprof/profile`. You can get the profile result by running the following commands: 135 | 136 | ```bash 137 | curl -G "127.0.0.1:10080/debug/pprof/profile?seconds=45" > profile.profile 138 | go tool pprof -http 127.0.0.1:4001 profile.profile 139 | ``` 140 | 141 | The commands capture the profiling information for 45 seconds, and then provide a web view of the profiling result at `127.0.0.1:4001`. This view contains a [flame graph](http://www.brendangregg.com/flamegraphs.html) of the execution and more views that can help you diagnose the performance bottleneck. 142 | 143 | You can also gather other runtime information through this end point. For example: 144 | 145 | * Goroutine: 146 | 147 | ```bash 148 | curl -G "127.0.0.1:10080/debug/pprof/goroutine" > goroutine.profile 149 | ``` 150 | 151 | * Trace: 152 | 153 | ```bash 154 | curl -G "127.0.0.1:10080/debug/pprof/trace?seconds=3" > trace.profile 155 | go tool trace -http 127.0.0.1:4001 trace.profile 156 | ``` 157 | 158 | * Heap: 159 | 160 | ```bash 161 | curl -G "127.0.0.1:10080/debug/pprof/heap" > heap.profile 162 | go tool pprof -http 127.0.0.1:4001 heap.profile 163 | ``` 164 | 165 | To learn how the runtime information is analyzed, see Go's [diagnostics document](https://golang.org/doc/diagnostics). 166 | 167 | ### Profiling during benchmarking 168 | 169 | When you are proposing a performance-related feature for TiDB, we recommend that you also include a benchmark result as proof of the performance gain or to show that your code won't introduce any performance regression. In this case, you need to write your own benchmark test like in `executor/benchmark.go`. 170 | 171 | For example, if you want to benchmark the window functions, because `BenchmarkWindow` are already in the benchmark tests, you can run the following commands to get the benchmark result: 172 | 173 | ```bash 174 | cd executor 175 | go test -bench BenchmarkWindow -run BenchmarkWindow -benchmem 176 | ``` 177 | 178 | If you find any performance regression, and you want to know the cause of it, you could use a command like the following: 179 | 180 | ```bash 181 | go test -bench BenchmarkWindow -run BenchmarkWindow -benchmem -memprofile memprofile.out -cpuprofile profile.out 182 | ``` 183 | 184 | Then, you can use the steps described above to generate and analyze the profiling information. 185 | -------------------------------------------------------------------------------- /src/get-started/install-golang.md: -------------------------------------------------------------------------------- 1 | # Install Golang 2 | 3 | To build TiDB from source code, you need to install Go in your development environment first. If Go is not installed yet, you can follow the instructions in this document for installation. 4 | 5 | ## Install Go 6 | 7 | TiDB periodically upgrades its Go version to keep up with Golang. Currently, upgrade plans are announced on [TiDB Internals forum](https://internals.tidb.io/tags/c/general/announcement). 8 | 9 | To get the right version of Go, take a look at the [`go.mod` file in TiDB's repository](https://github.com/pingcap/tidb/blob/master/go.mod). You should see that there is a line like `go 1.21` (the number may be different) in the first few lines of this file. You can also run the following command to get the Go version: 10 | 11 | ```bash 12 | curl -s -S -L https://github.com/pingcap/tidb/blob/master/go.mod | grep -Eo "\"go [[:digit:]]+(\.[[:digit:]]+)+\"" 13 | ``` 14 | 15 | Now that you've got the version number, go to [Go's download page](https://golang.org/dl/), choose the corresponding version, and then follow the [installation instructions](https://golang.org/doc/install). 16 | 17 | ## Manage the Go toolchain using gvm 18 | 19 | If you are using Linux or MacOS, you can manage Go versions with [Go Version Manager (gvm)](https://github.com/moovweb/gvm) easily. 20 | 21 | To install gvm, run the following command: 22 | 23 | ```bash 24 | curl -s -S -L https://raw.githubusercontent.com/moovweb/gvm/master/binscripts/gvm-installer | sh 25 | ``` 26 | 27 | Once you have gvm installed, you can use it to manage multiple different Go compilers with different versions. Let's install the corresponding Go version and set it as default: 28 | 29 | ```bash 30 | TIDB_GOVERSION=$(curl -s -S -L https://github.com/pingcap/tidb/blob/master/go.mod | grep -Eo "\"go [[:digit:]]+(\.[[:digit:]]+)+\"" | grep -Eo "[[:digit:]]+\.[[:digit:]]+(\.[[:digit:]]+)?") 31 | gvm install go${TIDB_GOVERSION} 32 | gvm use go${TIDB_GOVERSION} --default 33 | ``` 34 | 35 | Now, you can type `go version` in the shell to verify the installation: 36 | 37 | ```bash 38 | go version 39 | # Note: In your case, the version number might not be '1.21', it should be the 40 | # same as the value of ${TIDB_GOVERSION}. 41 | # 42 | # OUTPUT: 43 | # go version go1.21 linux/amd64 44 | ``` 45 | 46 | In the next chapter, you will learn how to obtain the TiDB source code and how to build it. 47 | 48 | If you encounter any problems during your journey, do not hesitate to reach out on the [TiDB Internals forum](https://internals.tidb.io/). 49 | -------------------------------------------------------------------------------- /src/get-started/introduction.md: -------------------------------------------------------------------------------- 1 | # Get Started 2 | 3 | Let's start your TiDB journey! There's a lot to learn, but every journey starts somewhere. In this chapter, we'll discuss: 4 | 5 | * [Install Golang](install-golang.md) 6 | * [Get the code, build and run](build-tidb-from-source.md) 7 | * [Setup an IDE](setup-an-ide.md) 8 | * [Write and run unit tests](write-and-run-unit-tests.md) 9 | * [Debug and profile](debug-and-profile.md) 10 | * [Run and debug integration tests](run-and-debug-integration-tests.md) 11 | * [Commit code and submit a pull request](commit-code-and-submit-a-pull-request.md) 12 | -------------------------------------------------------------------------------- /src/get-started/run-and-debug-integration-tests.md: -------------------------------------------------------------------------------- 1 | # Run and debug integration tests 2 | 3 | TiDB now has some integration tests inside the folder `tests/integrationtest`. Those tests are simply organized as SQL files. The test inputs are put in the folder `t/` and the pre-stored result files are in the folder `r/`. 4 | 5 | # How to run integration tests 6 | 7 | Unlike the production environment, we enable extra assertion in integration tests to do some strict checking. You'll need to run `make server_check` instead of `make`/`make server` to build the TiDB binary. The main difference between them is that the `make server_check` adds two new go build tags named `enableassert` and `deadlock`. 8 | 9 | Once you have built a TiDB binary. You can go to the folder `tests/integrationtest` and run `./run-tests.sh -s ../../bin/tidb-server` to run all the integration tests. 10 | 11 | And you can use `-t` to specify a test file to test. If you run `./run-tests.sh -s ../../bin/tidb-server -t executor/simple`, the scripts will run the tests inside the file `t/executor/simple.test`.(It's corresponding result file is `r/executor/simple.result`) 12 | 13 | If you find out that the changes of tests are expected, you can pass `-r` to re-generate the result file. If you run `./run-tests.sh -s ../../bin/tidb-server -r executor/simple`, it will overwirte the `r/executor/simple.test` by the new binary. 14 | 15 | # How to debug integration tests 16 | 17 | Sometimes you will need to debug the test when you find out that some changes are not expected. You will need to do some preparation before attach to `dlv` if you prefer to use `dlv` to do live debugging. 18 | 19 | We use some build tags to inject some check in the testing environment. Those tags are `intest,deadlock,integrationtest`, so you need to pass `--tags intest,deadlock,integrationtest` when you're building the binary. Or you can directly run `make server_check` to build the needed binary and then use `dlv` to do the debugging. 20 | 21 | After you build the correct binary, you can follow the descriptions in [Use delve for debugging](./debug-and-profile.md#use-delve-for-debugging) to debug TiDB. 22 | 23 | ## How to use IDE to debug 24 | 25 | If you're using IDEs, you'll need to create a special build file for it. 26 | 27 | ### Visual Studio Code 28 | 29 | Find the `Run and Debug` in the left sidebar. If you haven't created any config, click the `create a launch.json file.` 30 | 31 | ![VS Code create debug file](../img/vscode-go-to-edit.png) 32 | 33 | And then you are able to create different run&debug configurations. 34 | 35 | ![VS Code config sample](../img/vscode-debug-config.png) 36 | 37 | The above image shows some samples of the configuration. And the `tidb-on-2432-integration-test` is the one enabling build tags. 38 | 39 | Then you just need to click the run button, VS Code will build the binary and then use `dlv` to attach on it. 40 | 41 | ![VS Code begin debugging](../img/vscode-debug-binary.png) 42 | 43 | You can find more about VS Code's debugging on its official documents. 44 | -------------------------------------------------------------------------------- /src/get-started/setup-an-ide.md: -------------------------------------------------------------------------------- 1 | # Setup an IDE 2 | 3 | Using an IDE is recommended as it makes it a lot easier to work with the TiDB code, for example to see the fields of a `struct`. However it is not required to use a specific IDE or editor. You can use the IDE or editor of your choice. 4 | 5 | ## GoLand 6 | 7 | You can use [GoLand](https://www.jetbrains.com/go/) to easily run or debug TiDB in many situations. 8 | 9 | ### Prerequisites 10 | 11 | * `go`: TiDB is a Go project. Therefore, you need a working Go environment to build it. See the previous [Install Golang](install-golang.md) section to prepare the environment. 12 | * TiDB source code: See the previous [Get the code, build and run](build-tidb-from-source.md) section to get the source code. 13 | 14 | ### Download GoLand 15 | 16 | Download GoLand from [here](https://www.jetbrains.com/go/download) and install it. 17 | 18 | ### Open the TiDB source code in GoLand 19 | 20 | Follow the [instructions](https://www.jetbrains.com/help/go/quick-start-guide-goland.html#open-project) and open the TiDB source code in GoLand. 21 | 22 | ![Open TiDB source code in GoLand](../img/open-tidb-in-goland.png) 23 | 24 | ### Populate run configurations 25 | 26 | Under the root directory of the TiDB source code, execute the following commands to add config files: 27 | 28 | ```bash 29 | mkdir -p .idea/runConfigurations/ && cd .idea/runConfigurations/ 30 | 31 | cat < unistore_4000.xml 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | EOF 44 | 45 | cat < playground_attach_4001.xml 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | EOF 59 | 60 | cat < unit_test.xml 61 | 62 | 63 | 64 | 65 | 66 | 67 | 68 | 69 | 70 | 71 | 72 | 73 | 74 | 75 | EOF 76 | ``` 77 | 78 | Now, confirm there are three config files: 79 | 80 | ```bash 81 | ls 82 | # OUTPUT: 83 | # playground_attach_4001.xml 84 | # unistore_4000.xml 85 | # unit_test.xml 86 | ``` 87 | 88 | ### Run or debug 89 | 90 | Now you can see the run/debug configs right upper the window. 91 | 92 | ![Run Configs](../img/run-configs.png) 93 | 94 | The first config is `unistore 4000`, which enables you to run/debug TiDB independently without TiKV, PD, and TiFlash. 95 | 96 | ![unistore config](../img/unistore-config.png) 97 | 98 | ![unistore output](../img/unistore-output.png) 99 | 100 | The second config is `playground attach 4001`, which enables you to run/debug TiDB by attaching to an existing cluster; for example, a cluster deployed with [`tiup playground`](https://docs.pingcap.com/tidb/stable/tiup-playground). 101 | 102 | After the server process starts, you can connect to the origin TiDB by port 4000, or connect to your TiDB by port 4001 at the same time. 103 | 104 | ![playground attach config](../img/playground-attach-config.png) 105 | ![playground attach debug](../img/playground-attach-debug.png) 106 | 107 | The third config is `unit test`, which enables you to run/debug unit tests. You may modify the `Directory` and `Pattern` to run other tests. 108 | 109 | ![unit test config](../img/unit-test-config.png) 110 | ![unit test output](../img/unit-test-output.png) 111 | 112 | If you encounter any problems during your journey, do not hesitate to reach out on the [TiDB Internals forum](https://internals.tidb.io/). 113 | 114 | ## Visual Studio Code 115 | 116 | VS Code is a generic IDE that has good extensions for working with Go and TiDB. 117 | 118 | ![VS Code with TiDE](../img/vscode_tide.png) 119 | 120 | ### Prerequisites 121 | 122 | * `go`: TiDB is a Go project thus its building requires a working `go` environment. See the previous [Install Golang](install-golang.md) section to prepare the environment. 123 | * TiDB source code: See the previous [Get the code, build and run](build-tidb-from-source.md) section to get the source code. 124 | 125 | ### Download VS Code 126 | 127 | Download VS Code from [here](https://code.visualstudio.com/Download) and install it. 128 | 129 | Now install these extensions: 130 | 131 | * [Go](https://marketplace.visualstudio.com/items?itemName=golang.Go) 132 | * [TiDE](https://marketplace.visualstudio.com/items?itemName=dragonly.ticode) 133 | * [GitHub Pull Requests and Issues](https://marketplace.visualstudio.com/items?itemName=github.vscode-pull-request-github) 134 | 135 | ### Work with TiDB code in VS Code 136 | 137 | Open the folder containing TiDB code via `File→Open Folder`. See the [VS Code docs](https://code.visualstudio.com/docs) for how to edit and commit code. 138 | 139 | There is [detailed guide](https://github.com/tidb-incubator/tide/blob/HEAD/doc/guide.md) explaining how to use the TiDE extension. 140 | 141 | ### Populate run configurations 142 | 143 | Under the root directory of the TiDB source code, execute the following commands to add config files: 144 | 145 | ``` 146 | mkdir -p .vscode 147 | 148 | echo "{ 149 | \"go.testTags\": \"intest,deadlock\" 150 | }" > .vscode/settings.json 151 | ``` 152 | -------------------------------------------------------------------------------- /src/img/cbo-example-logicalPlan.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pingcap/tidb-dev-guide/1885826db775f609450527712e52a5cc95200471/src/img/cbo-example-logicalPlan.png -------------------------------------------------------------------------------- /src/img/cbo-explain-getBestTask.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pingcap/tidb-dev-guide/1885826db775f609450527712e52a5cc95200471/src/img/cbo-explain-getBestTask.png -------------------------------------------------------------------------------- /src/img/ddl-owner-detail-flow-chart.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pingcap/tidb-dev-guide/1885826db775f609450527712e52a5cc95200471/src/img/ddl-owner-detail-flow-chart.png -------------------------------------------------------------------------------- /src/img/ddl-structure-flow-chart.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pingcap/tidb-dev-guide/1885826db775f609450527712e52a5cc95200471/src/img/ddl-structure-flow-chart.png -------------------------------------------------------------------------------- /src/img/dml-contention.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pingcap/tidb-dev-guide/1885826db775f609450527712e52a5cc95200471/src/img/dml-contention.png -------------------------------------------------------------------------------- /src/img/dql-frame-diagram.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pingcap/tidb-dev-guide/1885826db775f609450527712e52a5cc95200471/src/img/dql-frame-diagram.png -------------------------------------------------------------------------------- /src/img/dql-volcano.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pingcap/tidb-dev-guide/1885826db775f609450527712e52a5cc95200471/src/img/dql-volcano.png -------------------------------------------------------------------------------- /src/img/gc-leader-election.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pingcap/tidb-dev-guide/1885826db775f609450527712e52a5cc95200471/src/img/gc-leader-election.png -------------------------------------------------------------------------------- /src/img/goland-run-tests.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pingcap/tidb-dev-guide/1885826db775f609450527712e52a5cc95200471/src/img/goland-run-tests.png -------------------------------------------------------------------------------- /src/img/open-tidb-in-goland.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pingcap/tidb-dev-guide/1885826db775f609450527712e52a5cc95200471/src/img/open-tidb-in-goland.png -------------------------------------------------------------------------------- /src/img/plan-cache-execute.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pingcap/tidb-dev-guide/1885826db775f609450527712e52a5cc95200471/src/img/plan-cache-execute.png -------------------------------------------------------------------------------- /src/img/plan-cache-parameter.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pingcap/tidb-dev-guide/1885826db775f609450527712e52a5cc95200471/src/img/plan-cache-parameter.png -------------------------------------------------------------------------------- /src/img/plan-cache-rebuilding.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pingcap/tidb-dev-guide/1885826db775f609450527712e52a5cc95200471/src/img/plan-cache-rebuilding.png -------------------------------------------------------------------------------- /src/img/playground-attach-config.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pingcap/tidb-dev-guide/1885826db775f609450527712e52a5cc95200471/src/img/playground-attach-config.png -------------------------------------------------------------------------------- /src/img/playground-attach-debug.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pingcap/tidb-dev-guide/1885826db775f609450527712e52a5cc95200471/src/img/playground-attach-debug.png -------------------------------------------------------------------------------- /src/img/raftStore.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pingcap/tidb-dev-guide/1885826db775f609450527712e52a5cc95200471/src/img/raftStore.png -------------------------------------------------------------------------------- /src/img/run-configs.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pingcap/tidb-dev-guide/1885826db775f609450527712e52a5cc95200471/src/img/run-configs.png -------------------------------------------------------------------------------- /src/img/sql-layer-architecture.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pingcap/tidb-dev-guide/1885826db775f609450527712e52a5cc95200471/src/img/sql-layer-architecture.png -------------------------------------------------------------------------------- /src/img/stats-cmsketch.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pingcap/tidb-dev-guide/1885826db775f609450527712e52a5cc95200471/src/img/stats-cmsketch.png -------------------------------------------------------------------------------- /src/img/stats-histogram.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pingcap/tidb-dev-guide/1885826db775f609450527712e52a5cc95200471/src/img/stats-histogram.png -------------------------------------------------------------------------------- /src/img/tidb-architecture.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pingcap/tidb-dev-guide/1885826db775f609450527712e52a5cc95200471/src/img/tidb-architecture.png -------------------------------------------------------------------------------- /src/img/transaction-architecture.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pingcap/tidb-dev-guide/1885826db775f609450527712e52a5cc95200471/src/img/transaction-architecture.png -------------------------------------------------------------------------------- /src/img/transaction_scheduler.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pingcap/tidb-dev-guide/1885826db775f609450527712e52a5cc95200471/src/img/transaction_scheduler.png -------------------------------------------------------------------------------- /src/img/unistore-config.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pingcap/tidb-dev-guide/1885826db775f609450527712e52a5cc95200471/src/img/unistore-config.png -------------------------------------------------------------------------------- /src/img/unistore-output.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pingcap/tidb-dev-guide/1885826db775f609450527712e52a5cc95200471/src/img/unistore-output.png -------------------------------------------------------------------------------- /src/img/unit-test-config.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pingcap/tidb-dev-guide/1885826db775f609450527712e52a5cc95200471/src/img/unit-test-config.png -------------------------------------------------------------------------------- /src/img/unit-test-output.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pingcap/tidb-dev-guide/1885826db775f609450527712e52a5cc95200471/src/img/unit-test-output.png -------------------------------------------------------------------------------- /src/img/vscode-debug-binary.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pingcap/tidb-dev-guide/1885826db775f609450527712e52a5cc95200471/src/img/vscode-debug-binary.png -------------------------------------------------------------------------------- /src/img/vscode-debug-config.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pingcap/tidb-dev-guide/1885826db775f609450527712e52a5cc95200471/src/img/vscode-debug-config.png -------------------------------------------------------------------------------- /src/img/vscode-go-to-edit.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pingcap/tidb-dev-guide/1885826db775f609450527712e52a5cc95200471/src/img/vscode-go-to-edit.png -------------------------------------------------------------------------------- /src/img/vscode-run-tests.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pingcap/tidb-dev-guide/1885826db775f609450527712e52a5cc95200471/src/img/vscode-run-tests.png -------------------------------------------------------------------------------- /src/img/vscode_tide.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pingcap/tidb-dev-guide/1885826db775f609450527712e52a5cc95200471/src/img/vscode_tide.png -------------------------------------------------------------------------------- /src/project-management/introduction.md: -------------------------------------------------------------------------------- 1 | # Project Management 2 | 3 | Practices for managing the TiDB project: 4 | 5 | * [Release Train Model](release-train-model.md) 6 | * [TiDB Versioning](tidb-versioning.md) 7 | -------------------------------------------------------------------------------- /src/project-management/release-train-model.md: -------------------------------------------------------------------------------- 1 | # Release Train Model 2 | 3 | ## What is the release train model? 4 | 5 | Before introducing the concept of the release train model, let us take a review of the delivery mode of TiDB in the past. 6 | 7 | In releases earlier than v5.0, the release frequency of TiDB major versions was a year or half a year, which is quite a long development cycle. The long development cycle has both benefits and drawbacks as follows: 8 | 9 | - Benefits: the longer a development cycle is, the more features one release can deliver. 10 | - Drawbacks: the longer a development cycle is, the more difficulties we have to coordinate regression and acceptance tests, and the more possibly a delay happens. Also, if new feature requests are received during the long development cycle, these new features are added to the development backlog after the start of the development cycle. In this case, development tasks are hardly converged before the release date. 11 | 12 | Starting from v5.0, TiDB adopts the release train model, which is a product development model for requirements gathering, analysis, decision making, release, and issue feedback. 13 | 14 | Just like a train delivering goods, decisions need to be made about the priorities of the goods, destination, arrival time, which train to load on, which carriage, etc., before the train departs. 15 | 16 | The benefits of moving to the release train model are as follows: 17 | 18 | 1. A shorter feedback cycle: users can benefit from features shipped faster. 19 | 2. Easier predictability for contributors and users: 20 | 1. Developers and reviewers can decide in advance the target release to deliver specific features. 21 | 2. If a feature misses a release train, we have a good idea of when the feature will show up later. 22 | 3. Users know when to expect their features. 23 | 3. Transparency. There will be a published cut-off date (AKA code freeze) for the release and people will know about the date in advance. Hopefully this will remove the contention around which features will be included in the release. 24 | 4. Quality. we've seen issues pop up in release candidates due to last-minute features that didn't have proper time to bake in. More time between code freeze and release will let us test more, document more and resolve more issues. 25 | 5. Project visibility and activity. Having frequent releases improves our visibility and gives the community more opportunities to talk about TiDB. 26 | 27 | Because nothing is ever perfect, the release train model has some downsides as well: 28 | 29 | 1. Most notably, for features that miss the code-freeze date for a release, we have to wait for a few months to catch the next release train. Most features will reach users faster as per benefit #1, but it is possible that a few features missing the code-freeze date might lose out. 30 | 2. With the frequent releases, users need to figure out which release to use. Also, having frequent new releases to upgrade may be a bit confusing. 31 | 3. Frequent releases means more branches. To fix a bug of an old release, we need to work on more old branches. 32 | 33 | We decided to experiment with release train model and see if the benefits for us as a community exceed the drawbacks. 34 | 35 | ## How will TiDB development process look like? 36 | 37 | At this stage we are planning to make a release every two months. 38 | 39 | Thus, a typical development cycle takes two months, which we call a sprint. For example, the development cycle of v5.2 is from the end of June to the end of August and is called Sprint 4. 40 | 41 | Two weeks before the release date, the release manager will create a branch for the new release based on the master branch, publish a list of features to be included in the release, and also announce the code-freeze, after which only fixes for blocking bugs can be merged. 42 | 43 | For the release train model, we strictly ensure that a release happens on the planned date. For example, we decide to deliver the v5.2 release by the end of August so we will stick to it. If any features cannot be completed by the code-freeze date, we will drop them and avoid taking them into the new release branch. In this case, the development in the master branch can still work as usual and those features will be moved to the following release. 44 | 45 | Ideally, we would have started stabilization once we create the new release branch. After the code-freeze date, only pull requests of blocker bugs can be merged to the new release branch. In a rare scenario, it is possible that few features pass the code freeze bar but still fail to be completed on time. Such features will also be dropped from the release train in the end to meet the release deadline. 46 | 47 | Developers who want to contribute features to TiDB could follow the procedure described in [Make a Proposal](../contribute-to-tidb/make-a-proposal.md). Once all the requirements are met and all the codes are merged into the master branch, the feature will be boxed into the nearest release. 48 | 49 | Except for feature releases, there also exists patch releases. Patch releases are scheduled when needed, there is no fixed calendar for such releases. When a patch release is scheduled, there are two rounds of triage. A bug fix could only be boxed into a release when it occurs before triage. Get more information about TiDB releases and versions in [TiDB versioning](tidb-versioning.md). 50 | 51 | ## What happens if features are not completed? 52 | 53 | Different features have different complexities. Some features can be implemented within a single release while some features span multiple releases. There are two conventional development strategies: 54 | 55 | 1. Ensure that each feature is split into testable units and only testable units get merged. This means that a good set of unit tests and system tests are written for sub-tasks before they are merged. This approach ensures that the master branch is in a relatively stable state and can be released at any time. 56 | 57 | 2. Use feature branches. For a specific feature, the feature developers create a branch from the master branch and ensure that the branch is in sync with the master branch from time to time. Only when the feature developers and reviewers have a high level of confidence in the feature stability, the feature can be merged into master. This approach brings the additional overhead of branching and performing merges from time to time. 58 | 59 | With the release train model, to ensure that ongoing features do not affect the stability of the release, TiDB chooses feature branches strategy. 60 | 61 | ## Current Maintained Releases 62 | 63 | | version | branch | status | triage label | latest release | issue 64 | |:--------------------|:-------------------|:------------------|:------------------------|:-------------------------|:------------------------------------------------------- 65 | | v7.3 | release-7.3 | DMR | affects-7.3 | v7.3.0 | 66 | | v7.2 | release-7.2 | DMR | affects-7.2 | v7.2.0 | 67 | | v7.1 | release-7.1 | LTS | affects-7.1 | v7.1.1 | 68 | | v7.0 | release-7.0 | DMR | affects-7.0 | v7.0.0-DMR | 69 | | v6.6 | release-6.6 | DMR | affects-6.6 | v6.6.0-DMR | 70 | | v6.5 | release-6.5 | LTS | affects-6.5 | v6.5.4 | 71 | | v6.4 | release-6.4 | DMR | affects-6.4 | v6.4.0-DMR | 72 | | v6.3 | release-6.3 | DMR | affects-6.3 | v6.3.0-DMR | 73 | | v6.2 | release-6.2 | DMR | affects-6.2 | v6.2.0-DMR | 74 | | v6.1 | release-6.1 | LTS | affects-6.1 | v6.1.7 | 75 | | v6.0 | release-6.0 | DMR | affects-6.0 | v6.0.0-DMR | 76 | | v5.0 | release-5.0 | LTS | affects-5.0 | v5.0.6 | 77 | | v5.1 | release-5.1 | LTS | affects-5.1 | v5.1.5 | 78 | | v5.2 | release-5.2 | LTS | affects-5.2 | v5.2.4 | 79 | | v5.3 | release-5.3 | LTS | affects-5.3 | v5.3.4 | 80 | | v5.4 | release-5.4 | LTS | affects-5.4 | v5.4.3 | 81 | | v4.0 | release-4.0 | LTS | affects-4.0 | v4.0.16 | 82 | 83 | For more versions' information, please check . 84 | -------------------------------------------------------------------------------- /src/project-management/tidb-versioning.md: -------------------------------------------------------------------------------- 1 | # TiDB Versioning 2 | 3 | TiDB versioning has the form `X.Y.Z` where `X.Y` refers to the release series and `Z` refers to the patch number. Starting with TiDB 6.0, TiDB is released as two different release series: 4 | 5 | - LTS(Long-Term Support) Releases 6 | - DMR(Development Milestone) Releases 7 | 8 | ## LTS Releases 9 | 10 | LTS releases are made available approximately every six months. They carry new features and improvements and are recommended to deploy into production environments. There will be patch releases based on the LTS releases in their lifecycle. Example versions: 11 | 12 | - `5.4` 13 | - `6.1` 14 | 15 | Release `4.0` and `5.x` are treated like LTS releases although they are earlier than `6.0`. 16 | 17 | ## DMR Releases 18 | 19 | DMR releases are made available approximately every two months. Every 3rd DMR release turns into a LTS release. Same as LTS releases, a DMR release introduces new features and improvements. But there is no patch releases based on the DMR release. Bugs in the DMR release are going to be fixed in the next DMR/LTS releases. There is a `-DMR` suffix of DMR versioning. Example versions: 20 | 21 | - `6.0.0-DMR` 22 | - `6.2.0-DMR` 23 | 24 | ## Patch Releases 25 | 26 | Patch releases generally include bug fixes for LTS releases. There is no fixed release schedule for patch releases. Example versions: 27 | 28 | - `6.1.1` 29 | - `6.1.2` 30 | 31 | ## Historical Versioning 32 | 33 | There are some other versioning in history which are not used any more. 34 | 35 | ### GA(General Availability) Releases 36 | 37 | Stable release series, released after RC releases. GA releases are recommended for production usage. Example versions: 38 | 39 | - `2.1 GA` 40 | - `5.0 GA` 41 | 42 | ### RC(Release Candidate) Releases 43 | 44 | RC releases introduces new features and improvements and meant for early test. Comparing with Beta releases, RC releases are much more stable and suitable for test, but not suitable for production usage. Example versions: 45 | 46 | - `2.0-RC1` 47 | - `3.0.0-rc.1` 48 | 49 | ### Beta Releases 50 | 51 | Beta releases introduces new features and improvements. Comparing with Alpha releases, Beta releases shall not carry any critical bugs. Early adopters could use Beta releases to try new features. Example versions: 52 | 53 | - `1.1 Beta` 54 | - `4.0.0-beta.1` 55 | 56 | ### Alpha Releases 57 | 58 | The very first releases in a series. Used for fundamental functionality and performance test. Example versions: 59 | 60 | - `1.1 Alpha` 61 | -------------------------------------------------------------------------------- /src/system-tables/introduction.md: -------------------------------------------------------------------------------- 1 | # System tables 2 | -------------------------------------------------------------------------------- /src/system-tables/slow_query.md: -------------------------------------------------------------------------------- 1 | # slow_query 2 | -------------------------------------------------------------------------------- /src/understand-tidb/dml.md: -------------------------------------------------------------------------------- 1 | # DML 2 | 3 | ## Overview 4 | 5 | [DML](https://en.wikipedia.org/wiki/Data_manipulation_language) is a sublanguage of SQL which is used as data manipulation. This document talks about the DML processing in TiDB. 6 | 7 | This document refers to the code of [TiDB v5.2.1](https://github.com/pingcap/tidb/tree/v5.2.1) and [TiKV v5.2.1](https://github.com/tikv/tikv/tree/v5.2.1). 8 | 9 | ## Execution Process 10 | 11 | The [lifecycle](https://pingcap.github.io/tidb-dev-guide/understand-tidb/the-lifecycle-of-a-statement.html) chapter explains how queries are handled in TiDB. Different from DQLs which may write a lot of content to the client and should be processed in a streaming-like way, DMLs only report the result statistics(count of rows affected and inserted), which are handled by [`handleNoDelay`](https://github.com/pingcap/tidb/blob/v5.2.1/executor/adapter.go#L440-L482) function. 12 | 13 | Generally, a DML statement is converted into delta changes in the execution process. When a transaction is committed, the changes will be applied atomically. Without starting a transaction explicitly, it'll be committed automatically. In which, this document focuses on how a DML is executed only. 14 | 15 | Compare with DQLs, DMLs are relatively simple in optimization, it's easy to imagine how the delete, insert, update, and replace statements look like and how they should be executed. There may be some data sources in DMLs, like `insert into table1 select from table2` which will insert the data from table2 into table1, however, you may not care about the data sources too much, since the data can be read by just calling `Next` of reading executors. 16 | 17 | Like DQLs, the physical plans will be built into executors after optimization in the [`build`](https://github.com/pingcap/tidb/blob/v5.2.1/executor/builder.go#L118-L270) function. The replace statement is treated as a kind of insert statement. 18 | 19 | ```rust 20 | func (b *executorBuilder) build(p plannercore.Plan) Executor { 21 | switch v := p.(type) { 22 | case nil: 23 | return nil 24 | ... 25 | case *plannercore.Delete: 26 | return b.buildDelete(v) 27 | ... 28 | case *plannercore.Insert: 29 | return b.buildInsert(v) 30 | ... 31 | case *plannercore.Update: 32 | return b.buildUpdate(v) 33 | ... 34 | } 35 | } 36 | ``` 37 | 38 | After the execution, the input SQL statements will be converted into delta changes and stored in MemDB, we'll talk about it later. 39 | 40 | Like DQLs, DMLs also rely on the schema. When a SQL is compiled, the schema is assigned. Manipulating data should base on the corresponding schema. Tables in the schema offer [`Table`](https://github.com/pingcap/tidb/blob/v5.2.1/table/table.go#L164-L197) interface, which is a medium of data manipulation. 41 | 42 | ## Conflict 43 | 44 | Without conflicts, DMLs are easy, they are converted into delta changes and waiting to be committed during execution. This section will talk about the conflict handle issue. 45 | 46 | ### Optimistic 47 | 48 | TiDB can check the conflict for optimistic transactions whether during execution and commit. The behavior is controlled by [`tidb_constraint_check_in_place`](https://docs.pingcap.com/tidb/stable/system-variables#tidb_constraint_check_in_place). By default, TiDB won't check if there is a conflict when executing optimistic transactions. A conflict means that there is a record whose `commit_ts` is less than the current transaction's `start_ts`, by checking this situation TiDB needs to look up TiKV and see if such a record exists, there the latency is introduced. The code for handling the [row key](https://github.com/pingcap/tidb/blob/v5.2.1/table/tables/tables.go#L781-L792) and [unique index key](https://github.com/pingcap/tidb/blob/v5.2.1/table/tables/index.go#L205-L209) is below. 49 | 50 | ```go 51 | // AddRecord implements table.Table AddRecord interface. 52 | func (t *TableCommon) AddRecord(sctx sessionctx.Context, r []types.Datum, opts ...table.AddRecordOption) (recordID kv.Handle, err error) { 53 | ... 54 | } else if sctx.GetSessionVars().LazyCheckKeyNotExists() { 55 | var v []byte 56 | v, err = txn.GetMemBuffer().Get(ctx, key) 57 | if err != nil { 58 | setPresume = true 59 | } 60 | if err == nil && len(v) == 0 { 61 | err = kv.ErrNotExist 62 | } 63 | } else { 64 | _, err = txn.Get(ctx, key) 65 | } 66 | ... 67 | } 68 | 69 | // Create creates a new entry in the kvIndex data. 70 | // If the index is unique and there is an existing entry with the same key, 71 | // Create will return the existing entry's handle as the first return value, ErrKeyExists as the second return value. 72 | func (c *index) Create(sctx sessionctx.Context, txn kv.Transaction, indexedValues []types.Datum, h kv.Handle, handleRestoreData []types.Datum, opts ...table.CreateIdxOptFunc) (kv.Handle, error) { 73 | ... 74 | } else if sctx.GetSessionVars().LazyCheckKeyNotExists() { 75 | value, err = txn.GetMemBuffer().Get(ctx, key) 76 | } else { 77 | value, err = txn.Get(ctx, key) 78 | } 79 | ... 80 | } 81 | ``` 82 | 83 | Skipping checking the existence of records reduces latency, and if there is such a record, the optimistic transactions will suffer an error when committing. In the prewrite phase, if there is a record whose `commit_ts` is less than self's `start_ts`, an [already-exist error](https://github.com/tikv/tikv/blob/v5.2.1/src/storage/txn/actions/prewrite.rs#L359) will be returned and the current transaction should abort, so there are no correctness issues. 84 | 85 | As for the use case which requires checking if the key exists in execution, just turn `tidb_constraint_check_in_place` on. 86 | 87 | ### Pessimistic 88 | 89 | Since optimistic transactions check if can submit in prewrite phases, in a high contention use case, there may be cascade abort, and the fail rate of the transaction will increase rapidly. Further, aborted transactions need to be cleaned up, as a result, many resources are wasted. 90 | 91 | As the image shows, pessimistic transactions go through a 2PL way. 92 | 93 | ![Comparison between OCC and 2PL](../img/dml-contention.png) 94 | 95 | By any write operations, the corresponding pessimistic locks will be written into TiKV, then other transactions which tend to modify the locked data, a key is locked error will be returned from TiKV, then TiDB will block the statement until the lock is cleared or expired. 96 | 97 | The pessimistic transaction lock key is in two steps, e.g. `UPDATE t SET v = v + 1 WHERE id < 10`: 98 | 99 | - Read the required data from TiKV, it's like `SELECT row_id FROM t WHERE id < 10` got (1, 1), (2, 4). 100 | - Now TiDB knows which key exists, then based on the read data, lock keys. 101 | 102 | However, there is a specific way to add a pessimistic, aka. lock and read in the same time. It's an atomic operation, read keys from TiKV and the result is returned with keys locked. Luckily, this way is not widely used, only in [point_get](https://github.com/pingcap/tidb/blob/v5.2.1/executor/point_get.go#L351-L371) and [batch_point_get](https://github.com/pingcap/tidb/blob/v5.2.1/executor/batch_point_get.go#L516-L537). This operation will lock the not-exist keys, which allows the client to lock keys first and write some values later. TiDB handles this case by adding the KVs to a pessimistic lock cache after pessimistic lock is done. 103 | 104 | 105 | 106 | There is a special read which is different from snapshot read, which read the data from the latest snapshot. We can call it current read or for-update read. In [snapshot isolation](https://en.wikipedia.org/wiki/Snapshot_isolation), all statements in a transaction are executed in the same snapshot, so there are no non-repeatable or non-phantom reads. However, write operations in pessimistic transactions should affect the latest version of data, which means that they use the different snapshot from the read snapshot. 107 | 108 | ## MemDB 109 | 110 | The delta changes are stored in [`MemDB`](https://github.com/tikv/client-go/blob/daddf73a0706d78c9e980c91c97cc9ed100f1919/internal/unionstore/memdb.go#L64-L88) in TiDB until the transaction is committed. `MemDB` is an ordered in-memory storage(implemented in the red-black tree) with the following requirements. 111 | 112 | - Revert changes. 113 | - Flag support. 114 | 115 | Think about an in-transaction statement get failed because of conflict or constraint violation, then an error is reported to the client, this statement should take no effect. However, there may already be some changes are written to the `MemDB` before encountering the error. These changes need to be reverted. In TiDB, [`StmtCommit`](https://github.com/pingcap/tidb/blob/v5.2.1/session/txn.go#L535-L548) handles the statement level commit which will flush the changes of successful statements from the staging buffer into `MemDB`. Here are the key methods of the [`MemBuffer`](https://github.com/pingcap/tidb/blob/v5.2.1/kv/kv.go#L118-L126) interface. 116 | 117 | ```go 118 | // MemBuffer is an in-memory kv collection, can be used to buffer write operations. 119 | type MemBuffer interface { 120 | ... 121 | // Staging create a new staging buffer inside the MemBuffer. 122 | // Subsequent writes will be temporarily stored in this new staging buffer. 123 | // When you think all modifications looks good, you can call `Release` to public all of them to the upper level buffer. 124 | Staging() StagingHandle 125 | // Release publish all modifications in the latest staging buffer to upper level. 126 | Release(StagingHandle) 127 | // Cleanup cleanup the resources referenced by the StagingHandle. 128 | // If the changes are not published by `Release`, they will be discarded. 129 | Cleanup(StagingHandle) 130 | ... 131 | } 132 | ``` 133 | 134 | The [`KeyFlags`](https://github.com/tikv/client-go/blob/daddf73a0706d78c9e980c91c97cc9ed100f1919/kv/keyflags.go#L35-L48) are the metadata of keys, they mark the keys with states. You can learn the meaning from their names, e.g. if `flagPresumeKNE` is set, the key is presumed as not existing in TiKV, which means this is an inserted key, otherwise, it's an updated key. 135 | 136 | ## Summary 137 | 138 | This document talks about DML generally. However, TiDB is a complex system, and DML has correlation with most components, you may be confused about many details with manipulation of data, reading other documents or the source code would make it clear. 139 | -------------------------------------------------------------------------------- /src/understand-tidb/dql.md: -------------------------------------------------------------------------------- 1 | # DQL 2 | 3 | ## Overview 4 | 5 | This chapter describes the execution process of a data query statement in TiDB. Starting from the SQL processing flow, it describes how a SQL statement is sent to TiDB, how TiDB processes it after receiving the SQL statement, and how the execution result is returned. 6 | 7 | ## Execution Process 8 | 9 | Briefly, the execution process of a SQL statement can be divided into three stages: 10 | 11 | 1. Protocol Layer 12 | 13 | Protocol layer is responsible for parsing the network protocol. Its code locates at `server` package, mainly consisting of two parts: one for connection establishing and management, every connection corresponds to one session separately; one for handling the packets read from the connection. 14 | 15 | 2. SQL Layer 16 | 17 | SQL layer is the most complex part in TiDB, handling SQL statement parsing and execution. SQL is a complex language, having various data types and operators, numerous syntax combinations. Besides, TiDB uses a distributed storage engine underneath, so it will encounter many problems standalone storage engines won't. 18 | 19 | 3. KV API Layer 20 | 21 | KV API layer routes requests to the right KV server and passes the results back to SQL layer. It should handle the exceptions happened in this stage. 22 | 23 | A SQL statement goes through the above three stages sequentially, get parsed and transformed, then handled by SQL layer. In SQL layer, query plans are generated and executed, retrieving data from the underneath storage engine. We'll give a detailed introduction to SQL layer. 24 | 25 | ### Protocol Layer 26 | 27 | #### Entry 28 | 29 | The entry of TiDB's SQL layer is in `server/conn.go`. After a connection is established between the client and TiDB, TiDB spawns a goroutine to listen and poll on the port. In [clientConn.Run()](https://github.com/pingcap/tidb/blob/05d2210647d6a1503a8d772477e43b14a024f609/server/conn.go#L911), a loop keeps reading network packets and calls [clientConn.dispatch()](https://github.com/pingcap/tidb/blob/05d2210647d6a1503a8d772477e43b14a024f609/server/conn.go#L1111) to handle them: 30 | 31 | ```go 32 | data, err := cc.readPacket() 33 | if err = cc.dispatch(ctx, data) 34 | ``` 35 | 36 | `dispatch` handles the raw data array. The first byte of the array represents command type. Among the types, `COM_QUERY` represents data query statement. You can refer to [MySQL protocol](https://dev.mysql.com/doc/dev/mysql-server/latest/PAGE_PROTOCOL.html) for more information about the data array. For `COM_QUERY`, its content is SQL statement. [clientConn.handleQuery()](https://github.com/pingcap/tidb/blob/05d2210647d6a1503a8d772477e43b14a024f609/server/conn.go#L1633) handles the SQL statement. It calls [TiDBContext.ExecuteStmt()](https://github.com/pingcap/tidb/blob/05d2210647d6a1503a8d772477e43b14a024f609/server/driver_tidb.go#L217) in `server/driver_tidb.go`: 37 | 38 | ```go 39 | func (tc *TiDBContext) ExecuteStmt(ctx context.Context, stmt ast.StmtNode) (ResultSet, error) { 40 | rs, err := tc.Session.ExecuteStmt(ctx, stmt) 41 | ``` 42 | 43 | [session.ExecuteStmt()](https://github.com/pingcap/tidb/blob/05d2210647d6a1503a8d772477e43b14a024f609/session/session.go#L1620) is the entry of the SQL layer kernel and returns the result of the SQL execution. 44 | 45 | #### Exit 46 | 47 | After a series of operations described above, the execution results will be returned to the client in [COM_QUERY response](https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_com_query_response.html) format by [clientConn.writeResultSet()](https://github.com/pingcap/tidb/blob/05d2210647d6a1503a8d772477e43b14a024f609/server/conn.go#L1943). 48 | 49 | ### SQL Layer 50 | 51 | In SQL layer, there are multiple concepts and interfaces we need to pay close attention to: 52 | 53 | * [Session](https://github.com/pingcap/tidb/blob/05d2210647d6a1503a8d772477e43b14a024f609/session/session.go#L123) 54 | * [RecordSet](https://github.com/pingcap/tidb/blob/05d2210647d6a1503a8d772477e43b14a024f609/util/sqlexec/restricted_sql_executor.go#L133) 55 | * [Plan](https://github.com/pingcap/tidb/blob/05d2210647d6a1503a8d772477e43b14a024f609/planner/core/plan.go#L36) 56 | * [Executor](https://github.com/pingcap/tidb/blob/05d2210647d6a1503a8d772477e43b14a024f609/executor/executor.go#L258) 57 | 58 | #### [Session](session.md) 59 | 60 | The most important function in `Session` is `ExecuteStmt`. It wraps calls to other modules. The SQL execution will respect environment variables in `Session` like `AutoCommit` and timezone. 61 | 62 | #### [Parser](parser.md) 63 | 64 | [Parser](https://github.com/pingcap/tidb/blob/master/pkg/parser/yy_parser.go) consists of [Lexer](https://github.com/pingcap/tidb/blob/master/pkg/parser/lexer.go) and Yacc. It turns the SQL text to AST: 65 | 66 | ```go 67 | p := parserPool.Get().(*parser.Parser) 68 | defer parserPool.Put(p) 69 | p.SetSQLMode(s.sessionVars.SQLMode) 70 | p.SetParserConfig(s.sessionVars.BuildParserConfig()) 71 | tmp, warn, err := p.Parse(sql, charset, collation) 72 | ``` 73 | 74 | In the parsing process, lexer first transforms the SQL text to tokens, and then parser accepts the tokens as inputs and generates appropriate AST nodes. For example, statement `SELECT * FROM t WHERE c > 1;` matches [SelectStmt rule](https://github.com/pingcap/tidb/blob/45457ea8810ca7b835da4ba7f55d0eee02043ac5/parser/parser.y#L3936) finally turns to the structure below: 75 | 76 | ```go 77 | type SelectStmt struct { 78 | dmlNode 79 | // SelectStmtOpts wraps around select hints and switches. 80 | *SelectStmtOpts 81 | // Distinct represents whether the select has distinct option. 82 | Distinct bool 83 | // From is the from clause of the query. 84 | From *TableRefsClause 85 | // Where is the where clause in select statement. 86 | Where ExprNode 87 | // Fields is the select expression list. 88 | Fields *FieldList 89 | // GroupBy is the group by expression list. 90 | GroupBy *GroupByClause 91 | // Having is the having condition. 92 | Having *HavingClause 93 | // WindowSpecs is the window specification list. 94 | WindowSpecs []WindowSpec 95 | // OrderBy is the ordering expression list. 96 | OrderBy *OrderByClause 97 | // Limit is the limit clause. 98 | Limit *Limit 99 | // LockInfo is the lock type 100 | LockInfo *SelectLockInfo 101 | // TableHints represents the table level Optimizer Hint for join type 102 | TableHints []*TableOptimizerHint 103 | // IsInBraces indicates whether it's a stmt in brace. 104 | IsInBraces bool 105 | // WithBeforeBraces indicates whether stmt's with clause is before the brace. 106 | // It's used to distinguish (with xxx select xxx) and with xxx (select xxx) 107 | WithBeforeBraces bool 108 | // QueryBlockOffset indicates the order of this SelectStmt if counted from left to right in the sql text. 109 | QueryBlockOffset int 110 | // SelectIntoOpt is the select-into option. 111 | SelectIntoOpt *SelectIntoOption 112 | // AfterSetOperator indicates the SelectStmt after which type of set operator 113 | AfterSetOperator *SetOprType 114 | // Kind refer to three kind of statement: SelectStmt, TableStmt and ValuesStmt 115 | Kind SelectStmtKind 116 | // Lists is filled only when Kind == SelectStmtKindValues 117 | Lists []*RowExpr 118 | With *WithClause 119 | } 120 | ``` 121 | 122 | `From t` is parsed to `From` field. `WHERE c > 1` is parsed to `Where` field. `*` is parsed to `Fields` field. Most data structures in `ast` package implement `ast.Node` interface. This interface has a `Accept` method, implementing the classic visitor pattern, used by following procedures to traverse the tree. 123 | 124 | #### Compile 125 | 126 | After the AST is generated, it's going to be validated, transformed and optimized in [Compiler.Compile()](https://github.com/pingcap/tidb/blob/05d2210647d6a1503a8d772477e43b14a024f609/executor/compiler.go#L50): 127 | 128 | ```go 129 | compiler := executor.Compiler{Ctx: s} 130 | stmt, err := compiler.Compile(ctx, stmtNode) 131 | ``` 132 | 133 | There are three steps: 134 | 135 | 1. `plan.Preprocess`: do validations and name binding. 136 | 2. `plan.Optimize`: make and optimize query plans, this is the core part. 137 | 3. construct `executor.ExecStmt` structure: [ExecStmt](https://github.com/pingcap/tidb/blob/05d2210647d6a1503a8d772477e43b14a024f609/executor/adapter.go#L186) holds the query plans. It's the foundation for following execution. 138 | 139 | #### [Executor](execution.md) 140 | 141 | While constructing the executor in [ExecStmt.buildExecutor()](https://github.com/pingcap/tidb/blob/05d2210647d6a1503a8d772477e43b14a024f609/executor/adapter.go#L764), query plans are turned to executor. Then the execution engine could perform the query plans via the executor. The generated executor is encapsulated in a `recordSet` structure: 142 | 143 | ```go 144 | return &recordSet{ 145 | executor: e, 146 | stmt: a, 147 | txnStartTS: txnStartTS, 148 | } 149 | ``` 150 | 151 | This structure implements [ast.RecordSet](https://github.com/pingcap/tidb/blob/05d2210647d6a1503a8d772477e43b14a024f609/util/sqlexec/restricted_sql_executor.go#L133) interface. It abstracts the query results and has the following methods: 152 | 153 | ```go 154 | type RecordSet interface { 155 | // Fields gets result fields. 156 | Fields() []*ast.ResultField 157 | // Next reads records into chunk. 158 | Next(ctx context.Context, req *chunk.Chunk) error 159 | // NewChunk create a chunk. 160 | NewChunk() *chunk.Chunk 161 | // Close closes the underlying iterator, call Next after Close will 162 | // restart the iteration. 163 | Close() error 164 | } 165 | ``` 166 | 167 | The functionality of each method is described in the comments. In short, `Fields()` retrieves the type of each column. `Next()` returns a batch of the result. `Close()` closes the result set. 168 | 169 | TiDB's execution engine executes in Volcano model. All the executors constitute an executor tree. Every upper layer gathers results from the lower layer by calling its `Next()` method. Assuming we have a SQL statement `SELECT c1 FROM t WHERE c2 > 1;` and the query plan is full table scanning plus filtering, the executor tree is like: 170 | 171 | ![](../img/dql-volcano.png) 172 | 173 | From the above picture, we can see the data flow between executors. The starting point of a SQL statement execution, also the first `Next()` call is in [the function returning data back to the client](https://github.com/pingcap/tidb/blob/05d2210647d6a1503a8d772477e43b14a024f609/server/conn.go#L2016): 174 | 175 | ```go 176 | err := rs.Next(ctx, req) 177 | ``` 178 | 179 | `rs` is a `RecordSet` instance. Keep calling its `Next` method to get more results to return to the client. 180 | 181 | ## Overall Diagram 182 | 183 | The above SQL query statement execution process can in general be described as the following picture: 184 | 185 | ![](../img/dql-frame-diagram.png) 186 | -------------------------------------------------------------------------------- /src/understand-tidb/execution.md: -------------------------------------------------------------------------------- 1 | # Execution 2 | 3 | The `executor` package contains most of the codes related to execution. The input of the executor is a plan tree of the query returned from the planner, and the output of the executor is the result of the query. The entry function of execution module is `executorBuild::build`, the output result is fetched in `clientConn::writeChunks`. 4 | 5 | ## Execution Framework 6 | 7 | TiDB builds the computing engine based on the distributed storage provided by TiKV. The TiKV server implements a coprocessor framework to support distributed computing. The computing operations will be pushed to the TiKV coprocessor as far as possible to accelerate the computation speed. That is to say, a sub-plan of the SQL execution plan will be executed in parallel on different TiKV servers, and the result of each sub-plan will be collected to a TiDB server to compute for the final result. 8 | 9 | The processing model of the execution plan tree is known as the Volcano iterator model. The essential of the Volcano model is abstracted to 3 interfaces: `Open`, `Next`, and `Close`. All operators offer the same interfaces and the implementation is opaque. 10 | 11 | `Open` will be invoked in turn for each operator to init the needed resources before computing. Conversely, `Close` will release the resources. To obtain the query output, the final operator in the plan tree will keep invoking `Next` until no tuple is pulled from its child. 12 | 13 | It's easy to understand how the Volcano model works for single-process execution. For parallelism issues, the Volcano introduces an operator called `Exchange` at any desired point in a plan tree. Further explanation about the parallelism-related issues would be introduced in the [Parallel Execution Framework](parallel-execution-framework.md) section. 14 | 15 | ## Vectorized Execution 16 | 17 | Vectorization uses the Volcano iteration model where each operator has a `Next` method that produces result tuples. However, each `Next` call fetches a block of tuples instead of just one tuple. 18 | 19 | The main principle of vectorized execution is batched execution on a columnar data representation: every "work" primitive function that manipulates data does not work on a single data item, but on a vector (an array) of such data items that represents multiple tuples. The idea behind vectorized execution is to amortize the iterator call overhead by performing as much as possible inside the data manipulation methods. For example, this work can be to hash 1000s of values, compare 1000s of string pairs, update a 1000 aggregates, or fetch a 1000 values from 1000s of addresses. 20 | 21 | Columnar Different from the row-oriented data representation, columnar format organize data by column rather by row. By storing data in columns rather than rows, the database can more precisely access the data it needs to answer a query rather than scanning and discarding unwanted data in rows. The memory columnar data representation in TiDB is defined as `Chunk`, which is inspired by [Apache Arrow](https://arrow.apache.org/). 22 | 23 | The detailed definition and usage of `Chunk` will be introduced in the [Implementation of Vectorized Execution](implementation-of-vectorized-execution.md) section. 24 | 25 | ## Memory Management Mechanism 26 | 27 | In TiDB, we set a memory usage quota for a query, and introduce two interfaces called `Tracker` and `OOMAction` for memory management. 28 | 29 | The `Tracker` is used to track the memory usage of each element. The `OOMAction` is used to abstract the strategies to be used when the memory usage of a SQL exceeds the memory quota. 30 | 31 | For example, we define the spill to disk strategy as `SpillDiskAction`. `SpillDiskAction` might be triggered by `HashJoin` or `Sort` when the memory quota is exceeded. If a query requires an order guarantee, and there is no index to guarantee the order, then the execution must sort the input before proceeding. If the input is small then the sort occurs in memory. We can split the input into multiple partitions and perform a merge sort on them. If the input is large, the `SpillDiskAction` will be triggered, an external sort algorithm is used. External sorting algorithms generally fall into two ways, sorting and merge. In the sorting phase, chunks of data small enough to fit in main memory are read, sorted, and written out to a temporary file. In the merge phase, the sorted subfiles are combined, and the final result will be outputted. 32 | 33 | For more details, you can refers to the [Memory Management Mechanism](memory-management-mechanism.md) section. 34 | 35 | ## Typical Operators 36 | 37 | TiDB implements multiple algorithms for the Join, Aggregation, Sort operators, and so on. We'll take some of them for detailed introduction. If you are interested, you can refer to the [Implementation of Typical Operators](implementation-of-typical-operators.md) section. 38 | -------------------------------------------------------------------------------- /src/understand-tidb/memory-management-mechanism.md: -------------------------------------------------------------------------------- 1 | # Memory Management Mechanism 2 | 3 | TiDB's memory management basically consists of a memory usage quota settings for each query, and two interfaces, called `Tracker` and `OOMAction`. 4 | 5 | ## Tracker 6 | 7 | `Tracker` tracks the memory usage of each element with a tree structure. 8 | 9 | Genral use cases: 10 | 11 | ```text 12 | /--- Tracker(Component in Executor, e.g. list/rowContainer/worker) 13 | | ... 14 | /--- Tracker(Executor1) ---+--- Tracker(Component) 15 | | 16 | Tracker(Session) ---+--- Tracker(Executor2) 17 | | | ... 18 | | \--- Tracker(Executor3) 19 | OOM-Action1 20 | | 21 | | 22 | OOM-Action2 23 | ... 24 | ``` 25 | 26 | When a component allocates some memory, it will call the function `Tracker.Consume(bytes)` to tell the `Tracker` how much memory it uses. `Tracker.Comsume` will traverse all its ancestor nodes, accumulate memory usage and trigger OOM-Action when exceeded. 27 | 28 | ## OOM-Action 29 | 30 | `OOM-Action` is a series of actions grouped in a linked list to reduce memory usage. Each node on the linked list abstracts a strategy to be used when the memory usage of a SQL exceeds the memory quota. For example, we define the spill to disk strategy as `SpillDiskAction`, rate limit strategy as `rateLimitAction` and cancel strategy as `PanicOnExceed`. 31 | 32 | ### Rate Limit 33 | 34 | TiDB supports dynamic memory control for the operator that reads data. By default, this operator uses the maximum number of threads that `tidb_disql_scan_concurrency` allows to read data. When the memory usage of a single SQL execution exceeds `tidb_mem_quota_query` each time, the operator that reads data stops one thread. 35 | 36 | We use `rateLimitAction` to dynamically control the data reading speed of `TableReader`. 37 | 38 | ### Spill Disk 39 | 40 | TiDB supports disk spill for execution operators. When the memory usage of a SQL execution exceeds the memory quota, tidb-server can spill the intermediate data of execution operators to the disk to relieve memory pressure. Operators supporting disk spill include Sort, MergeJoin, HashJoin, and HashAgg. 41 | 42 | #### SpillDiskAction 43 | 44 | We use `SpillDiskAction` to control the spill disk of `HashJoin` and `MergeJoin`. The data will be placed in Chunk unit when spilling. We can get any data in Chunk through random I/O. 45 | 46 | #### SortAndSpillDiskAction 47 | 48 | We use `SortAndSpillDiskAction` to control the spill disk of `Sort`. 49 | 50 | If the input of `SortExec` is small, then it sorts in memory. If the input is large, the `SortAndSpillDiskAction` will be triggered, and an external sort algorithm will be used. We can split the input into multiple partitions and perform a merge sort on them. 51 | 52 | External sorting algorithms generally have two stages, sort and merge. In the sort stage, chunks of data small enough to fit in main memory are read, sorted, and written out to a temporary file. In the merge stage, the sorted subfiles are combined, and the final result will be outputted. 53 | 54 | #### AggSpillDiskAction 55 | 56 | We use `AggSpillDiskAction` to control the spill disk of `HashAgg`. When `AggSpillDiskAction` is triggered, it will switch HashAgg executor to spill-mode, and the memory usage of HashAgg won't grow. 57 | 58 | We use the following algorithm to control the memory increasing: 59 | 60 | 1. When the memory usage is higher than the `mem-quota-query`, switch the HashAgg executor to spill-mode. 61 | 2. When HashAgg is in spill-mode, keep the tuple in the hash map no longer growing. 62 | a. If the processing key exists in the Map, aggregate the result. 63 | b. If the processing key doesn't exist in the Map, spill the data to disk. 64 | 3. After all data have been processed, output the aggregate result in the map, clear the map. Then read the spilling data from disk, repeat the Step1-Step3 until all data gets aggregated. 65 | 66 | As we can see, unlike other spilling implementations, `AggSpillDiskAction` does not make the memory drop immediately, but keeps the memory no longer growing. 67 | 68 | ### Log/Cancel 69 | 70 | When the above methods cannot control the memory within the threshold, we will try to use `PanicOnExceed` to cancel the SQL or use `LogOnExceed` to log the SQL info. -------------------------------------------------------------------------------- /src/understand-tidb/parallel-execution-framework.md: -------------------------------------------------------------------------------- 1 | # Parallel Execution Framework 2 | 3 | In order to make full use of the multi-core ability of modern hardware, most popular DBMS have implemented the ability of parallel execution SQL execution engine. 4 | 5 | There are three common parallel implementations: Intra operator parallelism, Exchange operator, and Morsel-Driven parallelism. And TiDB adopts the first approach. 6 | 7 | ## TiDB Implementation 8 | 9 | In intra-operator parallelism, multiple goroutines will be created inside the operator for parallel processing. The creation, termination and synchronization of the goroutines are handled by the operator itself. 10 | 11 | Most operators will create multiple goroutines in `Open()` or the first call of `Next()`. And they will wait on channels for input Chunk. Also, a special channel is responsible for notifying whether to stop the computation. And `WaitGroup` can be used to wait for the termination of all goroutines. This usually happens when `Close()` is called. 12 | 13 | Taking HashJoin as an example, all workers will be started at the first call of `Next()`, including: 14 | 15 | 1. buildSideFetcher: Will call `buildSideExec.Next()` to fetch input Chunk. 16 | 2. builderWorker: Receive data from the buildSideFetcher and build the HashTable. 17 | 3. probeSideFetcher: Wait for the end of building of the HashTable and call `probeSideExec.Next()` to fetch input Chunk. 18 | 4. probeWorker: Receive data from the probeSideFetcher and then probe the HashTable. 19 | 20 | When the main goroutine calls `HashJoinExec.Next()`, it will read data from the result channel and send an empty Chunk to the resource channel. When `HashJoinExec.Close()` is called, a special channel will be closed and all workers waiting on the channel will exit. 21 | 22 | The parallel implementations of operators are various. For example, HashAgg is divided into two stages. Each stage will start multiple goroutines (partialWorkers and finalWorkers). 23 | 24 | It is worth noting that we still use the traditional Volcano-model, the `Next()` is only called by a single thread. The parallelism only occurs in the internal processing of the operator. 25 | 26 | The degree of parallelism (DOP) can be controlled by the session variable. For example, `tidb_executor_concurrency` is 5 by default. It means HashJoin will create five goroutines to probe HashTable. You can also control the parallelism of a specific operator by changing the session variable, such as `tidb_hash_join_concurrency`. 27 | 28 | At present, most important operators have implemented intra-operator parallelism: 29 | 30 | 1. Join: HashJoin, IndexJoin, IndexHashJoin, IndexMergeJoin 31 | 2. Apply: ParallelApply 32 | 3. Aggregation: HashAgg 33 | 4. Other: Union, Projection 34 | 5. Reader: TableReader, IndexReader 35 | 36 | Other operators are still single threaded: TopN, Limit, Sort, MergeJoin, StreamAgg, Selection, WindowFunc. But some of them (TopN, Limit, StreamAgg, Selection) can be pushed down to TiKV. 37 | 38 | ## Other Parallelism Implementation 39 | 40 | Intra operator parallelism is the most intuitive way to implement parallelism, but its implementation is complex, because every operator needs to implement parallelism independently. What's worse, too many threads will increase the scheduling overhead. Although, the use of goroutines can alleviate this problem. 41 | 42 | A more traditional way to implement parallelism is to use the exchange operator. By encapsulating the parallel logic into the exchange operator, all other operators only need to consider the single thread implementation. 43 | 44 | The exchange operator is responsible for data exchange between different threads. The internal implementation of exchange is divided into two parts: sender and receiver. And data is transferred from sender to receiver in different ways, such as hash partition, random distribution or sort merge. 45 | 46 | There are usually two ways to control DOP: 47 | 48 | 1. Users can use hints to specify DOP explicitly. 49 | 2. The optimizer can choose DOP according to the table size of the scan operation automatically. 50 | 51 | This approach requires the optimizer to generate parallel plans. Generally, plan generation is divided into two stages. The first stage generates serial plans, and the second stage generates its corresponding parallel plans. The second stage is mainly responsible for inserting the exchange operator into the plan tree at the appropriate position. Both heuristic rules and cost model can be used to get the optimal parallel plan. 52 | 53 | Currently, TiDB has a simplified implementation of exchange operator: `Shuffle Operator`. It can make MergeJoin, StreamAgg and WindowFunc run in parallel. And you can enable MergeJoin to be parallel by setting `tidb_merge_join_concurrency` be greater than 1. 54 | 55 | For Morsel-Driven, it implements parallelism by dividing data into fixed size blocks (Morsel: usually 100000 rows). And a customized scheduler will be responsible for task scheduling to achieve better load balancing. And TiDB doesn't use this approach for now. 56 | -------------------------------------------------------------------------------- /src/understand-tidb/parser.md: -------------------------------------------------------------------------------- 1 | # Parser 2 | 3 | Parser is responsible for interpreting a SQL string into an abstract syntax tree (AST), which is more structural and easier to process. AST can be used for preprocessing, syntactic analysis, and so on. 4 | 5 | The code lives in the [pingcap/tidb repo, parser directory](https://github.com/pingcap/tidb/tree/master/pkg/parser). 6 | 7 | ## Understand Parser 8 | 9 | Parser is generated by a parser generator named [yacc](https://github.com/cznic/parser). It takes the grammar file `parser.y` as the input and outputs the source code file `parser.go`, which is the real parser imported by TiDB. Thus, the core file is `parser.y` because when the SQL syntax changes, most of the changes take place in `parser.y`. 10 | 11 | In case you are unfamiliar with yacc, some concepts are listed here: 12 | 13 | * **Terminal Symbol** is also known as "token". When a SQL string reaches parser, the first step is to tokenize them into an array of tokens. For example, `"SELECT * FROM t"` is tokenized to `[selectKeyword, '*', fromKeyword, identifier(t)]` by [lexer.Lex()](https://github.com/pingcap/tidb/blob/4b110036e1a7d7c3584113512a544c8350459157/parser/lexer.go#L169). 14 | * **Non-terminal Symbol** is a syntactic variable, which can represent a group of terminal/non-terminal symbols. 15 | * **Grammar Rule** specifies which symbols can replace which other non-terminal symbol. 16 | * **Semantic Action** defines how an AST node is constructed. 17 | 18 | An example of a grammar rule is as follows: 19 | 20 | ``` 21 | AlterDatabaseStmt: 22 | "ALTER" DatabaseSym DBName DatabaseOptionList 23 | { 24 | $$ = &ast.AlterDatabaseStmt{ 25 | Name: $3, 26 | AlterDefaultDatabase: false, 27 | Options: $4.([]*ast.DatabaseOption), 28 | } 29 | } 30 | ``` 31 | 32 | * `AlterDatabaseStmt` is a non-terminal symbol because there is no such token. 33 | * `"ALTER"` is a terminal symbol. 34 | * `DatabaseSym`, `DBName` and `DatabaseOptionList` are non-terminal symbols that are defined in other grammar rules. 35 | * The pseudo-code in brackets is the semantic action. It means an AST node `ast.AlterDatabaseStmt` will be constructed when the rule is reduced by the parser. Note that a dollar character `$` followed by a number represents the binding Golang value previously (in other rules), where the number is the index of symbol in rule (1-based). `$$` represents current binding value. After goyacc substitution, this code snippet will be valid Golang code. 36 | 37 | Getting back to `parser.y`, the structure of this file is divided into three parts: 38 | 39 | 1. `%union` enumerates all the Golang types that can be passed around grammar rules. 40 | 2. `%token` or `%type` declares the terminal or non-terminal symbols that will be used in grammar rules. 41 | 3. Grammar rules define the syntax of SQL and related semantic actions. 42 | 43 | Except for `parser.y`, other sub-package/files should be easy to understand, feel free to explore them by yourself: 44 | 45 | | Package | Description | 46 | |-------------|--------------------------------------------------------------------------| 47 | | ast | The AST definition used by TiDB. | 48 | | auth | Authentication-related functions. | 49 | | charset | Currently supported charsets and encodings. | 50 | | format | The Formatters for yacc file and the functions for restoring AST to SQL. | 51 | | goyacc | The generator for parser.go. | 52 | | model | Basic structures in TiDB like TableInfo, ColumnInfo... | 53 | | mysql | MySQL constants, errors, privileges, types, and others. | 54 | | opcode | Operator code like <, >, +, =... | 55 | | terror | The errors used by TiDB. | 56 | | test_driver | A parser driver only for unit tests. | 57 | | tidb | TiDB related features' keywords. | 58 | | types | The field types and evaluation types that used in TiDB. | 59 | 60 | ## Develop and Build 61 | 62 | To get started with the parser development, please also take a look at [quickstart.md](https://github.com/pingcap/tidb/blob/master/parser/docs/quickstart.md). It shows the basic usage of the parser and it explains some concepts like parser_driver. 63 | 64 | Run `make parser` in the project root directory to generate a new `parser.go`. 65 | 66 | ## FAQ 67 | 68 | 1. How to debug the parsing procedure? 69 | 70 | Put the test file in the `parser` package. Set [`yyDebug`](https://github.com/pingcap/tidb/blob/4b110036e1a7d7c3584113512a544c8350459157/parser/goyacc/main.go#L525) level to `4`(or any integers >= 4) before calling `Parse()`. The parser will try to show state information in each step. 71 | 72 | 2. How to resolve shift-reduce or reduce-reduce conflicts? 73 | 74 | Shift means "move the next token in" to match the current rule. Reduce means "replace current tokens/symbols to a non-terminal symbol". Shift-reduce conflicts occur when the parser cannot decide the next step is to shift or to reduce. 75 | 76 | When yacc reports such conflicts, it also keeps the file `y.output`. You can search "conflict on" in the file to locate which rule conflicts with other rules. Then you can try to annotate the `%precedence` to tokens, rewrite the grammar rule, or ask for help on GitHub. 77 | -------------------------------------------------------------------------------- /src/understand-tidb/pessimistic-transaction.md: -------------------------------------------------------------------------------- 1 | # Pessimistic Transaction 2 | -------------------------------------------------------------------------------- /src/understand-tidb/plan-cache.md: -------------------------------------------------------------------------------- 1 | # Plan Cache 2 | 3 | ## Introduction 4 | 5 | TiDB supports `PlanCache` for `prepare` and `execute` statements. By using `PlanCache`, TiDB can skip the optimization phase to gain some performance benefits. 6 | 7 | There are several limitations to current `PlanCache`: 8 | 1. Only support `prepare` and `execute` statements, not support general queries; 9 | 2. Only session-level `PlanCache` is supported, cached plans cannot be reused across sessions; 10 | 3. Some complex plans cannot be cached, and you can see [this document](https://docs.pingcap.com/tidb/stable/sql-prepare-plan-cache) for more details; 11 | 12 | ## Handling Prepare/Execute Statement 13 | 14 | The process of handling a `prepare` statement is below: 15 | 1. Parse the original SQL to an AST; 16 | 2. Encapsulate the AST with some other necessary information(like `current-db`, `statement-text`, ...) to `CachedPrepareStmt`; 17 | 3. Store the `CachedPrepareStmt` into this session's `PreparedStmtMap`; 18 | 19 | The process of handling an `execute` statement is below: 20 | 1. Parse all parameters and store their values into this session's `PreparedParamMap`; 21 | 2. Get the corresponding `CachedPrepareStmt` from this session's `PreparedStmtMap`; 22 | 3. Construct the plan-cache key with information in `CachedPrepareStmt`; 23 | 4. Fetch a plan from the session's `PlanCache` by using this key, and: 24 | 1. if it succeeds, then 25 | 1. check whether this plan is valid for current parameters, if it is, then 26 | 2. rebuild the plan, and 27 | 3. run the rebuilt plan; 28 | 2. else, then 29 | 1. optimize the AST in `CachedPrepareStmt` with current parameters to get a new plan, 30 | 2. store the new plan into current session's `PlanCache`, and 31 | 3. run the new plan. 32 | 33 | ![plan-cache-execute](../img/plan-cache-execute.png) 34 | 35 | The main function of handling `execute` is `common_plans.go:Execute.getPhysicalPlan()`. 36 | 37 | ## Plan Rebuilding 38 | 39 | A cached plan cannot be reused directly unless it is rebuilt. The main goal of rebuilding is to re-calculate the access range. 40 | 41 | For example, if the query is `select * from t where a1`, then the filter `a+?>1` will be converted to a `Selection` operator with an expression-tree `a+?>1`: 52 | 53 | ![plan-cache-parameter](../img/plan-cache-parameter.png) 54 | 55 | The parameter placeholder(`?`) is converted to a `ParamMaker` in `Constant`. 56 | 57 | You can regard it as a special kind of pointer, and when the plan is processed and the parameter's value is needed, we can use it to get the corresponding value of this parameter. 58 | 59 | ``` 60 | type Constant struct { 61 | ... 62 | ParamMaker *ParamMaker 63 | } 64 | 65 | func (c *Constant) Eval(row) Datum { 66 | if c.ParamMaker != nil {return c.ParamMaker.GetUserVal()} 67 | ... 68 | } 69 | 70 | func (d *ParamMaker) GetUserVal() Datum { 71 | return d.ctx.GetSessionVars().PreparedParams[d.order] 72 | } 73 | ``` -------------------------------------------------------------------------------- /src/understand-tidb/planner.md: -------------------------------------------------------------------------------- 1 | # Planner 2 | 3 | The `planner` package contains most of the codes related to SQL optimization. The input of the planner is an AST of the query returned from the parser, and the output of the planner is a plan tree that would be used for further execution. 4 | 5 | ## Package Structure 6 | 7 | | Package | Description | 8 | | :----------------------------------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------| 9 | | [tidb/pkg/planner/cascades](https://github.com/pingcap/tidb/tree/master/pkg/planner/cascades) | The next generation Cascades model planner, which is under development and disabled by default | 10 | | [tidb/pkg/planner/core](https://github.com/pingcap/tidb/tree/master/pkg/planner/core) | The core logic of the currently used System R model planner. The Cascades model planner also calls utility functions in this package | | 11 | | [tidb/pkg/planner/implementation](https://github.com/pingcap/tidb/tree/master/pkg/planner/implementation) | Physical implementations for the operators in Cascades planner | 12 | | [tidb/pkg/planner/memo](https://github.com/pingcap/tidb/tree/master/pkg/planner/memo) | Intermediate results for the searching procedure of Cascades planner | 13 | | [tidb/pkg/planner/property](https://github.com/pingcap/tidb/tree/master/pkg/planner/property) | Properties about the output of operators, including schema, stats, order property, partition property, etc | 14 | | [tidb/pkg/planner/util](https://github.com/pingcap/tidb/tree/master/pkg/planner/util) | Common utility functions / structures shared by the two planners | 15 | 16 | We can see that, TiDB has two planners, one is of System R model, which is defaultly used, and the other is of Cascades model, which is still under development. The unified entry function of planner module is `Optimize()`, before diving into either of the two planners, it would firstly check if there is any intervention for the planner from the "SQL Plan Management" module, if yes, the AST of the query would be modified before going through the optimization procedures. "SQL Plan Management" module is beyond the scope of this article, and it would be introduced in the [SQL Plan Management](sql-plan-management.md) section. 17 | 18 | This article would only focus on introducing the System R planner, i.e, the `core` package, readers who are interested in the Cascacdes planner can refer to this [design](https://github.com/pingcap/tidb/tree/master/docs/design/2018-08-29-new-planner.md) doc. 19 | 20 | ## Optimization Procedures 21 | 22 | Ignore the trivial steps, the query optimization procedures can be briefly divided into 4 phases: 23 | 24 | 1. build an initial logical plan 25 | 2. logically optimize the initial logical plan 26 | 3. physically optimize the logical plan 27 | 4. tidy up the physical plan 28 | 29 | ### Plan Building 30 | 31 | The entry function of this phase is `PlanBuilder.Build()`, it would translate the input AST to a logical plan tree from bottom up according to the predefined rules / orders. Specifically, it would check each sub-clause of the query, and build a corresponding operator for the clause. The operators are connected as a DAG, which is known as a logical plan tree. 32 | 33 | A key step in this phase is translating the expressions for each clause, e.g, `where a = 1` would have a `Selection` operator built correspondingly, and an expression `eq(a, 1)` would be translated and saved in the `Selection operator`. The expression translation logics are encapsulated in a structure `expressionRewriter` and its methods. The `expressionRewriter` would traverse and transalte the AST expressions recursively, and utilize a result stack for intermediate results. 34 | 35 | `expressionRewriter` would not only do the simple expression transaltions, but would optimize subqueries in the expressions. The details of subquery optimization would not be explained here, because they are pretty complicated. Briefly speaking, for most of the uncorrelated subqueries, `expressionRewriter` would directly execute them and substitute them with the result constants. For correlated subqueries, or some of the uncorrelated subqueries, `expressionRewriter` would build a subtree from them and connect it with the main plan tree using a `LogicalJoin` or `LogicalApply` operator. Note that, `LogicalApply` is a special kind of join operator which can only be executed in a nested-loop approach. `LogicalApply` operator in some plan trees can be converted to a regular `LogicalJoin`, which can be executed in other more efficient join algorithms, and planner would do this conversion in the subsequent logical optimization phase if possible. 36 | 37 | During the plan building process, optimization flags would be collected for each operator built. For example, if a `Selection` operator is built, then an optimization flag like `flagPredicatePushDown` would be set in the plan builder. These saved flags would be used later in the logical optimization phase. 38 | 39 | ### Logical Optimization 40 | 41 | The entry function of this phase (also known as rule-based optimization) is `logicalOptimize()`. This function would do logically equivalent transformations for the initial plan tree according to relational algebra, and the result plan tree should be better than the initial one from the execution efficiency perspective in principle. Specifically, `logicalOptimize()` would traverse all the logical optimization rules predefined as `optRuleList` in order, and check if a rule is applicable by referring to the optimization flags saved during the plan building phase. If the flag is set for a rule, planner would traverse the plan tree from top down, and apply the transformations implied by the rule to the subtree satisfying the rule prerequisites. 42 | 43 | An example logical optimization rule is "column pruning", for each operator in the plan tree, it would collect the columns needed by the upper operators, and prune the unneeded columns from the output. Another example rule is "decorrelation", it would try to pull up operators referring correlated columns, and resolve the column dependency, hence convert the `LogicalApply` operator to a regular `LogicalJoin`. 44 | 45 | ### Physical Optimization 46 | 47 | The entry function of this phase (also known as cost-based optimization) is `physicalOptimize()`, it would do cost based enumeration for the implementations of each logical operator, and find a combination of all operators with the lowest cost as the final physical plan. Specifically, each logical operator would implement an interface function `exhaustPhysicalPlans()` to list all the possible physical algorithms, e.g, `LogicalAggregation` would have two possible implementations including `PhysicalStreamAggregation` and `PhysicalHashAggregation`. Each implementation may require specific properties for its child's output, e.g, `PhysicalStreamAggregation` would require that the child's output rows should be in order of the `GROUP BY` columns. These properties are recorded in `PhysicalProperty` structure, and passed down to the enumeration procedure of the child operators. 48 | 49 | Once the planner knows the specific implementation of the plan tree, or of a subtree, it can compute a cost for this implementation. The cost of one implementation is calculated as a sum of its resource consumptions including CPU, Memory, Network, IO, etc. For each kind of resource specifically, the consumption is measured based on a unit factor (e.g, scanFactor is the unit factor for IO consumption, which means the cost of scanning 1 byte data on TiKV or TiFlash), and the estimated number of rows / bytes to be processed by this operator. Note that, these unit factors can be customized by setting system variables like `tidb_opt_xxx_factor` to fit clusters of different hardware configurations. Each implementation of the whole logical plan tree would have a cost then, planner would choose the one with the lowest cost for execution. 50 | 51 | One thing worth mention is that, TiDB supports pushing some operators down to storage engine to speed up the query execution, e.g, we can push `Selection` operator down to the coprocessor of TiKV, and reduce the rows returned from TiKV to TiDB through the network. The logic about deciding whether to push operators down to storage engine or not is piggybacked on the search framework of the physical optimization. Specifically, it is achieved by introducing `TaskType` field into `PhysicalProperty`, for example, once the planner wants to push down a `Limit` operator to TiKV, it would enumerate an implementation `PhysicalLimit` which has `CopXXXTaskType` as the `TaskType` of the required `PhysicalProperty` for its child. Once the child of `PhysicalLimit` has generated a TiKV implementation, these two plan snippets would be concatenated in `attach2Task()` interface, hence achieving the operator pushdown for storage engine. 52 | 53 | ### Post Optimization 54 | 55 | The entry function of this phase is `postOptimize()`. The query optimization has almost finished when coming here, this phase would not apply big changes to the plan, it would only do some clean and tidy up works. The jobs in this phase include a new round of projection elimination(the first round is applied in logical optimization), and projection injection to simplify the code implementations of executor package, and so on. 56 | 57 | ## Summary 58 | 59 | This section talks about the brief steps of query optimization, and the corresponding entry functions for each step. 60 | -------------------------------------------------------------------------------- /src/understand-tidb/plugin.md: -------------------------------------------------------------------------------- 1 | # Plugin 2 | 3 | The plugin API allows TiDB to be extended with new features such as audit logging or IP allow/deny listing. 4 | 5 | Sample code is provided for a basic audit logging plugin at [`plugin/conn_ip_example/`](https://github.com/pingcap/tidb/tree/master/pkg/plugin/conn_ip_example). For an example on compiling TiDB and this plugin: 6 | 7 | ```bash 8 | plugin="conn_ip_example" 9 | cd cmd/pluginpkg 10 | go install 11 | cd ../../plugin/$plugin 12 | pluginpkg -pkg-dir . -out-dir . 13 | cd ../.. 14 | 15 | ./bin/tidb-server -plugin-dir plugin/$plugin -plugin-load $plugin-1 16 | ``` 17 | 18 | An explanation of what this does: 19 | 20 | - `cd cmd/pluginpkg` and `go install` compiles the command line utility called `pluginpkg`, which is used to build the plugin. 21 | - `pluginpkg -pkg-dir . -out-dir .` reads the plugin code + `manifest.toml` file and generates a shared object file for the plugin (`conn_ip_example-1.so`). 22 | - When the tidb-server starts, it can load plugins in a specified directory (`plugin-dir`). 23 | 24 | You can confirm which plugins are installed with the `SHOW PLUGINS` statement: 25 | 26 | ```sql 27 | mysql> show plugins; 28 | +-----------------+--------------+-------+--------------------------------------------------------------------------------------+---------+---------+ 29 | | Name | Status | Type | Library | License | Version | 30 | +-----------------+--------------+-------+--------------------------------------------------------------------------------------+---------+---------+ 31 | | conn_ip_example | Ready-enable | Audit | /home/morgo/go/src/github.com/morgo/tidb/plugin/conn_ip_example/conn_ip_example-1.so | | 1 | 32 | +-----------------+--------------+-------+--------------------------------------------------------------------------------------+---------+---------+ 33 | 1 row in set (0.00 sec) 34 | ``` 35 | 36 | ## Customizing the example plugin 37 | 38 | The manifest file describes the capabilities of the plugin, and which features it implements. For a basic version: 39 | 40 | ```toml 41 | name = "conn_ip_example" 42 | kind = "Audit" 43 | description = "just a test" 44 | version = "1" 45 | license = "" # Suggested: APLv2 or GPLv3. See https://choosealicense.com/ for details 46 | validate = "Validate" 47 | onInit = "OnInit" 48 | onShutdown = "OnShutdown" 49 | export = [ 50 | {extPoint="OnGeneralEvent", impl="OnGeneralEvent"}, 51 | {extPoint="OnConnectionEvent", impl="OnConnectionEvent"} 52 | ] 53 | ``` 54 | 55 | In addition to this basic example, plugins can also implement an [`OnFlush`](https://github.com/pingcap/tidb/blob/d58d39e9476f2503a1e8790f78a0d25272d0aabe/plugin/spi.go#L51) function. This is called when the statement `FLUSH TIDB PLUGINS pluginName` is executed. TiDB does not require plugins to implement a `OnFlush` function, but when specified it will call this method on all TiDB nodes in the cluster. 56 | 57 | ### OnConnectionEvent 58 | 59 | The `OnConnectionEvent` is called when a new connection is initially created (`event plugin.ConnectionEvent == plugin.PreAuth`) and again when the connection is successfully established (`event plugin.ConnectionEvent == plugin.Connected`). 60 | 61 | To prevent a connection from being created, an error should be returned for the event `plugin.PreAuth`. 62 | 63 | ### OnGeneralEvent 64 | 65 | The `OnGeneralEvent` is called: 66 | - Before a statement starts execution (`event plugin.GeneralEvent == plugin.Starting`) 67 | - Ater a statement has completed execution (`event plugin.GeneralEvent == plugin.Completed`) 68 | 69 | General events are useful for auditing operations performed by users. Because [`sctx SessionVars`](https://github.com/pingcap/tidb/blob/b2a1d21284b75e3137f499d8954071a7b32f7b3b/sessionctx/variable/session.go#L432-L436) is available in the `OnGeneralEvent` function, it is possible to obtain a lot of additional information about the statement being executed. For example: 70 | 71 | * `sctx.User` contains the `*auth.UserIdentity` of the user who is executing this session, and `sctx.ActiveRoles` contains the list of active roles associated with the session. 72 | * `sctx.DBName` contains the name of the database the user is executing in. 73 | * `sctx.StmtCtx` contains the context of the statement that was executed. For example `sctx.StmtCtx.SQLDigest()` can be called to get a digest of the executed statement, and `sctx.StmtCtx.Tables` contains a slice of tables that are accessed by the statement. 74 | 75 | The current implementation of `OnGeneralEvent` does not permit errors to be returned. It is possible that this may change in a future version, since this will allow pre-execution checks to be performed on statements. 76 | 77 | ## Additional Reading 78 | 79 | * [Plugin Framework RFC Proposal](https://github.com/pingcap/tidb/blob/master/docs/design/2018-12-10-plugin-framework.md) 80 | -------------------------------------------------------------------------------- /src/understand-tidb/privilege.md: -------------------------------------------------------------------------------- 1 | # Privilege 2 | 3 | At its core, TiDB's approach to user privileges is similar to that of MySQL: 4 | 5 | * The privileges are stored in tables such as `mysql.user` and `mysql.db`. 6 | * The privilege tables are then loaded into an [in-memory cache](https://github.com/pingcap/tidb/blob/master/privilege/privileges/cache.go). The cache is then used by the privilege manager to determine the privileges of a user. 7 | * The cache is automatically updated when using privilege control statements such as `GRANT` and `REVOKE`. The statement `FLUSH PRIVILEGES` can also be used to manually reload the cache for when manual changes are made to the privilege tables. 8 | 9 | ## Behavior differences from MySQL 10 | 11 | Implicit updates to the privilege cache (i.e. when `GRANT` or `REVOKE` statements are executed) run immediately on the instance of TiDB that is executing the statement. A [notification is also sent to all TiDB instances](https://github.com/pingcap/tidb/blob/5e05922de6a253859cfbfe19356de8a2e2db39da/domain/domain.go#L1355-L1373) to rebuild their cache. This notification is sent asynchronously, so it is possible that when a load balancer is used, the cache will be out of date when attempting to reconnect to a TiDB instance immediately. 12 | 13 | Because the asynchronous notifications do not guarantee delivery, TiDB will also [automatically rebuild the privilege cache](https://github.com/pingcap/tidb/blob/5e05922de6a253859cfbfe19356de8a2e2db39da/domain/domain.go#L852-L908) every 5-10 minutes in a loop. This behavior is not strictly MySQL compatible, because in MySQL the privilege cache will only ever be rebuilt from a `FLUSH PRIVILEGES` statement, a restart, or a privilege control statement. 14 | 15 | Client certificate options are stored in the `mysql.global_priv` table instead of the `mysql.user` table. This behavior is not intentional, and may be changed in the future. 16 | 17 | ## Adding privilege checks to a statement 18 | 19 | Some privilege checks are automatically assigned during plan building, for example ensuring that you have permissions to the tables that will be accessed. These checks are skipped for `information_schema` tables, and should you add an additional statement (such as `SHOW xyz`), you will also need to ensure that privilege checks are added. 20 | 21 | Should you need to add privilege checks there are two options: 22 | 23 | 1. During plan building you can attach `visitInfo` to the plan (examples: [`SET CONFIG`](https://github.com/pingcap/tidb/blob/5e05922de6a253859cfbfe19356de8a2e2db39da/planner/core/planbuilder.go#L745), [`SHOW BACKUPS`](https://github.com/pingcap/tidb/blob/5e05922de6a253859cfbfe19356de8a2e2db39da/planner/core/planbuilder.go#L2378-L2380)) 24 | 25 | 2. In the executor function which handles the statement (examples: [`SHOW PROCESSLIST`](https://github.com/pingcap/tidb/blob/1a54708a7f8f86515236626c78e97a33d8adf583/executor/show.go#L368-L380)). 26 | 27 | The first option is recommended, as it is much less verbose. However, `visitInfo` does not handle cases where the statement can behave differently depending on the permissions of the user executing it. All users can execute the `SHOW PROCESSLIST` statement, but to see the sessions of other users requires the `PROCESS` privilege. 28 | 29 | `visitInfo` also only supports **AND** semantics. For complex scenarios (such as `DROP USER` requiring either `CREATE USER` **OR** `DELETE` privileges on the `mysql.user` table), option 2 is required. 30 | 31 | ### Manually checking with the privilege manager 32 | 33 | For (2) above, manual checks should follow the following pattern: 34 | 35 | ```go 36 | checker := privilege.GetPrivilegeManager(e.ctx) 37 | if checker != nil && !checker.RequestVerification(ctx.GetSessionVars().ActiveRoles, schema.Name.L, table.Name.L, "", mysql.AllPrivMask) { 38 | /* do something */ 39 | } 40 | .. 41 | if checker == nil || !checker.RequestDynamicVerification(ctx.GetSessionVars().ActiveRoles, "RESTRICTED_TABLES_ADMIN", false) { 42 | /* do something */ 43 | } 44 | ``` 45 | 46 | The check for `checker != nil` is important because for internal SQL statements the privilege manager is not present. These statements are expected to fall through and satisfy the privilege check. 47 | 48 | ### Static and dynamic privileges 49 | 50 | Privileges fall into two categories: 51 | 52 | * Static privileges: These are the "traditional" privileges such as `INSERT`, `UPDATE`, `SELECT`, `DELETE`, `SUPER`, `PROCESS` which have existed in MySQL for a long time. They can *usually* be assigned to a user on either a global or database/table level. 53 | * Dynamic privileges: These are new privileges such as `BACKUP_ADMIN`, `RESTORE_ADMIN`, `CONNECTION_ADMIN`. They can only be assigned on a global level, and each have their own "grantable" attribute. 54 | 55 | Dynamic privileges were introduced in MySQL 8.0 (and [TiDB 5.1](https://github.com/pingcap/tidb/blob/master/docs/design/2021-03-09-dynamic-privileges.md)) to solve a specific issue, which is that the `SUPER` privilege is too coarse. There are many scenarios where a user needs to be assigned the `SUPER` privilege to perform a specific action, but too many other privileges are granted at the same time. 56 | 57 | Any statements added to TiDB **should no longer require** the `SUPER` privilege directly. Instead, a dynamic privilege should be added [which will be satified](https://github.com/pingcap/tidb/blob/5e05922de6a253859cfbfe19356de8a2e2db39da/privilege/privileges/cache.go#L1009) by the `SUPER` privilege. 58 | 59 | ### Security Enhanced Mode 60 | 61 | TiDB features an extension to MySQL called [Security Enhanced Mode](https://github.com/pingcap/tidb/blob/master/docs/design/2021-03-09-security-enhanced-mode.md) (SEM), which is disabled by default. One of the main aims of SEM is to reduce the privileges of `SUPER` and instead require specific "restricted" dynamic privileges instead. The design is inspired by features such as "Security Enhanced Linux" (SeLinux) and AppArmor. 62 | 63 | SEM plugs directly into the privilege manager, but the hard coded list of restricted objects lives in [`./util/sem/*`](https://github.com/pingcap/tidb/blob/master/util/sem/sem.go). It is expected that over time SEM will protect against additional operations which are considered to be high risk or too broad. 64 | 65 | ### Recommended Reading 66 | 67 | * [Technical Design: Security Enhanced Mode](https://github.com/pingcap/tidb/blob/master/docs/design/2021-03-09-security-enhanced-mode.md) 68 | * [Technical Design: Dynamic Privileges](https://github.com/pingcap/tidb/blob/master/docs/design/2021-03-09-dynamic-privileges.md) 69 | * [MySQL Worklog: Pluggable Dynamic Privileges](https://dev.mysql.com/worklog/task/?id=8131) 70 | -------------------------------------------------------------------------------- /src/understand-tidb/session.md: -------------------------------------------------------------------------------- 1 | # Session 2 | 3 | The `session` package (and related packages such as `sessionctx` and `sessionctx/variable`) are responsible for maintaining the state of both sessions and transactions. 4 | 5 | ## New session origins 6 | 7 | New connections are first established in the `server` package. After some initial protocol negotiation, the `server` package [calls](https://github.com/pingcap/tidb/blob/246c59d5c926780235dd25aef27c469ffe376e21/server/driver_tidb.go#L181-L186) `session.CreateSession()`. This function then calls `session.createSessionWithOpt()` (via `CreateSessionWithOpt()`) which creates the session. 8 | 9 | Sessions used for internal server operations are usually created in a different manner, with the sessionctx being retrieved from a pool of sessions maintained by `domain`. For example: 10 | 11 | ```go 12 | dom := domain.GetDomain(e.ctx) 13 | sysSessionPool := dom.SysSessionPool() 14 | ctx, err := sysSessionPool.Get() 15 | if err != nil { 16 | return nil, err 17 | } 18 | restrictedCtx := ctx.(sessionctx.Context) 19 | restrictedCtx.GetSessionVars().InRestrictedSQL = true 20 | ``` 21 | 22 | Internal sessions will not show up in the output of `SHOW PROCESSLIST`, and skip all privilege checks because they do not have a privilege manager handle attached. 23 | 24 | ## System variable state 25 | 26 | System variables follow similar semantics to MySQL: 27 | 28 | - If a variable includes `SESSION` scope, the value is copied to the session state when the session is created. 29 | - Any changes to the `GLOBAL` value will not apply to any existing sessions. 30 | 31 | The state of the variables is stored in [`sessionVars`](https://github.com/pingcap/tidb/blob/246c59d5c926780235dd25aef27c469ffe376e21/sessionctx/variable/session.go#L432-L958). The raw _string_ values are stored in a map named [`systems`](https://github.com/pingcap/tidb/blob/246c59d5c926780235dd25aef27c469ffe376e21/sessionctx/variable/session.go#L449-L450). This string value is used for persistence in the `mysql.global_variables` table. 32 | 33 | For many variables, as well as a _string_ value there is a typed field in `sessionVars`. For example: 34 | 35 | `SessionVars.systems["tidb_skip_utf8_check"]` (string) maps to `SessionVars.SkipUTF8Check` (bool). 36 | 37 | The typed value is set when the `SetSession` attached to the [system variable definition](https://github.com/pingcap/tidb/blob/246c59d5c926780235dd25aef27c469ffe376e21/sessionctx/variable/sysvar.go#L1029-L1032) is called. For `tidb_skip_utf8_check` this is as follows: 38 | 39 | ```go 40 | {Scope: ScopeGlobal | ScopeSession, Name: TiDBSkipUTF8Check, Value: BoolToOnOff(DefSkipUTF8Check), Type: TypeBool, SetSession: func(s *SessionVars, val string) error { 41 | s.SkipUTF8Check = TiDBOptOn(val) 42 | return nil 43 | }}, 44 | ``` 45 | 46 | The `SetSession` function can also be considered an `Init` function, since it is called when the session is created and the values are copied from global scope. To disable `SetSession` from being called on creation, `skipInit` can be set to `true`. For example with [`CharsetDatabase`](https://github.com/pingcap/tidb/blob/246c59d5c926780235dd25aef27c469ffe376e21/sessionctx/variable/sysvar.go#L745-L752): 47 | 48 | ```go 49 | {Scope: ScopeGlobal | ScopeSession, Name: CharsetDatabase, Value: mysql.DefaultCharset, skipInit: true, Validation: func(vars *SessionVars, normalizedValue string, originalValue string, scope ScopeFlag) (string, error) { 50 | return checkCharacterSet(normalizedValue, CharsetDatabase) 51 | }, SetSession: func(s *SessionVars, val string) error { 52 | if cs, err := charset.GetCharsetInfo(val); err == nil { 53 | s.systems[CollationDatabase] = cs.DefaultCollation 54 | } 55 | return nil 56 | }}, 57 | ``` 58 | 59 | In the above example, skipping the `SetSession` function is useful because it prevents the `CollationDatabase` from being overwritten when the session is initialized. This is only expected if the user issues a statement to change the `CharsetDatabase` value. 60 | 61 | ### Differences from MySQL 62 | 63 | In TiDB, changes to `GLOBAL` scoped system variables are propagated to other TiDB servers in the cluster and persist across restarts. The notification event to other servers is sent via an etcd channel in the call `domain.GetDomain(s).NotifyUpdateSysVarCache()`: 64 | 65 | ```go 66 | // replaceGlobalVariablesTableValue executes restricted sql updates the variable value 67 | // It will then notify the etcd channel that the value has changed. 68 | func (s *session) replaceGlobalVariablesTableValue(ctx context.Context, varName, val string) error { 69 | stmt, err := s.ParseWithParams(ctx, `REPLACE INTO %n.%n (variable_name, variable_value) VALUES (%?, %?)`, mysql.SystemDB, mysql.GlobalVariablesTable, varName, val) 70 | if err != nil { 71 | return err 72 | } 73 | _, _, err = s.ExecRestrictedStmt(ctx, stmt) 74 | domain.GetDomain(s).NotifyUpdateSysVarCache() // <-- the notification happens here 75 | return err 76 | } 77 | ``` 78 | 79 | Because `GLOBAL` scoped variables are propagated to other servers, TiDB also has a special concept of "instance-scoped variables". An instance scoped variable is actually a `SESSION` scoped variable that has a `GetSession` method which returns data that is specific to an instance. For example, [`tidb_general_log`](https://github.com/pingcap/tidb/blob/246c59d5c926780235dd25aef27c469ffe376e21/sessionctx/variable/sysvar.go#L1315-L1321): 80 | 81 | ```go 82 | {Scope: ScopeSession, Name: TiDBGeneralLog, Value: BoolToOnOff(DefTiDBGeneralLog), Type: TypeBool, skipInit: true, SetSession: func(s *SessionVars, val string) error { 83 | ProcessGeneralLog.Store(TiDBOptOn(val)) 84 | return nil 85 | }, GetSession: func(s *SessionVars) (string, error) { 86 | return BoolToOnOff(ProcessGeneralLog.Load()), nil 87 | }}, 88 | ``` 89 | 90 | The decision to make an option such as `tidb_general_log` instance scoped is because it references a file on the local filesystem. This may create issues when global, as the path may not be writable on each tidb-server in the cluster. 91 | 92 | As you can see by the `Scope: Session`, instance-scoped variables are not natively handled by the sysvar framework, but are instead denoted by the `GetSession()` function reading from a global location. The documentation for [`tidb_general_log`](https://docs.pingcap.com/tidb/dev/system-variables#tidb_general_log) also notes it as "instance" scoped by convention. 93 | 94 | ## Transaction state 95 | 96 | The `session` struct (`s.txn`) is responsible for keeping modified key-value pairs in a [`LazyTxn`](https://github.com/pingcap/tidb/blob/bfbea9c3ef4232d76296a9c8390eb8b7da5bf45d/session/txn.go#L46-L71) until the transaction commits. A `commit` statement only sets the session variable state that it is [no longer in an active transaction](https://github.com/pingcap/tidb/blob/246c59d5c926780235dd25aef27c469ffe376e21/executor/simple.go#L701-L703): 97 | 98 | ```go 99 | func (e *SimpleExec) executeCommit(s *ast.CommitStmt) { 100 | e.ctx.GetSessionVars().SetInTxn(false) 101 | } 102 | ``` 103 | 104 | The function [`autoCommitAfterStmt()`](https://github.com/pingcap/tidb/blob/246c59d5c926780235dd25aef27c469ffe376e21/session/tidb.go#L242-L267) which is called as part of [`finishStmt()`](https://github.com/pingcap/tidb/blob/246c59d5c926780235dd25aef27c469ffe376e21/session/tidb.go#L207-L224) is responsible for committing the transaction: 105 | 106 | ```go 107 | if !sessVars.InTxn() { 108 | if err := se.CommitTxn(ctx); err != nil { 109 | if _, ok := sql.(*executor.ExecStmt).StmtNode.(*ast.CommitStmt); ok { 110 | err = errors.Annotatef(err, "previous statement: %s", se.GetSessionVars().PrevStmt) 111 | } 112 | return err 113 | } 114 | return nil 115 | } 116 | ``` 117 | 118 | The `session.CommitTxn()` function will handle the `commit`, including retry (if permitted). There is also special handling for both pessimistic and optimistic transactions, as well as removing the key-value pairs which apply to temporary tables from the transaction buffer. 119 | 120 | ## See also 121 | 122 | - [The lifecycle of a statement](the-lifecycle-of-a-statement.md) 123 | - [Privilege management](privilege.md) 124 | - [Transaction](transaction.md) -------------------------------------------------------------------------------- /src/understand-tidb/sql-plan-management.md: -------------------------------------------------------------------------------- 1 | # SQL Plan Management 2 | -------------------------------------------------------------------------------- /src/understand-tidb/system-tables/information_schema/introduction.md: -------------------------------------------------------------------------------- 1 | # information_schema 2 | 3 | * [slow_query](slow_query.md) -------------------------------------------------------------------------------- /src/understand-tidb/system-tables/information_schema/slow_query.md: -------------------------------------------------------------------------------- 1 | # slow_query 2 | 3 | ## Overview 4 | 5 | Slow queries logs are written to `tidb-slow.log` (set with `slow-query-file` in the config) if they are over the `slow-threshold` (300ms by default). If `record-plan-in-slow-log` is enabled this will include the execution plan. This uses [`slowLogEncoder`](https://github.com/pingcap/tidb/blob/master/pkg/util/logutil/slow_query_logger.go). 6 | 7 | Each TiDB node writes its own slow logs. 8 | 9 | ### SLOW_QUERY 10 | ``` 11 | +------------+ 12 | | App or CLI | 13 | +------------+ 14 | | 15 | SQL Query 16 | SLOW_QUERY 17 | | 18 | +--------+ 19 | | TiDB | 20 | +--------+ 21 | | 22 | +---------------+ 23 | | tidb-slow.log | 24 | +---------------+ 25 | ``` 26 | 27 | The slow log can be viewed directly or via the `information_schema.SLOW_QUERY` table. 28 | 29 | The column definition is in [`slowQueryCols`](https://github.com/pingcap/tidb/blob/1521bf723dd023da655add0f883acaab5ee69683/pkg/infoschema/tables.go#L874-L956). 30 | 31 | The [`SlowQueryExtractor`](https://github.com/pingcap/tidb/blob/1521bf723dd023da655add0f883acaab5ee69683/pkg/planner/core/memtable_predicate_extractor.go#L1282) helps to extract some predicates of `slow_query`. 32 | 33 | A lot of the other logic can be found in [`slow_query.go`](https://github.com/pingcap/tidb/blob/master/pkg/executor/slow_query.go). 34 | 35 | This table has [`RuntimeStats`](https://github.com/pingcap/tidb/blob/2e51209f483bb7909be1eb0b55e5f18f0c437a25/pkg/executor/slow_query.go#L1114-L1119), which adds stats to `EXPLAIN ANALYZE` output. 36 | 37 | ### CLUSTER_SLOW_QUERY 38 | 39 | There is also the `information_schema.CLUSTER_SLOW_QUERY` table that combines the slow log from all nodes into a single table. 40 | 41 | The [TiDB Dashboard](https://docs.pingcap.com/tidb/stable/dashboard-slow-query) uses the `CLUSTER_SLOW_QUERY` table to display the slow queries in a webpage. 42 | 43 | ``` 44 | +----------------+ 45 | | TiDB Dashboard | 46 | | or App/CLI | 47 | +----------------+ 48 | | 49 | SQL Query 50 | CLUSTER_SLOW_QUERY 51 | | 52 | +--------+ 53 | +--gRPC-------| TiDB 1 |------gRPC--+ 54 | | +--------+ | 55 | +--------+ | +--------+ 56 | | TiDB 0 | | | TiDB 2 | 57 | +--------+ | +--------+ 58 | | | | 59 | +---------------+ +---------------+ +---------------+ 60 | | tidb-slow.log | | tidb-slow.log | | tidb-slow.log | 61 | +---------------+ +---------------+ +---------------+ 62 | ``` 63 | 64 | `CLUSTER_SLOW_QUERY` is using the `coprocessor` gRPC interface to send cop requests to the TiDB node. This allows the reuse of query condition push down and column prune, to reduce unnecessary data transfer. 65 | 66 | There are no `RuntimeStats` for this table, see [this issue](https://github.com/pingcap/tidb/issues/56707) for a request to add that. 67 | 68 | ## Documentation 69 | 70 | - User docs 71 | - [information_schema.slow_query](https://docs.pingcap.com/tidb/stable/information-schema-slow-query) 72 | - [Identify Slow Queries](https://docs.pingcap.com/tidb/stable/identify-slow-queries#query-the-number-of-slow-queries-for-each-tidb-node-in-a-cluster) 73 | -------------------------------------------------------------------------------- /src/understand-tidb/system-tables/introduction.md: -------------------------------------------------------------------------------- 1 | # System tables 2 | 3 | There are multiple types of system tables: 4 | 5 | - `mysql` schema: There are mostly real tables stored on TiKV 6 | - [`information_schema`](information_schema/introduction.md): These tables are generated on request and not stored on disk. 7 | - `performance_schema`: These tables are generated on request and not stored on disk. 8 | - `metrics_schema`: These are metrics from Prometheus, not real tables. 9 | - `SYS`: these are views etc to make `performance_schema` easier to use. 10 | 11 | See also the docs: 12 | - [TiDB mysql schema](https://docs.pingcap.com/tidb/stable/mysql-schema) 13 | - [TiDB information_schema](https://docs.pingcap.com/tidb/stable/information-schema) 14 | - [TiDB performance_schema](https://docs.pingcap.com/tidb/stable/performance-schema) 15 | - [TiDB sys schema](https://docs.pingcap.com/tidb/stable/sys-schema) 16 | - [TiDB metrics_schema](https://docs.pingcap.com/tidb/stable/metrics-schema) 17 | - [MySQL performance_schema](https://dev.mysql.com/doc/refman/8.4/en/performance-schema.html) 18 | - [MySQL information_schema](https://dev.mysql.com/doc/refman/8.4/en/information-schema.html) 19 | - [MySQL sys schema](https://dev.mysql.com/doc/refman/8.4/en/sys-schema.html) 20 | - [MySQL mysql schema](https://dev.mysql.com/doc/refman/8.4/en/system-schema.html) -------------------------------------------------------------------------------- /src/understand-tidb/transaction-on-tikv.md: -------------------------------------------------------------------------------- 1 | # Transaction on TiKV 2 | 3 | [The previous section](transaction.md) introduces the architecture of the transaction engine and several implementation details in TiDB part. This document is mainly about the TiKV part. 4 | 5 | As described in the previous section, the distributed transaction coordinator is `tidb-server` which receives and processes `COMMIT` query, and the transaction participants involved are `tikv-server`s. 6 | 7 | # Transactional Protocol 8 | 9 | The RPC interfaces in TiDB are defined in [a protobuf file](https://github.com/pingcap/kvproto/blob/0f5764a128ad77ccf0a5b0ce0d6e2bfa50a108ce/proto/kvrpcpb.proto#L77), based on the Percolator model. 10 | 11 | These interfaces will be used by the transaction coordinator to drive the whole commit process. For example `Prewrite` will be used to write the lock record in TiKV: 12 | 13 | ```proto 14 | message PrewriteRequest { 15 | Context context = 1; 16 | // The data to be written to the database. 17 | repeated Mutation mutations = 2; 18 | // The client picks one key to be primary (unrelated to the primary key concept in SQL). This 19 | // key's lock is the source of truth for the state of a transaction. All other locks due to a 20 | // transaction will point to the primary lock. 21 | bytes primary_lock = 3; 22 | // Identifies the transaction being written. 23 | uint64 start_version = 4; 24 | uint64 lock_ttl = 5; 25 | // TiKV can skip some checks, used for speeding up data migration. 26 | bool skip_constraint_check = 6; 27 | // For pessimistic transaction, some mutations don't need to be locked, for example, non-unique index key. 28 | repeated bool is_pessimistic_lock = 7; 29 | // How many keys this transaction involves in this region. 30 | uint64 txn_size = 8; 31 | // For pessimistic transactions only; used to check if a conflict lock is already committed. 32 | uint64 for_update_ts = 9; 33 | // If min_commit_ts > 0, this is a large transaction request, the final commit_ts 34 | // will be inferred from `min_commit_ts`. 35 | uint64 min_commit_ts = 10; 36 | // When async commit is enabled, `secondaries` should be set as the key list of all secondary 37 | // locks if the request prewrites the primary lock. 38 | bool use_async_commit = 11; 39 | repeated bytes secondaries = 12; 40 | // When the transaction involves only one region, it's possible to commit the transaction 41 | // directly with 1PC protocol. 42 | bool try_one_pc = 13; 43 | // The max commit ts is reserved for limiting the commit ts of 1PC or async commit, which can be used to avoid 44 | // inconsistency with schema change. 45 | uint64 max_commit_ts = 14; 46 | } 47 | ``` 48 | 49 | * `mutations` are changes made by the transaction. 50 | * `start_version` is the transaction identifier fetched from PD. 51 | * `for_update_ts` is used by the pessimistic transactions. 52 | * `try_one_pc` field is used if the transaction is committed using `one-phase` protocol. 53 | * `use_async_commit` and `secondaries` will be used if the transaction is committed in the `async-commit` mode. 54 | 55 | Besides `prewrite` request, there are some other important request types: 56 | 57 | * [`pessimistic_lock` request](https://github.com/pingcap/kvproto/blob/0f5764a128ad77ccf0a5b0ce0d6e2bfa50a108ce/proto/kvrpcpb.proto#L125) is used to lock a key. Note pessimistic locking happens in the transaction execution phase, for example a `select for update` statement will need to pessimistically lock the corresponding rows. 58 | * [`commit` request](https://github.com/pingcap/kvproto/blob/0f5764a128ad77ccf0a5b0ce0d6e2bfa50a108ce/proto/kvrpcpb.proto#L268) is used to commit a key. After commit the write content is visible to other read or write transactions. 59 | * [`check_txn_status` request](https://github.com/pingcap/kvproto/blob/master/proto/kvrpcpb.proto#L206) will be used to check the status of a given transaction, so that it could be decided how to process the conflicts. 60 | * [`resolve` request](https://github.com/pingcap/kvproto/blob/0f5764a128ad77ccf0a5b0ce0d6e2bfa50a108ce/proto/kvrpcpb.proto#L374) will be used to help doing the transaction crash recovery. 61 | * [`check_secondary_locks` request](https://github.com/pingcap/kvproto/blob/master/proto/kvrpcpb.proto#L247) is a special API, it will be used if the commit mode of the transaction is `async-commit`. 62 | 63 | # Transaction Scheduler 64 | 65 | ![transaction-scheduler](../img/transaction_scheduler.png) 66 | 67 | The receiving input transaction requests will be translated into transaction [commands](https://github.com/tikv/tikv/blob/6be3893f7f787b04bf34d99d1369092404ab5cfc/src/storage/txn/commands/mod.rs#L114). Then the transaction [scheduler](https://github.com/tikv/tikv/blob/6be3893f7f787b04bf34d99d1369092404ab5cfc/src/storage/txn/scheduler.rs#L286) will handle these transaction commands, it will first try to fetch the needed key [latches](https://github.com/tikv/tikv/blob/6be3893f7f787b04bf34d99d1369092404ab5cfc/src/storage/txn/latch.rs#L22) (latch is used to sequence all the transaction commands on the same key)，then try to fetch a storage [snapshot](https://github.com/tikv/tikv/blob/6be3893f7f787b04bf34d99d1369092404ab5cfc/components/tikv_kv/src/lib.rs#L191) for the current transaction processing. 68 | 69 | The task will be processed as a [future](https://doc.rust-lang.org/stable/std/future/trait.Future.html). The future processing is done in the transaction scheduler thread-pool. Usually, there will be some tasks like conflict and constraint checks, write mutation generations. For example, the prewrite request processing will need to check if there is already a conflict [lock](https://github.com/tikv/tikv/blob/6be3893f7f787b04bf34d99d1369092404ab5cfc/src/storage/txn/actions/prewrite.rs#L45) or a conflict committed write [record](https://github.com/tikv/tikv/blob/6be3893f7f787b04bf34d99d1369092404ab5cfc/src/storage/txn/actions/prewrite.rs#L59). 70 | 71 | # Transaction Log Replication 72 | 73 | In TiDB, the key space is split into different ranges or regions. Each region is a raft group and its leader will be responsible for handling its key range related read/write requests. 74 | 75 | If the transaction command processing in the transaction scheduler is successful, the generated transaction writes will be written into the raft log engine by the region leaders in `raftStore` (raft store will be introduced in other documents in details). The work flow is like this: 76 | 77 | ![RaftStore](../img/raftStore.png) 78 | 79 | The writes generated by transaction commands will be sent to the raft peer message task queue first, then the raft batch system will poll each raft peer and handle these requests in the raft thread-pool. After all the raft logs are persisted on majority raft group members, they are regarded as `commit`. Then the correspond apply task be delivered to the apply worker pool to apply the actual write contents to the storage engine, after that the transaction command processing is considered successful and the callback will be invoked to response `OK` results to the RPC client. 80 | 81 | 82 | # Transaction Record In TiKV 83 | 84 | In TiDB, a transaction is considered committed only if its primary key lock is committed successfully (if async commit protocol is not used). The actual key and value written into storage engine is in the following format: 85 | 86 | 87 | | CF | RocksDB Key | RocksDB Value | 88 | | ------- | :-------------------- | ------------- | 89 | | Lock | user_key | lock_info | 90 | | Default | {user_key}{start_ts} | user_value | 91 | | Write | {user_key}{commit_ts} | write_info | 92 | 93 | After `prewrite`, the lock correspond records for the transaction will be written into the storage. Read and write conflicts on the "locked" key will need to consider if it's safe to bypass the lock or it must try to resolve the encountered locks. As `commit_ts` is part of the stored key, there could be different historical versions for it, and GC is responsible to clean up all the old versions which will not be needed any more. GC mechanism will be introduced in another document. 94 | 95 | # Transaction Recovery 96 | 97 | In TiDB, the transaction coordinator (in `tidb-server`) is stateless and it will not persist any information. If the transaction coordinator fails for example the `tidb-server` crashes, the transaction context in memory will get lost, and as the coordinator is gone the normal commit processing will stop. How to recover the transaction state and make a decision if it should commit or abort? 98 | 99 | Actually, there is no special mechanism to recover the undetermined-status transactions, the recovery is done by other concurrent conflict transactions, or the conflict transactions will help decide the actual states of the undetermined-status transactions. The lock resolve process will be triggered if the current ongoing transaction encounters locks of other transactions doing reads or writes. The whole resolve process will be introduced in other documents in details. 100 | 101 | # Transaction Optimizations 102 | 103 | Normally the transaction commit will need two phases, the `prewrite` phase and `commit` phase. Under certain circumstances transaction commit could be done in a single phase for example the generated transaction mutations could be processed by a single region leader. This optimization is called one-phase commit in TiDB. 104 | 105 | The final transaction status is determined by the commit status of the primary key lock, so the response to the client has to wait until the primary key commit has finished. This wait could be saved using the `async-commit` protocol so the latency of commit processing could be reduced. 106 | 107 | They will both be introduced other documents in details. 108 | 109 | # Summary 110 | 111 | This section talks about the brief steps of transaction processing in the TiKV part, and related interfaces, implementations and optimizations. 112 | -------------------------------------------------------------------------------- /src/understand-tidb/transaction.md: -------------------------------------------------------------------------------- 1 | # Transaction 2 | 3 | The transaction engine in TiDB is responsible to provide ACID guarantees for all the read and write requests. It consists of the client/coordinator part in the TiDB repository and the server/participant part in the TiKV repository. 4 | 5 | This document is mainly about the TiDB part. 6 | 7 | ## The Architecture 8 | 9 | In TiDB the transaction write flow is like this: 10 | 11 | ![transaction-architecture](../img/transaction-architecture.png) 12 | 13 | After the transaction starts in a session, all the reads and writes will use a snapshot to fetch data, and the written content will be buffered in the memory of the transaction. When the `commit` statement is received from the client, the Percolator protocol will be used to commit these changes to the storage system. 14 | 15 | ## The Interface 16 | 17 | In TiDB, the [`Transaction`](https://github.com/pingcap/tidb/blob/af70762cd52519f025daa5e869ba37465a7fb311/kv/kv.go#L181-L234) interface defines the commonly used transaction operations. 18 | 19 | ```go 20 | type Transaction interface { 21 | // Commit commits the transaction operations to KV store. 22 | Commit(context.Context) error 23 | // Rollback undoes the transaction operations to KV store. 24 | Rollback() error 25 | // LockKeys tries to lock the entries with the keys in KV store. 26 | // Will block until all keys are locked successfully or an error occurs. 27 | LockKeys(ctx context.Context, lockCtx *LockCtx, keys ...Key) error 28 | // SetOption sets an option with a value, when val is nil, uses the default 29 | // value of this option. 30 | SetOption(opt int, val interface{}) 31 | // GetOption returns the option 32 | GetOption(opt int) interface{} 33 | ... 34 | // StartTS returns the transaction start timestamp. 35 | StartTS() uint64 36 | // Valid returns if the transaction is valid. 37 | // A transaction become invalid after commit or rollback. 38 | Valid() bool 39 | // GetMemBuffer return the MemBuffer binding to this transaction. 40 | GetMemBuffer() MemBuffer 41 | // GetSnapshot returns the Snapshot binding to this transaction. 42 | GetSnapshot() Snapshot 43 | } 44 | ``` 45 | 46 | These are common interfaces the transaction will provide. 47 | 48 | For example, `Commit` will be used to commit the current ongoing transaction. The transaction is considered ongoing before the `Commit` operation is triggered. The two-phase commit processing will be used to commit a transaction and it will finally become committed or aborted. 49 | 50 | [`LazyTxn`](https://github.com/pingcap/tidb/blob/af70762cd52519f025daa5e869ba37465a7fb311/session/txn.go#L50-L71) is a wrapper of the transaction implementations, when the SQL statements are executed using a standalone session context, `LazyTxn` will be used to do things like： 51 | 52 | - Return the memory buffer for write. 53 | - Set specific operations or flags for the current transaction. 54 | - Return the snapshot of this transaction. 55 | - Commit the current transaction. 56 | - Lock specific keys. 57 | 58 | ## The Statement Execution 59 | 60 | Usually, the first thing that will be done executing a statement is to `activate` the related transaction 61 | 62 | By default, TiDB provides the snapshot isolation level. Thus, in each new transaction, a new global strong snapshot will be fetched first, before executing statements. 63 | 64 | In TiDB, the snapshot is represented by a global TSO which is fetched from the PD server, and it acts as the unique identifier for this transaction. After this operation, a transaction is regarded as `activated`. 65 | 66 | For the read SQL statements, the [snapshot](https://github.com/pingcap/tidb/blob/af70762cd52519f025daa5e869ba37465a7fb311/store/driver/txn/snapshot.go) will be used to provide a global strong consistent snapshot, all the reads will check data visibility using this snapshot. Most executors will set the timestamp doing the build, and the transaction could be activated by the building process. Some commonly used [snapshot](https://github.com/pingcap/tidb/blob/af70762cd52519f025daa5e869ba37465a7fb311/store/driver/txn/snapshot.go#L40) APIs are as below: 67 | 68 | ```go 69 | // BatchGet gets all the keys' value from kv-server and returns a map contains key/value pairs. 70 | // The map will not contain nonexistent keys. 71 | func (s *tikvSnapshot) BatchGet(ctx context.Context, keys []kv.Key) (map[string][]byte, error) { 72 | data, err := s.KVSnapshot.BatchGet(ctx, toTiKVKeys(keys)) 73 | return data, extractKeyErr(err) 74 | } 75 | 76 | // Get gets the value for key k from snapshot. 77 | func (s *tikvSnapshot) Get(ctx context.Context, k kv.Key) ([]byte, error) { 78 | data, err := s.KVSnapshot.Get(ctx, k) 79 | return data, extractKeyErr(err) 80 | } 81 | ``` 82 | 83 | For the write SQL statements, they will write data into the transaction memory buffer temporarily until the `commit` operation is triggered. There are 3 main interfaces which will write query data into the memory buffer. Here is the [table](https://github.com/pingcap/tidb/blob/af70762cd52519f025daa5e869ba37465a7fb311/table/table.go#L166) API: 84 | 85 | ```go 86 | // Table is used to retrieve and modify rows in table. 87 | type Table interface { 88 | // AddRecord inserts a row which should contain only public columns 89 | AddRecord(ctx sessionctx.Context, r []types.Datum, opts ...AddRecordOption) (recordID kv.Handle, err error) 90 | 91 | // UpdateRecord updates a row which should contain only writable columns. 92 | UpdateRecord(ctx context.Context, sctx sessionctx.Context, h kv.Handle, currData, newData []types.Datum, touched []bool) error 93 | 94 | // RemoveRecord removes a row in the table. 95 | RemoveRecord(ctx sessionctx.Context, h kv.Handle, r []types.Datum) error 96 | } 97 | ``` 98 | 99 | Every statement will use a `staging` buffer during its execution. If it's successful, the staging content will be merged into the transaction memory buffer. 100 | 101 | For example, `AddRecord` will try to write a row into the current statement staging buffer, and the `RemoveRecord` will try to remove a row from the staging statement buffer. The existing transaction memory buffer will not be affected if the statement has failed. 102 | 103 | The memory buffer implementation is wrapped in [memBuffer](https://github.com/pingcap/tidb/blob/af70762cd52519f025daa5e869ba37465a7fb311/store/driver/txn/unionstore_driver.go#L27-L29). The internal implementation is [`MemDB` struct](https://github.com/tikv/client-go/blob/4fc565e203a99400d0b080a25a93fb860b3b6fd6/internal/unionstore/memdb.go). 104 | 105 | The memory buffer is an ordered map and it provides the `staging` and `discard` utilities. For example, the memory content generated by a statement will be discarded if its execution has failed. 106 | 107 | ## The Two-phase Commit 108 | 109 | After the statement execution phase, the `commit` statement will trigger the commit execution for the current transaction. In TiDB, the Percolator protocol is used as the distributed transaction protocol, it's a two-phase protocol. 110 | 111 | In the first stage, the transaction coordinator (TiDB server) will try to prewrite all the related keys. If all of them are successful, the transaction coordinator will then commit the primary key. After that the transaction is considered committed successfully, and a success message will be responded to the client. All the other keys will be committed asynchronously. 112 | 113 | All the changes in the transaction memory buffer will be converted into [mutations](https://github.com/pingcap/kvproto/blob/dc1709169bb155de3bea6b28c871215387942994/proto/kvrpcpb.proto#L882) which will be used by the two-phase committer. These mutations will be grouped by their region locations, and prewrite requests will be sent to their region leaders. 114 | 115 | If all the prewrite requests are processed successfully, the commit request for the primary key will be sent to TiKV first. If the primary key commit is successful, the transaction is considered committed and will respond to the client with successful results. 116 | 117 | ## Recommended Reading 118 | 119 | * [Percolator](https://tikv.org/deep-dive/distributed-transaction/percolator/) 120 | * [Large-scale Incremental Processing Using Distributed Transactions and Notifications](https://research.google/pubs/pub36726/) 121 | --------------------------------------------------------------------------------