├── .gitignore ├── 0000-template.md ├── README.md └── rfc ├── 0001-rfc-process.md ├── 0002-subdocapi.md ├── 0003-indexmanagement.md ├── 0004-at_plus.md ├── 0005-vbucket-retries.md ├── 0007-cluster_level_auth.md ├── 0008-datastructures.md ├── 0010-cbft.md ├── 0011-connection-string.md ├── 0013-kv-error-map.md ├── 0016-rbac.md ├── 0020-common-flags.md ├── 0022-usermgmt.md ├── 0024-fast-failover.md ├── 0025-subdoc-xattr.md ├── 0026-ketama-hashing.md ├── 0027-analytics.md ├── 0028-enhanced_error_messages.md ├── 0030-compression.md ├── 0032-field-level-encryption.md ├── 0034-health-check.md ├── 0035-rto.md ├── 0036-client-cert-auth.md ├── 0048-sdk3-bootstrapping.md ├── 0049-sdk3-retry-handling.md ├── 0050-sdk3-datastructures.md ├── 0051-sdk3-views.md ├── 0052-sdk3-full-text-search.md ├── 0053-sdk3-crud.md ├── 0054-sdk3-management-apis.md ├── 0055-serializers-transcoders.md ├── 0056-sdk3-query.md ├── 0057-sdk3-analytics.md ├── 0058-error-handling.md ├── 0059-sdk3-foundation.md ├── 0061-sdk3-diagnostics.md ├── 0064-sdk3-field-level-encryption.md ├── 0069-kv-error-map-v2.md ├── 0071-http-client.md ├── 0074-config-profiles.md ├── 0076-subdoc-replica-reads.md ├── 0084-app-service-level-telemetry.md ├── figures ├── 0058-aggregate-exception.png ├── 0058-general-design.png ├── rfc30-fig1.png ├── rfc32-fig1.PNG ├── rfc35-fig1.png ├── rfc35-fig2.png ├── rfc36-fig1.png └── rfc55-uml1.png └── ketama-hashes.json /.gitignore: -------------------------------------------------------------------------------- 1 | **/*.DS_Store 2 | -------------------------------------------------------------------------------- /0000-template.md: -------------------------------------------------------------------------------- 1 | *MetaMeta:* Usually RFCs start off as GDrive Docs and then become Markdown as they're completed. There is a [handy GDrive Template too](https://docs.google.com/document/d/1z3doehquTjXDL6mimQH3Eyp7pv80r0tsPebvtx59j9A/edit#heading=h.pja6dcwixhvq). 2 | 3 | # Meta 4 | 5 | - RFC Name: [TBD: unique feature name] 6 | - RFC ID: [TBD: pick the next id] 7 | - Start Date: TBD: todays date, YYYY-MM-DD] 8 | - Owner: [TBD: your name] 9 | - Current Status: TBD: DRAFT -> REVIEW -> ACCEPTED] 10 | 11 | # Summary 12 | [TBD: A short paragraph summary of the SDK-RFC. Something appropriate for putting in a status dashboard.] 13 | 14 | # Motivation 15 | [TBD: Why are we doing this? What use cases does it support? What is the expected outcome?] 16 | 17 | # General Design 18 | [TBD: This part must cover the non-SDK specific API design. Concepts should be abstracted from an actual implementation, using some form of "pseudo" language that describes the concept rather than the actual language implementation. 19 | This part also covers the unified behavior that the user sees. What errors does the user see? If user does X, what happens on Y? 20 | What is the scope? What are the main components? How do they interact? What new interfaces are created? How are they presented in each of the platforms in scope? How will you know when you are done?] 21 | 22 | ## Errors 23 | [TBD: What new error handling needs to be considered] 24 | 25 | ## Documentation 26 | [TBD: What documentation artifacts need to be created or changed in support of this RFC] 27 | 28 | # Language Specifics 29 | In this section, the abstract design parts need to be broken down on the SDK level by each maintainer and signed off eventually. 30 | 31 | | Generic | .NET | Java | NodeJS | Go | C | PHP | Python | Ruby | 32 | | ------- | ------------ | ------ | ------ | -- | - | --- | ------ | ---- | 33 | | foo() | foo | Foo | ...... | .. | . | ... | ...... | .... | 34 | 35 | # Questions 36 | [TBD: A running log of larger questions, unresolved at first and later kept in place with their answers if the owner considers it useful to keep the background around.] 37 | 38 | # Changelog 39 | [TBD: YYYY-MM-DD 40 | Initial draft publication 41 | ] 42 | 43 | # Signoff 44 | If signed off, each representative agrees both the API and the behavior will be implemented as specified. 45 | 46 | | Language | Representative | Date | 47 | | -------- | -------------- | ---------- | 48 | | Java | Michael N. | 01.01.1960 | 49 | -------------------------------------------------------------------------------- /rfc/0001-rfc-process.md: -------------------------------------------------------------------------------- 1 | # Meta 2 | 3 | - RFC Name: The RFC Process 4 | - RFC ID: 0001 5 | - Start Date: 2015-10-23 6 | - Owner: Michael Nitschinger 7 | - Current Status: Review 8 | 9 | # Summary 10 | This RFC describes the newly established RFC process. 11 | 12 | # Motivation 13 | Since the inception of the "SDK", we struggled with inconsistencies between the different language implementations. Many different reasons are contributing to it (legacy codebases, customer escalations, forgetting to communicate,...), but in the end there are two noticable problems to the user: 14 | 15 | - Syntactical API inconsistencies 16 | - Semantical behavior differences in cluster communication 17 | 18 | Both lead to user dissatisfaction. The first one is often caught during development, the second one very often hits the user in production. In order to provide a better user experience, we should strive to improve our communication and synchronization activities. 19 | 20 | Note that this is different from PRDs that PM creates. These RFCs should get into the weeds and nitty gritty details of technical implementation and behavior. 21 | 22 | The RFC process is designed to be lightweight, collaborative and effective. Doing this in an open forum also allows easy access and comments by external contributors. 23 | 24 | Before creating an RFC, it needs to be clear if an RFC is actually needed. The following describes each situation where an RFC is mandatory: 25 | 26 | - SDK behavior or API changes when interacting with the User OR the Server. So every time either the user or the server communication changes, there is a high likelihood other SDKs have a similar impact. This explicitly includes utility classes 27 | which are very likely to be needed on other SDKs. 28 | - The RFC process itself needs to be changed. 29 | - Existing inconsistencies that need to be resolved across the board. 30 | 31 | If a prototype is already ongoing, it makes sense to drive the RFC alongside the prototype, but this is not a requirement. 32 | 33 | Two examples: if the java client includes a helper class for retry semantics on rxjava observables (to reduce the verbosity in writing it manually), this does not have an equivalent on any other SDK and therefore doesn't need a RFC. If the java client introduces a helper class for LDAP support, this does have an impact on the other SDKs and needs a RFC. 34 | 35 | When it is not clear if an RFC needs to be created, a quick email or discussion in the team should make it clear whether the change has an overall impact or not. 36 | 37 | **An RFC MUST be signed off by each SDK representative and consensus must be reached. Once the RFC is accepted, each SDK MUST implement it according to the RFC. If, after acceptance issues show up during implementation, the MUST lead to a revision of the RFC which again needs to be signed off by everyone.** 38 | 39 | # General Design 40 | 41 | 1. Look at latest issue to determine the RFC number (4 digits) - the `RFCID` is in the form *`rfcNumber`***-***`rfcShortName`* 42 | 2. open an issue titled `RFCID`, short description of the RFC, does the owner think it requires sdk specifics, etc... tag as appropriate (if we decide to put tags in). the issue allows to "reserve" the RFC number prior to any file being added publicly to the repo. 43 | 3. create the RFC branch: `drafts/RFCID` 44 | 4. create the RFC file as `RFCID.md`in `/rfc` directory 45 | 5. start the discussion by doing a Pull Request to master 46 | 6. **phase one** iterates on the global idea + "proof of concept" sdk specifics proposed by the owner 47 | 7. *end of this phase should probably be marked by a first informal round of votes?* 48 | 8. **phase two** iterates on SDK specifics: each SDK team/representative can then push commits in the branch to improve on his own SDK specifics 49 | 9. once everyone has contributed SDK specifics, the RFC must be signed off by a message following this convention: 50 | 51 | ``` 52 | rfc is (:+1:|:-1:) for `teamname` *anything after is a vote comment* 53 | ``` 54 | 55 | examples: 56 | 57 | ``` 58 | > rfc is :-1: for `java` - I'm not happy with the java specifics after toying with it 59 | > rfc is :+1: for `python` 60 | ``` 61 | 62 | Once content is :+1: by all teams, merge into master using the following strategy (it retains discussion on intermediate commits, but produces a linear history in master): 63 | 64 | - `git merge --squash -e drafts/RFCID` 65 | - `git commit` 66 | - commit message with 67 | - user that signed off for each `team` 68 | - link to PR: `Close #XX (see link for discussion)` (also closes the pr since github won't detect a merge --squash) 69 | - `Close #YY` (to close the issue) 70 | 71 | See the template in 0000-template.md. 72 | 73 | # Language Specifics 74 | Not applicable in this RFC. 75 | 76 | # Signoff 77 | If signed off, each representative agrees both the API and the behavior will be implemented as specified. 78 | 79 | Note that this signoff includes every SDK team member. 80 | 81 | | Language | Representative | Date | 82 | | -------- | -------------- | ---------- | 83 | | - | Michael N. | 2015-12-03 | 84 | | - | Simon B. | 2015-12-03 | 85 | | - | Brett L. | 2015-12-09 | 86 | | - | Jeff M. | 2015-12-11 | 87 | | - | Sergey A. | 2015-12-11 | 88 | | - | Mark N. | 2015-12-09 | 89 | | - | Todd G. | not voted | 90 | | - | Matt I. | not voted | 91 | -------------------------------------------------------------------------------- /rfc/0004-at_plus.md: -------------------------------------------------------------------------------- 1 | # Meta 2 | 3 | - RFC Name: AT_PLUS Consistency for N1QL Queries 4 | - RFC ID: 0004-at_plus 5 | - Start Date: 2015-12-16 6 | - Owner: Michael Nitschinger (@daschl) 7 | - Current Status: Accepted (2016-06-07) 8 | 9 | # Summary 10 | This RFC specifies the user API and interaction with the server for enhanced 11 | RYOW query performance (practically known as `AT_PLUS` with "Mutation Tokens"). 12 | 13 | # Motivation 14 | The current approach to perform RYOW ("Read Your Own Writes") queries is to use the 15 | `REQUEST_PLUS` scan consistency when executing a N1QL query. While this is 16 | semantically correct, it takes the performance hit of having to pick the absolute 17 | latest sequence number in the system and as a result potentially waiting for 18 | a longer time until the indexer has caught up under high mutation rates. 19 | 20 | One way to mitigate this is to only wait until specific mutations have 21 | been indexed instead of waiting for all of them up to the latest one. Also, the 22 | likelihood of the indexer already having indexed the mutation is directly 23 | proportional to the time delta between mutation and subsequent query. This also 24 | means that a performance win will be marginal if the `AT_PLUS` query immediately 25 | follows the mutations, in this case the programmatic and runtime overhead of using 26 | the tokens might not be worth the tradeoff. 27 | 28 | `AT_PLUS` is treated as a pure optimization over `REQUEST_PLUS` and has no semantic 29 | difference to it in terms of correctness inside the bounds specified through the 30 | mutation tokens. One cannot use `REQUEST_PLUS` and `AT_PLUS` together in the same 31 | query. Note that as described further below, the main tradeoff here is increased 32 | network utilization because the client needs to send more information to the server 33 | as part of the query request. 34 | 35 | # General Design 36 | In the big picture of `AT_PLUS` for the developer, there are two components that 37 | must be considered: 38 | 39 | 1. Mutations are executed on a bucket. They might or might not include 40 | `MutationTokens` as part of the response, depending on the SDK configuration 41 | (at this point, enabling tokens is optional since it leads to larger 42 | response bodies but is mandatory for this RFC to work). 43 | 2. The application state after the mutation happened needs to be applied at 44 | N1QL query time. 45 | 46 | This RFC does not focus on 1. since it has been implemented already as part of 47 | the enhanced durability requirements initiative in the 4.0 timeframe. This RFC 48 | assumes that the Mutation Token is available as a return value from 49 | a mutation operation, most likely exposed on the `Document` or equivalent or 50 | stored internally (see implementation details). `AT_PLUS` is not possible without 51 | `MutationTokens`. 52 | 53 | This RFC proposes an extension to the current `MutationToken` to also 54 | include the bucket name. This is important so that the 55 | token can be properly namespaced at query time to remove ambiguity in 56 | cross-bucket query scenarios and when transported between applications. 57 | 58 | Since it is important to this RFC, the properties of a `MutationToken` are as 59 | follows: 60 | 61 | 1. A `MutationToken` is opaque to the user and needs to be treated as such. 62 | 2. A `MutationToken` is immutable after creation. 63 | 3. A `MutatonToken`'s cannot be serialized/deserialized directly. 64 | 4. Numerous `MutationToken`'s must be aggregated into a `MutationState` which is 65 | serializable. 66 | 67 | The internal structure of a `MutationToken` looks like this: 68 | 69 | ``` 70 | MutationToken { 71 | [required, immutable] vbucket_id = long 72 | [required, immutable] vbucket_uuid = long 73 | [required, immutable] sequence_number = long 74 | [required, immutable] bucket_name = string 75 | } 76 | ``` 77 | 78 | The `MutationState` exposes the following methods used to aggragate multiple `MututationTokens` into a single unit. It can then be serialized/deserialized as well and passed to the `consistent_with` method for a N1Ql query. 79 | 80 | ``` 81 | MutationState { 82 | static from(docs: Document...) -> MutationState 83 | static from(fragments: DocumentFragment...) -> MutationState 84 | 85 | add(docs: Document...) -> MutationState 86 | add(fragments: DocumentFragment...) -> MutationState 87 | add(state: MutationState) -> MutationState 88 | } 89 | ``` 90 | 91 | The following behavior is defined for the methods based on the contents of the passed MutationState: 92 | 93 | - When called with a list of `Documents`/`Fragments` and the `MutationTokens` are 94 | present on the `Document`, then they are extracted and used as part of the query. 95 | - Subsequent calls to either a previously captured `Document`/`Fragment` 96 | reference must override the old one if the new one has a higher sequence number. 97 | There must not be duplicates values for any particular bucket and vbid pair. 98 | 99 | The N1QL query execution for AT_PLUS consists of the following user level API: 100 | 101 | ``` 102 | consistent_with(mutationState: MutationState) 103 | ``` 104 | 105 | Setting `SCAN_CONSISTENCY` to anything but `AT_PLUS` is a runtime 106 | error (see errors). If the SDK already exposes `AT_PLUS` is must be explicitly set, if 107 | `AT_PLUS` is not yet exposed it should be kept internal since there is no actual benefit 108 | in exposing it. If so, the user only specifies the `consistent_with` 109 | boundaries and the SDK needs to figure out the rest (see implementation). 110 | 111 | ## Internal implementation 112 | The N1QL Query API specifies the following API to be used for `AT_PLUS`: 113 | 114 | ``` 115 | {"scan_vectors": { 116 | "bucket1_name": { 117 | "": [, ""], 118 | "": [, ""] 119 | }, 120 | "bucket2_name": { 121 | "": [, ""] 122 | } 123 | }, 124 | "scan_consistency": "at_plus", 125 | // other optional query options... 126 | } 127 | ``` 128 | 129 | The `scan_vectors` object contains one entry for each bucket that takes part in this query. 130 | Inside the bucket scope all `MutationTokens` that need to be part of the index boundary are put in. 131 | **Be careful** that the `vbucket_uuid` and `vbucket_id` are strings while the `sequence_number` is a JSON number. 132 | 133 | Once the query is assembled, the Mutation Tokens are marshaled into their JSON 134 | representation (through the `MutationState`) and passed alongside all the other params to the N1QL query 135 | engine. 136 | 137 | The following edge cases need to be considered in the implementation: 138 | 139 | - `Document`/`Fragment`s that are passed in without a token set must fail at runtime immediately. 140 | - If the user passes in more than one `MutationToken` for a unique vbucket ID, the SDK needs to automatically pick the one with the higher sequence number, making sure that the "lower boundary" is satisfied automatically. This issue can come up in the case where multiple mutation tokens on the same ID are loaded and passed in subsequently to the same query. 141 | 142 | 143 | ## Serialization and Deserialization of MutationState 144 | In order to cover the use case where `MutationToken`s need to be shared across applications, the tokens need to be converted into a common format. 145 | 146 | This is done at the `MutationState` level (aggregated MutationToken's). How the marshalling is done depends on the language, but the wire format is defined in this RFC. This format is identical to the expected wire-format as received by the N1QL service as state above in this RFC. 147 | 148 | As an example, if the following two `MutationTokens` are stored in the `MutationState` and it is exported: 149 | 150 | - MutationToken(bucket=default, vbid=1, seqno=1, vbuuid=1234) 151 | - MutationToken(bucket=beer-sample, vbid=25, seqno=10, vbuuid=5678) 152 | 153 | ```json 154 | { 155 | "default": { 156 | "1": [1, "1234"] 157 | }, 158 | "beer-sample": { 159 | "25": [10, "5678"] 160 | } 161 | } 162 | ``` 163 | 164 | This ensures that a `MutationState` can be loaded from this exact JSON back into an object and then used as part of an `AT_PLUS` query. 165 | 166 | ## Errors 167 | The following user errors need to be covered by the SDKs: 168 | 169 | | Error | Behavior | 170 | | ----- | -------- | 171 | | Other `SCAN_CONSISTENCY` than `AT_PLUS` specified in addition to `consistent_with` | IllegalArgumentException or equivalent thrown at runtime | 172 | | `Document`/`Fragment` passed in with no mutation token set | IllegalArgumentException or equivalent thrown at runtime | 173 | | `consistent_with` is used but mutation tokens are not enabled on the config | IllegalArgumentException or equivalent thrown at runtime | 174 | 175 | # Language Specifics 176 | 177 | ## Setting Scan Consistency with Document 178 | 179 | - **Generic:** `consistent_with(mutationState: MutationState)` 180 | - **C:** 181 | 182 | ```c 183 | lcb_n1p_setconsistent_handle(lcb_N1QLPARAMS*, const lcb_t); 184 | lcb_n1p_setconsistent_token(lcb_N1QLPARAMS*, const lcb_MUTATION_TOKEN*); 185 | token = lcb_resp_get_muation_token(int cbtype, const lcb_RESPBASE* resp); 186 | ``` 187 | - **Go:** `func (nq *N1qlQuery) ConsistentWith(state *MutationState) *N1qlQuery` 188 | - **Java:** `N1qlParams consistentWith(MutationState mutationState)` 189 | - **.NET:** `ConsistentWith(MutationState.From(document1, document2, document3));` 190 | - **NodeJS:** `query.consistentWith(new MutationState(document1, document2, document3));` 191 | - **Python:** 192 | 193 | ```python 194 | ms = MutationState() 195 | ms.add_results(*results) # A `Result` is the Python equivalent of a Document 196 | q.consistent_with(ms) 197 | ``` 198 | - **PHP:** 199 | 200 | ```php 201 | # $source is either CouchbaseMetaDoc or CouchbaseDocumentFragment 202 | MutationState::from($source, ...) #-> MutationState 203 | 204 | $ms = new MutationState() 205 | # $source is either CouchbaseMetaDoc or CouchbaseDocumentFragment, 206 | # or another MutationState 207 | $ms.add($source, ...) #-> MutationState 208 | $q.consistent_with($ms) 209 | ``` 210 | - **Ruby:** *not included* 211 | 212 | # Unresolved Questions 213 | None. 214 | 215 | # Changelog 216 | - 2016-05-03: Clarified Serialization wire format with example, polishing and rewording. Weakened the constraint on the enum so that .NET can keep it's AT_PLUS enum variant exposed. 217 | - 2016-03-25: Moved into REVIEW state as discussed (michael n.) 218 | 219 | # Signoff 220 | 221 | | Language | Representative | Date | 222 | | -------- | -------------- | ---------- | 223 | | C | Mark N.| May 3rd, 2016 | 224 | | Go | Brett L.| May 3rd, 2016 | 225 | | Java | Michael N.| May 3rd, 2016 | 226 | | .NET | Jeff M.| May 3rd, 2016| 227 | | NodeJS | Brett L.| May 3rd, 2016 | 228 | | Python | Mark N.| May 3rd, 2016 | 229 | | PHP | Sergey A. | May 13th, 2016 | 230 | -------------------------------------------------------------------------------- /rfc/0005-vbucket-retries.md: -------------------------------------------------------------------------------- 1 | # Meta 2 | 3 | - RFC Name: VBucket Retry Logic 4 | - RFC ID: 0005-vbucket-retries 5 | - Start Date: 2016-01-12 6 | - Owner: Brett Lawson (@brett19) 7 | - Current Status: Accepted (2016-06-15) 8 | 9 | # Summary 10 | This RFC defines a consistent implementation of vbucket retry logic for all Couchbase Server SDKs. 11 | 12 | # Motivation 13 | In the existing suite of client libraries there is no consistent behaviour defined in terms of how to perform vbucket retries. This has resulted in a variety of methods being employed, each with its own benefits and drawbacks. We now need to align all the SDKs to have similar behaviour to ensure that performance expectations do not vary based on SDK. 14 | 15 | # General Design 16 | All client libraries must implement NMV status code interception to perform automatic retry. Retry logic should be trivial and only consider the current map and fast-forward map which is available within our bucket configuration. All operations should be attempted based on the current map, if the operation fails with NMV for the current map, then we should retry that operation using either the fast-forward map, or again with the current map should a fast-forward map not exist, every X milliseconds (100ms default, tuneable). Note that operations for any particular vbuckets must first be attempted on the current map, meaning the map that may have changed since the last try, prior to being attempted on the fast-forward map. Additionally, once an operation is retried on the fast-forward map reverting to the current map should not occur unless a configuration change is detected which indicates that the current map is once again valid. When a new map is received, the operation should be dispatched as soon as reasonable as defined by the SDK, this must be under under the tuneable value (default: 100ms), however it could be scheduled immediately. 17 | 18 | This will result in an operation following these steps: 19 | 20 | 1. Dispatch Operation using current map 21 | 1. If not NMV, DONE 22 | 1. If Fast-Forward map exists, go to Step 6 23 | 1. Sleep 100ms 24 | 1. Go to step 1 25 | 1. Dispatch operation using fast forward map 26 | 1. If not NMV, DONE 27 | 1. Sleep 100ms 28 | 1. If received new config revision, GO to Step 1 29 | 1. Go to step 6 30 | 31 | ### Why 100ms linear retry? 32 | The reason 100ms was chosen was because, it seemed conservative enough that it would definitely throttle the client talking to the network, so we don't end up in a pathological situation. But it is aggressive enough that a normal application probably would not notice the latency from a retry. The reason behind not making it exponential, because of the rate of new incoming requests, if it is on a per-request basis, incoming operations could cause a pathological state where incoming requests continue to build on the load already causing the server to fail requests. 33 | 34 | ## Errors 35 | No errors will be changed by this proposal. NMV replies MUST be caught and handled within the SDKs (never leaked to users). 36 | 37 | # Language Specifics 38 | 39 | No language specific API's or implementation details are required. 40 | 41 | # Unresolved Questions 42 | None. 43 | 44 | # Changelog 45 | - 2016-05-06: Clarified behaviour of operations following a new configuration change being received by the SDK. (@brett19) 46 | 47 | # Signoff 48 | 49 | | Language | Representative | Date | 50 | | -------- | -------------- | ---------- | 51 | | C | Mark N.| May 6th, 2016 | 52 | | Go | Brett L.| May 6th, 2016 | 53 | | Java | Michael N.| May 17th, 2016 | 54 | | .NET | Jeff M.| May 6th, 2016| 55 | | NodeJS | Brett L.| May 6th, 2016 | 56 | | Python | Mark N.| May 6th, 2016 | 57 | | PHP | Sergey A. | May 19th, 2016 | 58 | -------------------------------------------------------------------------------- /rfc/0007-cluster_level_auth.md: -------------------------------------------------------------------------------- 1 | # Meta 2 | * RFC Name: Cluster Level Authentication 3 | * RFC ID: 0007 4 | * Start Date: 2016-01-12 5 | * Owner: Brett Lawson (@brett19) 6 | * Current Status: Accepted 7 | 8 | # Summary 9 | This RFC defines an implementation of cluster level authentication for the 2.x series of Couchbase Server client libraries. 10 | 11 | # Motivation 12 | In the current implementation of client libraries, authentication is provided on an as-needed basis. This means that a password is provided to the OpenBucket method, and this password is used throughout the clients to authenticate to various cluster services. In addition, when a new manager object is created for a Cluster or Bucket, the credentials are provided at creation time. The addition of cluster-level queries, which currently rely on the credentials of multiple buckets to execute, as well as the inclusion of more advanced form of cluster level authentication such as RBAC or CBAUTH have made it clear that an improved client authentication model is necessary. 13 | 14 | # General Design 15 | This specification proposes the addition of a high level and generic 'Authenticator' interface, which would be passed to a Cluster object by virtue of an 'Authenticate' method. In addition to this generic interface, this specification will define the behaviour for an authenticator object which implements this interface according to the current authentication behaviour of Couchbase Server (PasswordAuthenticator). 16 | 17 | The Authenticator interface is intended to be completely generic. No expectations are set by this RFC in regards to the methods that it exposes or the internal implementation of this Authenticator. The Authenticator MUST provide the ability for the client to fetch credentials for the following cases: bucket-level kv, bucket-level views, bucket-level management, cluster-level management, bucket-level n1ql, cluster-level n1ql, bucket-level cbft or cluster-level cbft. These credential requests may require only a single username/password pair to be returned (such as for bucket kv), or may require a list of username/password pairs to be returned (such as for cluster-level N1QL queries). The ability to fetch these tokens MUST NOT be exposed directly to the user (private methods). In addition, it should be noted that the Authenticator's internal behaviour is expected to change through sdk releases and MUST NOT be exposed publicly, as it may change even within patch versions. 18 | 19 | The ClassicAuthenticator object is meant to expose our current paradigm of authentication behaviour. It MUST store a list of bucket/password pairs, along with an optional cluster management username and password (Administrator/…). When performing a bucket level authentication (kv, views, bucket-level n1ql query, etc…), the authenticator MUST return only the credential that matches the specified bucket from its list of bucket/password pairs. When performing cluster-level management operations, the cluster management user/password MUST be returned. When performing cluster-level operations, the entire list of bucket/password pairs MUST be returned. 20 | 21 | Note that in all current cases where an SDK has provided the user with 'default credentials' (for instance, calling OpenBucket without a password), the SDK MUST now make an attempt to satisfy these credentials through the Authenticator interface. For backwards compatibility, any SDK which currently has method overloads for passing passwords MUST use the password specified at the call-site over any passwords stored in the authenticator. In this case, the SDK MUST NOT make changes to any user-defined Authenticator objects. 22 | 23 | The following is a code example of a possible implementation: 24 | 25 | ```go 26 | type Authenticator interface { 27 | // Note that the following is just an example, in this case, simply returning 28 | // a username/password combination to use for the specific context. 29 | getCredentials(context string, specific string) (string,string, error) 30 | } 31 | 32 | Type ClassicAuthenticator struct { 33 | Buckets map[string]string 34 | Username string 35 | Password string 36 | } 37 | 38 | func (auth *ClassicAuthenticator) getCredentials(context string, specific string) ([][]string, error) { 39 | switch (context) { 40 | case "bucket-kv": 41 | return [][]string{ []string{ specific, auth.Buckets[specific] } }, nil 42 | case "bucket-n1ql": 43 | return [][]string{ []string{ specific, auth.Buckets[specific] } }, nil 44 | case "cluster-n1ql": 45 | return auth.Buckets, nil 46 | case "cluster-cbft": 47 | return auth.Buckets, nil 48 | case "cluster-mgmt": 49 | return [][]string{ []string{ auth.clusterUser, auth.clusterPass } }, nil 50 | } 51 | return nil, ErrAuthError 52 | } 53 | 54 | func (cluster *Cluster) Authenticate(authObj Authenticator) { 55 | cluster.auther = authObj 56 | } 57 | ``` 58 | 59 | The following is a code example of a possible usage: 60 | 61 | ```go 62 | auth := gocb.ClassicAuthenticator{ 63 | Buckets: gocb.BucketAuthenticatorMap{ 64 | "default": "default_password", 65 | "travel-sample": "go go gadet password" 66 | }, 67 | Username: "Administrator", 68 | Password: "Charles Darwin", 69 | } 70 | 71 | cluster := gocb.Connect(...) 72 | cluster.Authenticate(auth) 73 | 74 | bucket := cluster.OpenBucket("default") 75 | // Look ma', I have an authenticated bucket connection! 76 | ``` 77 | 78 | # Errors 79 | No new errors will be introduced by the introduction of the authenticator objects, however it is reasonable to assume that the addition of future authenticators may surface new errors. 80 | 81 | # Language Specifics 82 | None known. 83 | 84 | # Unresolved Questions 85 | - Perhaps the Authenticator class should instead be named SecurityContext? 86 | 87 | A: Authenticator will keep its name, the implementation however should be called 'ClassicAuthenticator' instead. 88 | - Use case: Make sure that there is a possibility to edit/change credentials in a running application. 89 | 90 | A: Due to the nature of the protocols we are dealing with, this RFC will not cover the case of attempting to allow the user to make 'live' modifications to the authenticator object. Authenticator's on clusters/buckets shall be immutable (note that this enforces that changes to the authenticator do not change behaviour of objects already constructed using them, not that the object itself is immutable, ie: It is plausible that passing the authenticator to the Cluster would be by-value, implicitly making the clusters authenticator object immutable). 91 | - Java specific? The Java SDK already has an openBucket(String bucketName) method that default to the empty string for the password. So introduction of the Authenticator breaks the behavior. Should that method first look into the Authenticator for the password, and then default to empty string if not found? 92 | 93 | A: This is answered in the RFC above. 94 | - With regard to "edit/change" use case above, should this be made easier by exposing a getter for the Authenticator on the Cluster? And on the Bucket as well? Or should it be discouraged by exposing no such method in the public API interface? (maybe still expose it in the concrete classes, if relevant). 95 | 96 | A: See the above question regarding credentials on a running application. 97 | - So, we have an entry in the credentials dictionary for cluster-level N1QL creds; what if you pass credentials with a query, that doesn't require those credentials? I guess it's dependent upon the server impl., I am assuming it will only check that the credentials are valid for the cluster and not the specific query? 98 | 99 | A: When performing cluster-level queries, all available credentials at the cluster level of the Authenticator should be sent to the server (this means all bucket names/passwords). It is not possible to determine the credentials that would be necessary to satisfy a particular query, so sending them all is the only implementable solution for the moment. RBAC will mostly make this irrelevant anyways. 100 | - With RBAC coming, will the username not always be the bucket name in the future? 101 | 102 | A: In the future with RBAC, usernames will almost always be different from the buckets that the username has access to. Note that with RBAC, a new authenticator object type will exist which only accepts a single username and password (the RBAC credentials). 103 | - Does "cluster-cbft" imply that the Cluster object will have a Query(statement or request) method in the future? If that is the case shouldn't Bucket.Query(statement or request) become obsolete and "bucket-n1ql" become obsolete as well? I know this is slightly out-of-scope of this RFC, but it begets the discussion. Same for CBFT - the API should be moved to the more global object (cluster). 104 | 105 | A: The Cluster object will definitely gain a Query method for performing cluster-level queries. There was originally an RFC that existed for this, and an RI exists in both Node.js and Golang. I can't seem to figure out what happened to that proposal/RFC though. 106 | 107 | # Changelog 108 | - 05/18/2016: Changed name of first Authenticator implementation from 'ClusterAuthenticator' to 'PasswordAuthenticator'. 109 | Reworded entirety of the RFC to better explain the motivation and behavioural expectations of implementations. 110 | 111 | # Signoff 112 | 113 | 114 | 115 | 116 | 117 | 118 | 119 | 120 | 121 | 122 | 123 | 124 | 125 | 126 | 127 | 128 | 129 | 130 | 131 | 132 | 133 | 134 | 135 | 136 | 137 | 138 | 139 | 140 | 141 | 142 | 143 | 144 | 145 | 146 | 147 | 148 | 149 | 150 | 151 | 152 | 153 | 154 |
LanguageRepresentativeDate
JavaMichael N.08/22/2016
CMark N.08/10/2016
PythonMark N.08/10/2016
.NETJeff M.8/19/2016
node.jsBrett L.06/15/2016
PHPSergey A.08/10/2016
GoBrett L.06/15/2016
155 | -------------------------------------------------------------------------------- /rfc/0011-connection-string.md: -------------------------------------------------------------------------------- 1 | # Meta 2 | 3 | - RFC Name: Connection String 4 | - RFC ID: 0011 5 | - Start Date: 2014-02-12 6 | - Owner: Brett Lawson 7 | - Current Status: Accepted 8 | 9 | # Summary 10 | Define a URI-like connection string format for specifying bootstrapping and connection options for Couchbase. 11 | 12 | # Motivation 13 | Currently there is a number of variables that define the way that a cluster can be connected to. This can be defined as a matrix with 3 axes: SSL/Plain, HTTP/CCCP, SRV/Direct. Currently we are planning to provide a number of different specifications between client libraries to connect to a cluster, whereas we should be providing one unified specification which defines all neccessary details and semantics explicitly. Additionally, this proposal will attempt to define the precise semantics for some edge scenarios which have not yet been defined. 14 | 15 | # General Design 16 | A connection string will be consist of a scheme followed by host list (which potentially includes a port) along with a a set of key-value pairs. This should follow the existing format for a URI, except multiple hosts-port pairs will be supported by using a `,` separated. Additionally, `;` must be supported as a valid seperator for backwards compatiblity. 17 | 18 | Two base URI schemes must be supported by all clients, these are the `couchbase` scheme along with the `http` scheme for backwards compatability. The `couchbase` scheme should use CCCP only to perform connections and is expected to be used with all new clusters. The `http` scheme will behave as current clients do, performing neccessary heuristics to attempt CCCP connections prior to HTTP fallback (exact heuristics defined below). If no scheme is provided, the `http` scheme should be assumed, including all related heuristics. 19 | 20 | In addition to the two base URI schemes, there will be an additional supported scheme `couchbases`, this will utilize the exact bootstrap and connection semantics as the `couchbase`, but will utilize SSL encrypted channels only for cluster communication. Note that the use of `couchbase` means the client must not use SSL, and `couchbases` means the client must use SSL. Note that the https scheme is not valid. 21 | 22 | Following the scheme, a list of hosts should be provided, with optional ports specified on a per-host basis. This list should be delimited by a `,`, or alternatively a `;` for backwards compatability. In addition, a single host with no optional port being specified should cause a SRV-record lookup to be performed prior to attempting A-record lookup. See below for the layout of SRV-records. Should no port be explicitly specified, the services default canonical port should be used. 23 | 24 | ### HTTP Scheme Heuristics 25 | If a client specifies the `http` scheme, and does not specify a port or specifies the default canonical service port (8091), then an attempt should be made to utilize CCCP via 11210 prior to attempting to request the configuration via HTTP. Should a non-default port be specified, a HTTP request should be made immediately without attemping CCCP first. Additionally, when performing bootstrapping with this heuristic mode, all hosts should be attempted for CCCP prior to HTTP. 26 | 27 | ### SRV Records 28 | A DNS SRV record lookup should be performed should the connection string match the following criteria: 29 | Contains only a single hostname 30 | (ie: `couchbase://bb.org` not `couchbase://bb.org,cc.org`). 31 | Provides no explicit port specifier 32 | (ie: `couchbase://example.org`, not `couchbase://example.org:11210`). 33 | The scheme matches `couchbase` OR `couchbases`. 34 | (ie: `couchbase://example.org` but not `http://example.org`) 35 | 36 | If all of these conditions are met, a DNS lookup should be performed for `_[$scheme]._tcp.[$hostname]` (ie: a connection string of `couchbases://example.org/` should perform a SRV record lookup for `_couchbases._tcp.example.org`). 37 | Should any SRV records exist for this hostname: 38 | The host/port list within the connection string should be replaced with the target/port from all returned records. The original hostname that was used to perform the lookup should not be included in the list, unless it also exists as an SRV record. 39 | Should no SRV records be found for the hostname: 40 | The connection string should be remain as-is, and no further SRV related processing need occur. 41 | 42 | 43 | SRV records will be defined with no specific logic regarding weighting and priority at this point in time. Documentations should reflect that these values MUST be 0, however clients MUST also ignore these values. 44 | 45 | At the time of writing, two possible valid SRV prefixes are considered valid. Clients should ensure to only query for the record matching the scheme of the URL, clients should ensure to only query the record matching the active security mode. 46 | _couchbase._tcp.* (for non-secured memcached) 47 | _couchbases._tcp.* (for SSL-secured memcached) 48 | 49 | 50 | ### IPv6 Addresses 51 | IPv6 addresses will be specified similar to the IETF RFC-2732 document. The literal address will be enclosed within "[" and "]" characters to disambiguate a IPv6 octal separator and an explicit port. Note that IPv4 addresses written in IPv6 notation (i.e. ::192.9.5.5, ::ffff:192.0.2.128) should still be resolved as IPv4 addresses within a client. 52 | 53 | ### Additional Points 54 | The user-specified bootstrap list should be considered invalid and replaced by the server configuration as soon as the first valid configuration is received from the server. 55 | Username and passwords should be passed outside of the connection string, and not as part of it. 56 | All key/value pairs should use lower-case, underscore seperated names. 57 | 58 | 59 | ### Examples 60 | - Valid 61 | - 10.0.0.1:8091 (deprecation warning) 62 | - http://10.0.0.1 63 | - couchbase://10.0.0.1 64 | - couchbases://10.0.0.1:11222,10.0.0.2,10.0.0.3:11207 65 | - couchbase://10.0.0.1;10.0.0.2:11210;10.0.0.3 66 | - couchbase://[3ffe:2a00:100:7031::1] 67 | - couchbases://[::ffff.192.168.0.1]:11207,[::ffff.192.168.0.2]:11207 68 | - couchbase://test.local:11210?key=value 69 | - http://fqdn 70 | - http://fqdn?key=value 71 | - couchbases://fqdn 72 | - Invalid 73 | - http://host1,http://host2 74 | - https://host2:8091,host3:8091 75 | - http://::ffff:00ee:2122 76 | 77 | # Unresolved Questions 78 | None 79 | 80 | # Signoff 81 | | Language | Representative | Date | 82 | | -------- | ------------------ | ----------- | 83 | | C | Mark Nunberg | ~2014-07-01 | 84 | | Java | Michael Nitchinger | ~2014-07-01 | 85 | | .NET | Jeff Morris | ~2014-07-01 | 86 | | NodeJS | Brett Lawson | ~2014-07-01 | 87 | | PHP | Brett Lawson | ~2014-07-01 | 88 | | Python | Mark Nunberg | ~2014-07-01 | 89 | | Ruby | Sergey Avseyev | ~2014-07-01 | 90 | 91 | 92 | # Change Log 93 | - May 23, 2014 94 | - Renamed project from 'Project Unify-My-Bootstrapping-Please' 95 | - Defined additional semantics for missing ports in the hosts list. 96 | - May 28, 2014 97 | - Update to using URI schemes. 98 | - June 20, 2014 99 | - Various minor changes related to open questions. 100 | 101 | 102 | > This document is copy of the document available below and was converted to an sdk-rfc on Jan 18th, 2016. 103 | > https://docs.google.com/document/d/172ausWsYt3eYYOZ1lYHVS8ccbrrVJaGwHIRsf_O_Hyc 104 | -------------------------------------------------------------------------------- /rfc/0016-rbac.md: -------------------------------------------------------------------------------- 1 | # Role Based Access Control (RBAC) 2 | 3 | ## Meta 4 | 5 | * RFC Name: [RBAC] 6 | * RFC ID: [16] 7 | * Start Date: [2017-02-10] 8 | * Owner: [Mike Goldsmith] 9 | * Current Status: [Accepted] 10 | 11 | ## Summary 12 | 13 | This RFC defines how the Couchbase client libraries are to authenticate and operate with a cluster that utilises the RBAC cluster authentication mechanism. 14 | 15 | ## Motivation 16 | 17 | Couchbase Server has an advanced authentication mechanism where custom user profiles are created with a username and password and are assigned roles, such as Get, Insert and Delete, on one or more buckets. When a client makes a socket connection with the cluster it provides its assigned username and password which are verified by the cluster. 18 | 19 | ## General Design 20 | 21 | The RBAC authentication mechanism extends the `Authenticator` API by adding a new `PasswordAuthenticator` object type. Those credentials will be provided to the cluster in the same fashion as the current `ClassicAuthenticator` but will be interpreted differently internally. The RBAC authenticator only contains a single username and password and will be used to authenticate all connections. 22 | 23 | The following is an example implementation: 24 | 25 | ```c# 26 | public class PasswordAuthenticator : IAuthenticator 27 | { 28 | public string Username { get; set; } 29 | public string Password { get; set; } 30 | } 31 | ``` 32 | 33 | The following is an example of creating and assign the new credentials: 34 | 35 | ```c# 36 | var authenticator = new PasswordAuthenticator("mike", "secure123"); 37 | var cluster = new Cluster(); 38 | cluster.Authenticate(authenticator); 39 | 40 | var bucket = cluster.OpenBucket("default"); 41 | ``` 42 | 43 | Internally the process to authenticate a connection with RBAC credentials has changed where before previously a bucketname and optional password were used, RBAC requires a username and password. Also, before retrieving a bucket configuration an additional `SelectBucket` operation must be executed which is used to verify the current user has access to that bucket. The `SelectBucket` operation is not supported on pre-Spock clusters and will return an error. The client will need to do a feature check to see if the cluster expects a `SelectBucket` operation to be executed. 44 | 45 | ### Select Bucket 46 | 47 | A new cluster feature has been added that indicates if the cluster can support the enhanced authentication process. This should be checked for with the Hello operation and if the server responds with a positive result for that feature, you must then execute the `SelectBucket` operation. 48 | 49 | The `SelectBucket` operation must be sent after authentication and before any bucket operation, such as `GetClusterConfig`. The memcached documentation for using RBAC that describes the `SelectBucket` operation usage ([time of writing](https://github.com/couchbase/memcached/blob/f84289c43816db32f616bb3919736f105f8e00d5/docs/rbac.md) / [latest](https://github.com/couchbase/memcached/blob/master/docs/rbac.md)). 50 | 51 | The SelectBucket operation properties are below: 52 | 53 | | Property | Value | 54 | | - | - | 55 | | Operation Code | 0x89 | 56 | | Key | The bucket name to authenticate against. | 57 | | Extras | None | 58 | | Body | None | 59 | 60 | The server responds with a basic result to indicate if the user has access or not, with a status of either `Success (0x0000)` or `AccessError (0x0024)`. On `AccessError`, an `Authentication` error should be returned as this indicates the RBAC credentials were not accepted by the cluster and should be treated a permanent Authorization error. 61 | 62 | Note: The `XERROR (0x07)` feature must be enabled in order to receive an EACCESS error. If the client does not send XERROR as part of the Hello operation when establishing a connection, the server will disconnect the connection. This behavior may change in the future, but is something client developers should be aware of. 63 | 64 | ### Authenticating KV Operations 65 | 66 | KV operations utilise the TCP connection that was used to authenticate with so subsequent requests do not need to re-authenticate. 67 | 68 | ### Authorizing KV Operations 69 | 70 | If a user performs an operation they is not authorized for, the server will respond with the error code EACCESS. See [Client Error Messages][1] for appropriate error message. 71 | 72 | ### Authenticating Service Requests (View, FTS and N1QL) 73 | 74 | For each service that does not use an already authenticated TCP connection, the RBAC credentials will need to be passed using the Authorization header with the Basic scheme. 75 | 76 | The value part of the header is to be a base64 encoded combination of the username and password, the format of the value is below: 77 | 78 | {username}:{password} 79 | 80 | ### Cluster Manager & Bucket Manager 81 | 82 | The cluster and bucket managers provides credentials in the same manner as with the classic authenticator via form encoded values. 83 | 84 | ### Cluster Level N1QL Query 85 | 86 | Currently, cluster level query retrieves the first bucket defined in the `ClassicAuthenticator` bucket credentials array and submits the query to that bucket. The RBAC Authenticator does not define bucket credentials. 87 | 88 | NOTE: Both .NET and Java require a bucket to be opened manually before a cluster level query can be submitted. This is because a bucket is used to construct and submit the query and this can be done using the cached bucket to send the query to the cluster. 89 | 90 | ## Backward Compatibility 91 | 92 | RBAC credentials supersede the bucket name / password combinations used in the classic authenticator and because they are passed to the cluster in the same fashion if the cluster does not support RBAC, a normal authentication error will be returned. 93 | 94 | ## Errors 95 | 96 | | Scenario | Error Message | 97 | | - | - | 98 | | Mixed Authentication After one type of authentication credentials has been provided, any other credentials (eg bucketname/password) types should return an error to indicate mixed authentication mechanisms are not support. The SDK maintains the credentials and will return an Authentication based error. For example, if `cluster.OpenBucket("default", "password")` is called after RBAC credentials have been provided. | Unable to mix authentication types between RBAC and bucketname/password combinations. | 99 | 100 | 101 | Each client is also expected to return consistent error messages for standard scenarios, such as invalid username and/or password. A list is managed on the following link [Client Error Messages][1]. 102 | 103 | ## Connection Strings 104 | 105 | The username for a given user may be embedded as part of the connection string. This allows an authenticator to not provide the username and it will be used from the connection string instead. The username is inserted between the protocol and hostname/IP with the following format: 106 | 107 | ://@ 108 | 109 | For example, if the following connection string was provided, the username "mike" would be used. 110 | 111 | couchbase://mike@127.0.0.1 112 | 113 | ## Username Priority Order 114 | 115 | With the introduction of username as a connection string parameter and other ways to provide the authentication credentials, it is important to record what the preference order should be. The following list is in ascending priority order with 1 having the highest priority: 116 | 117 | 1. Programmatic, e.g. client receives directly from code 118 | 2. Explicit configuration, e.g. if the client supports explicitly specifying username and/or password in configuration 119 | 3. Connection string, as discussed above in Connection Strings 120 | 121 | ## Cluster Authenticate (username, password) 122 | 123 | A new `Authenticate` overload (where possible) should be added to the `Cluster` interface that provides a easy means to store a username and password. This would internally create and store a `PasswordAuthenticator`. An example is below: 124 | 125 | ```c# 126 | cluster.Authenticate("username", "password"); 127 | ``` 128 | 129 | ## Implementation Notes 130 | 131 | ### C/Libcouchbase 132 | 133 | Libcouchbase allows RBAC (and more generally cluster based auth) through a number of interfaces. The interface most closely mirrored in this RFC is the `lcbauth` interface, which was initially implemented in 2.6.4, and will be updated in 2.7.4 ([http://review.couchbase.org/75348](http://review.couchbase.org/75348)). 134 | 135 | More C-friendly ways exist to set these parameters, and do not involve creating new objects and having to call function accessors: 136 | 137 | 1. For single bucket credentials, you may use the `username` field in the connection string, or the `username` field in `lcb_create_st`. Password is as normal. 138 | 2. For multi-bucket credentials, you can use the `LCB_CNTL_BUCKET_CRED` setting to incrementally add more `bucket:password` pairs (though this is not applicable to the RBAC model). 139 | 140 | Using `lcb_AUTHENTICATOR` (Authenticator cognate): 141 | 142 | ```c 143 | lcb_create_st params = { 0 }; 144 | params.version =3 ; 145 | params.v.v3.connstr = "couchbase://localhost/bucketName"; 146 | lcb_create(instance, ¶ms); 147 | lcbauth_new() // creates new authenticator 148 | lcbauth_set_mode(auth, LCBAUTH_MODE_RBAC) // RBAC mode 149 | lcbauth_add_pass(auth, "username", "password", LCBAUTH_F_CLUSTER); 150 | lcb_set_auth(instance, auth); 151 | lcbauth_unref(auth); // instance owns lone reference. 152 | ``` 153 | 154 | Without `lcb_AUTHENTICATOR`: 155 | 156 | ```c 157 | lcb_create_st params = { 0 }; 158 | params.version = 3; 159 | params.v.v3.connstr = "couchbase://localhost/bucketname"; 160 | params.v.v3.username = "username"; 161 | params.v.v3.passwd = "password"; 162 | lcb_create(instance, ¶ms); 163 | ``` 164 | 165 | ## Change Log 166 | 167 | - [2017-03-10] - Added Connection strings to support username 168 | - [2017-03-14] - Support SelectBucket feature 169 | - [2017-04-04] - Add info regarding disconnect for auth failure without XERROR enabled 170 | - [2017-04-04] - Add link to expected client error messages 171 | - [2017-04-27] - Add LibCouchbase implementation notes 172 | - [2017-05-04] - Add Authenticate overload for username & password 173 | - [2017-05-26] - Improved username priority wording 174 | 175 | ## Signoff 176 | 177 | | Language | Representative | Date | 178 | | - | - | - | 179 | | C | Mark Nunberg | 2017-05-17 | 180 | | Go | Brett Lawson | 2017-05-16 | 181 | | Java | Mark Nitschinger | 2017-05-16 | 182 | | .Net | Jeff Morris | 2017-05-16 | 183 | | NodeJS | Brett Lawson | 2017-05-16 | 184 | | PHP | Sergey Avseyev | 2017-05-10 | 185 | | Python | Mark Nunberg | 2017-05-16 | 186 | 187 | [1]: https://docs.google.com/document/d/1hJLwk-2-Ai8LlRKyiRuwnff9mEKxN8PBxMMBH5-g4AE/edit?ts=58e3b94c 188 | -------------------------------------------------------------------------------- /rfc/0020-common-flags.md: -------------------------------------------------------------------------------- 1 | # Meta 2 | 3 | - RFC Name: Common Flags 4 | - RFC ID: 0020 5 | - Start Date: Sept 5, 2014 6 | - Owner: Brett Lawson 7 | - Current Status: Accepted 8 | 9 | # Summary 10 | This project aims to provide a consistent method for storing document meta-data. Specifically it will define the exact formatting of the flags field as well as the datatype field to allow the server to make reasonable assumptions as well. This document additionally attempts to define a set of 'universally supported' document formats. Bits are currently set aside for compression in this proposal, but it does not constrain how those bits may be used and a future extension to this proposal may choose to use them differently. 11 | 12 | # Risks and Assumptions 13 | This proposal assumes that no clients currently abuse the top 8 bits of the flags object, this has been confirmed to be the case with all existing memcached and couchbase clients, but may not be the case for any clients developed by third-parties. It should be noted this will not break anything for those third-party clients; it merely won't solve any interoperability issues for them. Additionally, it is assumed that any consumer of the flags data (customers with custom transcoders) will continue to use custom transcoders with the new SDKs which implement this proposal, and thus will be a non-concern as the proposals logic will not be concerned with these implementations. 14 | 15 | # Problem Area 16 | Currently, there is no defined method by which multiple SDKs or the clients can coordinate. This means there is a fair chance that documents stored by one SDK may not be able to be retrieved by another. 17 | 18 | # Market / Requester 19 | Customers who are using more than one SDK, such is often the case with tools being written vs the application itself. 20 | 21 | # Completion Criteria 22 | All clients will speak one universal language in regards to meta data. Documents stored with one SDK will be retrievable and manipulable by any other SDK. 23 | 24 | #General Design 25 | This proposal will specify a format for the upper 8 bits of the flags field for common use among clients and allow for client-specific data in the lower 16 bits (the middle 8 bits are reserved for future use, or possibly backwards compatibility use if necessary). 26 | 27 | During reading, the client should check these upper 8 bits for a non-zero value, and if this is the case, the common flags format should be used. If these upper 8 bits are zeroed, the clients should fall back to the existing logic used in the respective client. 28 | When writing, the client should set the upper 8 bits according to the format below, and additionally set the lowest 16 bits according to the existing logic that was used in their client. 29 | 30 | If a client encounters a format or compression type which is unknown, an error should be propagated to the user advising them that the data is stored with an unknown format. Additionally, any bits which are marked as reserved must be zero, an error should be propagated if this is not the case. 31 | 32 | The format of the upper 8 bits is as follows: 33 | The top 3 bits are used to specify the type of compression that is being used, the lower 4 bits are used to define the format, the middle 1 bit is currently reserved to allow expanding of compression or format fields as needed. The following is a list of all supposed formats, note that the 'private' format is used for sdk-specific encodings and the 'reserved' format is used to avoid a completely zeroed upper 8 bits which would interfere with detecting the presence of common flags. 34 | 35 | Flags (32 bits): 36 | 37 | - ` 0 | ? ? ? ? ? ? ? ? | ? ? ? ? ? ? ? ? | ? ? ? ? ? ? ? ?` 38 | 39 | Formats (All must be supported): 40 | 41 | - 0 - Reserved 42 | - 1 - Private 43 | - 2 - JSON 44 | - 3 - Raw Binary 45 | - 4 - Non-JSON String (utf-8, no bom) 46 | 47 | Compressions (expected compatibility specified individually below): 48 | 49 | - 0 - None (MUST be supported) 50 | 51 | Special Note: Pure numeric values could be encoded as both utf-8 strings as well as JSON. These values should always be encoded as JSON for the purpose of the common flags format. 52 | 53 | # Related Tickets 54 | None 55 | 56 | # Other Solutions Previous Considered 57 | The possible use of the datatype field was originally considered, but due to an unrelated implementation, that proposal is no longer valid. 58 | 59 | # In Scope 60 | All Client Libraries (except libcouchbase, which does not perform value encoding, though all tools should still understand this specification) 61 | 62 | # Out of Scope 63 | All Server Components 64 | 65 | # Interfaces 66 | No public facing interfaces should be affect by this change. However, clients SHOULD provide a custom transcoder interface to ensure any edge-cases have a workaround, this may affect public facing interfaces. 67 | 68 | # Documentation Impact 69 | The change to the internal representation of flags field MUST be documented clearly for customers. This is especially important as data that was stored in the old format will end up being 'on the fly' converted and may not be fully compatible with old memcached clients without implementing a custom transcoder. 70 | 71 | # Packaging and Delivery Impact 72 | No impact. This proposal is attached to the SDK 2.0 proposal, which already encompasses all necessary changes. 73 | 74 | # Security Impact 75 | No impact. 76 | 77 | # Dependancies 78 | There is a soft dependency on the SDK 2.0 changes as this project is intended to be a sub-proposal of the SDK 2.0 proposal (mainly to wrap all behaviour changes together into a single release). However, this proposals changes themselves do not depend on any SDK 2.0 proposal changes. 79 | 80 | # Reference Documents 81 | Please additionally refer to the Server DataType changes document for information on how to accurately store meta data in the datatype field for server consumption. 82 | 83 | # Questions 84 | - Heuristic compression detection, or part of datatype bitfield? 85 | - Heuristic compression detection will not be used. 86 | - Spymemcached backwards-compatibility? 87 | - Spymemcached correctly reads the flags data as a bitfield, preventing any compatibility issues. 88 | - Should we support primitive datatypes? 89 | - 64-bit signed integers, doubles will be supported. (spec updated) 90 | - Which formats/compressions must be supported by each SDK? 91 | - All formats; compressions are SDK specific at the moment. (spec updated) 92 | - Should tools be updated to display the type information stored in flags? 93 | - Yes, but the timeline on this is currently uncertain. (spec updated) 94 | - Should snappy be part of common flags since it is now a direct server feature? 95 | - Probably not, however it is a support compression from a client-library perspective, and having this as a known value allows the user to pass already-compressed data to our libraries and avoid double-compression and compression overhead. 96 | - Endianness and exact representation should be defined for primitive data types. 97 | - The spec was updated to define these. 98 | 99 | 100 | # Change Log 101 | - Sept 5, 2015 102 | - Initially accepted. 103 | - October 6, 2017 104 | - Converted to Markdown. 105 | -------------------------------------------------------------------------------- /rfc/0022-usermgmt.md: -------------------------------------------------------------------------------- 1 | ## Meta 2 | 3 | * RFC Name: User Management Support 4 | * RFC ID: 22 5 | * Start Date: 2017-08-03 6 | * Owner: Subhashni Balakrishnan 7 | * Current Status: Accepted 8 | 9 | ## Summary 10 | The goal of this RFC is to introduce SDK APIs to manage users for RBAC (Role-Based Access Control). 11 | An user with appropriate administrative access is able to create/update/delete users and get all users info. 12 | 13 | ## Motivation 14 | This ability allows for administrative tools development using the SDK and also for testing purposes. 15 | 16 | ## General Design 17 | The user management API should be added to the cluster management interface existing in SDK and internally they will interact with `ns_server`. There are two user types in RBAC - `local` and `external`. Local users are managed by the server while the external (LDAP) users are role-mapped. 18 | 19 | The user types are referred to as `AuthDomain`: 20 | 21 | ``` 22 | Enum AuthDomain { 23 | LOCAL, // on the wire its "local" 24 | EXTERNAL // on the wire its "external" 25 | } 26 | ``` 27 | 28 | ## API 29 | 30 | ### `upsertUser` 31 | Creates/updates a Couchbase user. 32 | 33 | ``` 34 | boolean upsertUser(AuthDomain domain, String userid, UserSettings settings) 35 | ``` 36 | 37 | If the domain is `external` and a password is set in the user settings, the client should proceed with sending the request to the server and log a warning saying that the password cannot be updated for external users. 38 | 39 | `UserSettings` are described with the following properties: 40 | 41 | ``` 42 | UserSettings { 43 | String password; 44 | String name; 45 | Role[] roles; 46 | } 47 | 48 | Role { 49 | String role; 50 | String bucket_name; 51 | } 52 | ``` 53 | 54 | **Example** 55 | 56 | ```java 57 | clusterManager.upsertUser( 58 | AuthDomain.LOCAL, 59 | "alice", 60 | UserSettings.build() 61 | .password("password") 62 | .name("Alice Doe") 63 | .roles(new Role[]{ new Role("query_select","default"), new Role("fts_searcher", "default") }) 64 | ) 65 | ``` 66 | 67 | - Full path is based on the `AuthDomain`: `PUT /settings/rbac/users/local/alice` 68 | - For external, `PUT /settings/rbac/users/external/alice` 69 | - data payload is www-form-urlencoded: `"name=Alice Doe&roles=query_select[default],fts_searcher[default]&password=password"` 70 | - Roles should be concatenated to a single string and are comma separated. Client returns true if server responds OK, else false. 71 | 72 | **One caveat**: the user is asynchronously updated on other servers in the cluster and `kv_engine`, this 73 | should be properly documented in the API docs. 74 | 75 | ### `removeUser` 76 | Deletes an Couchbase user. Note that depending on the `AuthDomain` this might be supported by the server 77 | or not. 78 | 79 | ``` 80 | boolean removeUser(AuthDomain domain, String userid) 81 | ``` 82 | 83 | **Example** 84 | 85 | ```java 86 | removeUser(AuthDomain.LOCAL, "alice") 87 | ``` 88 | 89 | - For local, `DELETE /settings/rbac/users/local/alice` 90 | - For external, `DELETE /settings/rbac/users/external/alice` 91 | 92 | ### `getUsers` 93 | Get all users from Couchbase based on the `AuthDomain`. 94 | 95 | ``` 96 | List getUsers(AuthDomain domain) 97 | ``` 98 | 99 | A `User` is described with the following properties: 100 | 101 | ``` 102 | User { 103 | String name; 104 | String id; 105 | String domain; 106 | Role[] roles; 107 | } 108 | 109 | Role { 110 | String role; 111 | String bucket_name; 112 | } 113 | ``` 114 | 115 | **Example** 116 | 117 | ```java 118 | clusterManager.getUsers(AuthDomain.LOCAL) 119 | ``` 120 | 121 | The following shows an example server response to `GET /settings/rbac/users/local`: 122 | 123 | ```json 124 | [ 125 | { 126 | "name": "Alice Doe", 127 | "id": "alice", 128 | "domain": "local", 129 | "roles": [ 130 | { 131 | "role": "fts_searcher", 132 | "bucket_name": "default" 133 | }, 134 | { 135 | "role": "n1ql_select", 136 | "bucket_name": "default" 137 | } 138 | ] 139 | } 140 | ] 141 | ``` 142 | 143 | - For external, `GET /settings/rbac/users/external` 144 | 145 | 146 | ### `getUser` 147 | Get user info for a particular Couchbase user. 148 | 149 | ``` 150 | User getUser(AuthDomain domain, String username) 151 | ``` 152 | 153 | The `User` instance is the same as described in `getUsers`. 154 | 155 | **Example** 156 | 157 | ```java 158 | clusterManager.getUser(AuthDomain.LOCAL, "alice") 159 | ``` 160 | 161 | Example response from `GET /settings/rbac/users/local/alice`: 162 | 163 | ```json 164 | { 165 | "name": "Alice Doe", 166 | "id": "alice", 167 | "domain": "local", 168 | "roles": [ 169 | { 170 | "role": "fts_searcher", 171 | "bucket_name": "default" 172 | }, 173 | { 174 | "role": "n1ql_select", 175 | "bucket_name": "default" 176 | } 177 | ] 178 | } 179 | ``` 180 | 181 | - For external, `GET /settings/rbac/users/external/alice` 182 | 183 | ## Language Specifics 184 | 185 | ### C 186 | The object model of defining roles etc. is fairly complex for a C level API. Applications can manually access the RBAC API through `lcb_http3()` using `LCB_HTTP_TYPE_MANAGEMENT` as the type. 187 | 188 | ### .NET 189 | .NET Does not use setting builders, so the implementation is done by adding an additional parameter for the AuthenticationDomain: `UpsertUser(AuthenticationDomain domain, string username, string password, string name, params Role[] roles)`. 190 | 191 | Additionally, there are implementations for async versions of for each of the overloads which take an AuthenticationDomain object. 192 | 193 | AuthenticationDomain is an enum with the following values: `Local` and `External` 194 | 195 | **Example:** 196 | 197 | ```net 198 | var createResult = _clusterManager.UpsertUser(AuthenticationDomain.Local, user.Username, "secure123", user.Name, user.Roles.ToArray()); 199 | ``` 200 | 201 | ### PHP 202 | In PHP API the functions which return indexes of the objects use "list" verb to make them distinguishable from singular form. Compare getDesignDocument/getDesignDocuments versus getDesignDocument/listDesignDocuments. 203 | 204 | So PHP will use listUsers in this API, even though we don't have getUser function 205 | 206 | ```php 207 | $userSettings = \Couchbase\UserSettings->build() 208 | ->password("password") 209 | ->role("data_reader", "myBucket"); 210 | $cluster->manager()->upsertUser("alice", $userSettings, 211 | \Couchbase\ClusterManager::RBAC_DOMAIN_LOCAL); 212 | $cluster->manager()->removeUser("alice", 213 | \Couchbase\ClusterManager::RBAC_DOMAIN_LOCAL); 214 | $cluster->manager()->listUsers(\Couchbase\ClusterManager::RBAC_DOMAIN_LOCAL); 215 | ``` 216 | 217 | ### Node.js 218 | 219 | ```js 220 | cluster->manager()->upsertUser('local', 'alice', { 221 | password: 'password', 222 | roles: {role: 'data_reader', bucket: 'myBucket'} 223 | }, function(err) { }) 224 | cluster->manager()->removeUser('local', 'alice', function(err) { }); 225 | cluster->manager()->listUsers('local', function(err, users) { }); 226 | ``` 227 | 228 | ### Python 229 | ```python 230 | Mgr = cluster.cluster_manager() 231 | Users = cluster.users_get(AuthDomain.Local) 232 | cluster.user_upsert(AuthDomain.Local, 'mark', 's3cr3t', [('global-role'), ('bucket-role', 'bucket-name')]) 233 | cluster.user_remove(AuthDomain.Local, 'mark') 234 | 235 | #Get_user = user_get 236 | #Upsert_user = user_upsert 237 | #Remove_user = user_remove 238 | ``` 239 | 240 | In Python it's simpler to define a user and its roles using Python lists/tuples, with roles passed as discrete arguments rather than an actual object. 241 | 242 | The methods exist as object-verb to be internally consistent with other Python management methods (some predating the RFC). Aliases to the common name are mentioned here as well for the sake of completeness. 243 | 244 | ### Java 245 | ```java 246 | cluster.clusterManager().upsertUser(AuthDomain.LOCAL, "alice", UserSettings.build().password("password") 247 | .roles(Arrays.asList(new UserRole("data_reader", "myBucket")))); 248 | cluster.clusterManager().removeUser(AuthDomain.LOCAL, "alice"); 249 | cluster.clusterManager().getUsers(AuthDomain.LOCAL); 250 | ``` 251 | 252 | ### Go 253 | ```go 254 | err := cluster.Manager(...).UpsertUser(gocb.LocalDomain, "alice", gocb.UserSettings{ 255 | Password: "password", 256 | Roles: []gocb.UserRole{{"data_reader", "myBucket"}}, 257 | }) 258 | err := cluster.Manager(...).RemoveUser(gocb.LocalDomain, "alice") 259 | users, err := cluster.Manager(...).GetUsers(gocb.LocalDomain) 260 | ``` 261 | 262 | ## Additional Info 263 | User roles info from ns_server. "*" as bucket_name implies access to all buckets. 264 | https://github.com/couchbase/ns_server/blob/0038200eddba441182cf1743420cef78bad88029/src/menelaus_roles.erl 265 | 266 | ## Q&A 267 | 1. I think we need to document the possible error cases for the methods above (what happens if user doesn't exist and so forth as well and define the proper error messages for those cases) 268 | *Clients will throw exceptions similar to bucket management api. There is no special handling for user management.* 269 | 2. Should we emulate the bucket management API a bit, like we have "hasBucket", we could do a "hasUser" or similar that just loads all and checks if its in the list? 270 | *We are not doing this 100%, we will keep api simple. The main motivation is test scaffolding.* 271 | 3. If the user already exists and we do an upsert without password, what is the expected behaviour? Should it clear the password, or do we need to not send the password field? 272 | *Yes, don't send the password. It wouldn't clear the password if it is not updated.* 273 | 4. How can this API consider that there are external users. 274 | *The scope of this RFC is not to manage those external users, however we should make sure that it's possible.* 275 | 276 | ## Changelog 277 | 278 | - 03/10/2017 Rename deleteUser to removeUser similar to removeBucket 279 | - 03/10/2017 Rename getAllUsers to getUsers 280 | - 04/21/2017 Change rest endpoints to use local instead of builtin and getUsers response will include domain as internal instead of type. 281 | - 06/14/2017 Change endpoint for getUsers to fetch only internal users 282 | - 06/14/2017 Add getUser api to fetch a particular user's info 283 | - 06/14/2017 Removed existing sign offs 284 | - 07/06/2017 Adding authentication domain 285 | 286 | ## Signoff 287 | 288 | | Language | Representative | Date | 289 | | - | - | - | 290 | | C | Sergey Avseyev | 2017-08-29 | 291 | | Go | Brett Lawson | 2017-07-26 | 292 | | Java | Subhashni Balakrishnan| 2017-08-25 | 293 | | .Net | Jeff Morris | 2017-09-06 | 294 | | NodeJS | Brett Lawson | 2017-07-26 | 295 | | PHP | Sergey Avseyev | 2017-07-26 | 296 | | Python | Matt Ingenthron | 2017-08-29 | 297 | -------------------------------------------------------------------------------- /rfc/0024-fast-failover.md: -------------------------------------------------------------------------------- 1 | # Meta 2 | 3 | - RFC Name: Fast-Failover SDK 4 | - RFC ID: 24 5 | - Start Date: 2017-03-03 6 | - Owner: Jeffry Morris 7 | - Current Status: ACCEPTED 8 | 9 | # Summary 10 | Fast-failover is a new feature coming with the Couchbase Server "Spock" version (5.0). It allows for a configurable setting on the server side for determining how quickly the server will automatically failover a node. This sdk-rfc defines the method of discovery and recovery for a failed over node from the the perspective of an SDK. 11 | 12 | This sdk-rfc covers scenarios for Query, View, FTS, Analytics and K/V services and any combination of these services enabled on a Couchbase node. 13 | 14 | The scope of this document is limited to `couchbase` and `ephemeral` buckets which will be supported in the upcoming 5.0 release. 15 | 16 | # Motivation 17 | Couchbase Server offers high-availability as a core feature; the current automatic failover settings can be configured as low as 30 seconds. With a 30 second automatic failover setting an application could take as much as 120 seconds to detect the failover which would mean the application might be down for at least that amount of time. 18 | 19 | With fast-failover, that time can be reduced to as little as a 5 seconds on the server. In practice, that means failure may be declared after 5 seconds, but failover may take slightly longer. In turn, the SDK must be able to identify when a node may be down as quickly as possible and reconfigure itself to the updated cluster-map. Additionally, all SDKs should follow similar rules for identifying when a node is down and be consistent in their behavior and implementation (as much as possible barring platform idiosyncrasies). 20 | 21 | # Current Implementations 22 | ## .NET 23 | The .NET SDK uses two means of determining if a failover has occurred and/or whether a node is unreachable. One is a means of determining if a node is no longer responsive (connections dropped, cannot reconnect, etc), the other is a check and compare of the server config for each active node with the SDK's current configuration. 24 | 25 | In the former, if `x` number of failures occur within a timespan `y`, then the node will be put into a `dead` state. Any K/V operations mapped to this node will result in a `NodeUnavailableException` being returned in the Exception property of the `IResult` implementation; N1QL and View requests will be routed towards other active nodes. Once a node is considered dead, then a timer will fire every 1s in an attempt to create a new connection and execute a NOOP against it. If the node responds, the node will become active again. If not, the timer will continue to fire until successful. The point of this is to keep the client from tieing up the application threads waiting for operations that will end up failing at the network level and instead just fail fast with an error until a connection can be made with the node. 26 | 27 | For the latter, there is a timer that fires every 10s which requests a config from each available (not dead) node. The revision is compared against the current cluster configuration that the client is using. If the revision has changed, then the client will be updated with the new configuration. 28 | 29 | ## Java 30 | In Java `core-io` we detect node failures through the following mechanisms in CCCP mode: 31 | 32 | - If an endpoint (socket) goes into a `DISCONNECTED` state we'll proactively fetch a new configuration via `GET_CONFIG` from the cluster. 33 | - If the server returns a NMVB we take that config and apply it 34 | - Every 10 seconds, we also poll for a new config via `GET_CONFIG` from one of the nodes in the cluster (this is meant as a backup to grab a new config even if missed by the other two reactive approaches) 35 | As another proactive measure under CCCP when a config is marked as tainted (that is under rebalance when a FF-Map is available) we start polling every second for a new config until the rebalance finished (config is not marked tainted anymore). 36 | 37 | For HTTP mode (if CCCP is not available/disabled), we have the streaming conn attached anyways, so we still do the NMVB override but both `GET_CONFIG` approaches (on disconnect & every 10s as well as the rebalance tainted polling) are essentially a `NOOP`. 38 | 39 | ## LCB 40 | LCB does not do background polling (it could, but it wouldn't be useful for anything except node.js and other "true" async use cases). 41 | 42 | LCB has a default throttling of 10s: Whenever a network error occurs, lcb triggers a config request. This will cause it to query a node if no other config request has been performed in the prior 10 seconds. 43 | 44 | The config request takes place in a different subsystem, and varies on the config transport being used: 45 | 46 | - When using "CCCP" it will merely request from the next node in the list, with an index variable wrapping around. If the server does not respond within 2.5s (or some other error occurs), the config is requested from the next node. 47 | - When using HTTP it will expect a new config over the stream of the current 'entry point node'; if there is no config pushed within 2.5s (configurable too!) it will assume that the node is down and request a config from the next node, which will effectively become the EP node. 48 | 49 | The throttle interval is configurable. 50 | 51 | ### LCB - Modifications 52 | - Lower the refresh-on-error 'floor' to 10ms from 100s. 53 | - "Background-poll" every 2.5 seconds. For clients that are synchronous, this will still poll every 2.5 seconds, or whenever the client is next active again -- depending on how active the client is. For clients that are async like node.js, it will behave similar to other clients' behavior in the RFC. 54 | 55 | ## GO 56 | In gocbcore we detect node failures through the following mechanisms in CCCP mode: 57 | 58 | - If the server returns a NMVB we take that config and apply it 59 | - Every 10s seconds, we also poll for a new config via `GET_CONFIG` from one of the nodes in the cluster 60 | 61 | For HTTP mode (if CCCP is not available/disabled), we have the streaming conn attached anyways, so we still do the NMVB override but both `GET_CONFIG` approaches (every 10s) are essentially a `NOOP`. 62 | 63 | ## PHP, Python, NodeJS 64 | Since they are language bindings, the implementation is handled by LCB (see above). 65 | 66 | # General Design 67 | For Fast-Failover to ­­­­be effective, the client SDK must quickly determine if a config change has occurred. Server configurations are acquired by either a) explicitly requesting them using the Config Command (`0x05`) or b) as the body of a K/V operation that has failed with a NMVB status. Typically a configuration is requested by an event within the application such as an IO error or by polling every at set intervals. Once a configuration has been obtained, the revision should be compared against the current configuration and if greater, swapped in the client, along with the VBucket mapping and connections if cluster endpoints have been added or removed. 68 | 69 | For example, in a centralized location the following methods exist: 70 | 71 | ``` 72 | bool CompareAndSwap(config){ 73 | lastCheckedTime = Time.Now; 74 | 75 | if(config.Rev > this.currentConfig.Rev) { 76 | exchange(config, this.currentConfig); 77 | return true; 78 | } 79 | return false; 80 | } 81 | 82 | void CheckConfigs(excluded) { 83 | If(lastCheckedTime + ConfigPollFoorInterval < Time.Now()) { 84 | foreach(server in cluster) { 85 | If(server != excluded) { 86 | var config = new ConfigRequest(); 87 | server.Send(config); 88 | CompareAndSwap(config); 89 | break; 90 | } 91 | } 92 | } 93 | } 94 | ``` 95 | 96 | In the case of an IO error or NMVB, the following pseudo code would exist at the IO layer of an SDK: 97 | 98 | ``` 99 | result Send(operation) { 100 | Try { 101 | Connection.Write(operation.GetBytes()); 102 | Operation.ReadBytes( Connection.Read()); 103 | } catch(IOException) { 104 | CheckConfigs(this); 105 | } catch(NMVException) { 106 | CompareAndSwap(operation.GetConfig()); 107 | } 108 | return Operation.Result; 109 | } 110 | ``` 111 | 112 | For continuous polling for config changes, the following pseudocode illustrates: 113 | 114 | ``` 115 | startIndex = -1; 116 | while (true) { 117 | Time.Sleep(ConfigPollInterval); 118 | if (startIndex < 0){ 119 | startIndex = rand(_currentConfig.NodeCount); 120 | } 121 | 122 | for (i=0; i<_currentConfig.NodeCount; i++) { 123 | index = (startIndex + i) % _currentConfig.NodeCount; 124 | server = _currentConfig.GetServer(index); 125 | 126 | if (server.LastCheckedTime + ConfigPollFoorInterval < time.Now) { 127 | newConfig = server.Send(new ConfigRequest()); 128 | if (CompareAndSwap(newConfig())) { 129 | break; 130 | } 131 | } 132 | } 133 | } 134 | ``` 135 | 136 | The minimum configurable failover detection interval is 5s on the server. If we chose a configurable, default polling interval of 2.5s then we should roughly cover the minimum Fast-Failover configuration from the server. However, to minimize chatter, a tunable property with a default floor of 50ms must be implemented - a user could change the tunable to a lower value like 2.5s if necessary, but it should not go below the floor defined. The trade-offs here **must be documented**. 137 | 138 | ``` 139 | 140 | SDK poll, SDK poll, SDK poll, Failover SDK poll, 141 | no change no change no change Auto Failover complete no change 142 | SDK poll, Timeout SDK adjust to 143 | │ Failure occurs │ no change │ (failover detected) │ new topology │ 144 | │ │ │ ┌─> ┌─> │ │ 145 | ├──────────┬──────────┼─────────┬──────────┼───────┼─────┬───────┼───────┼───────────┬────────────┤ 146 | │ │ │ │ │ ─┘ │ ─┘ │ │ │ 147 | │ │ │ │ │ │ │ │ │ 148 | 149 | 0s 1s 5s 10s 15s 16s 17s 20s 25s 150 | 151 | 152 | ``` 153 | 154 | Above is a timeline showing the events that occur with a Fast-failover time of 15 seconds on the server and a client polling every 5 seconds for a new config revision. In this case, the failover occurs at 1 second and is detected by the cluster at 16 seconds and the failover completes at 17 seconds (assuming it takes 1 second for the failover to occur). At the 20 second mark, the client will detect the new config revision and reconfigure itself to the new topology. By reducing the server side Fast failover value, and adjusting the frequency of the client-side polling interval, the amount of time for a failure to occur, be detected and for the client to adjust can be minimized. 155 | 156 | Since we want to minimize blocking, platforms that provide multi-thread or some kind of parallel IO support should use a separate worker thread or equivalent means of offloading the polling from main processing. 157 | 158 | Once we have gotten a cluster map, the revision should be compared with the client's current revision. If the revision is newer, the client should re-configure itself with the newer configuration to reflect the current server topology. Note that the polling should stop at the first config found and not continue through checking the configs over the entire cluster. 159 | 160 | # Conclusion 161 | Since we want to discover a cluster map change ASAP and because a cluster configuration can come in many flavors, a hybrid strategy for failover detection is probably our best bet. The client should react to errors related to connectivity (host not found, timeouts, etc) by checking for a config update when such errors occur; additionally, the client should implement a cluster map poll mechanism for detecting cluster map changes concurrently. In both cases the comparison algorithm (old vs new rev) should be the same and so should the reconfiguration process within the SDK. 162 | 163 | To ensure that we don't spam the server with config requests from polling and pulling, the same timestamp checking for floor and ceiling should be used as introduced in the "K/V Pulling" section above. The floor for config checking should be some value such as 50ms where we skip a check if a config request has already happened within that time. The ceiling should be a larger value such as 2.5s - both values should be tunable via client configuration. 164 | 165 | # Addendum 166 | The following configuration properties and default values are defined - note that all must be tunable: 167 | 168 | | Name | Default Value | 169 | | ---------------------------- | ------------- | 170 | | ConfigPollInterval | 2.5s | 171 | | ConfigPollFloorInterval | 50ms | 172 | 173 | Note that these values must be tunable. The individual SDK naming conventions for configuration names may be applied. 174 | 175 | # Questions 176 | No questions recorded. 177 | 178 | # Changelog 179 | - 7/6/2017: Initial draft publication 180 | - 10/3/2017 181 | - Change ConfigCheckInterval to ConfigPollInterval 182 | - Change ConfigCheckFloorInterval to ConfigPollFloorInterval 183 | 184 | 185 | # Signoff 186 | If signed off, each representative agrees both the API and the behavior will be implemented as specified. 187 | 188 | | Language | Representative | Date | 189 | | -------- | -------------- | ---------- | 190 | | Java | Michael N. | 2017-07-27 | 191 | | .NET | Jeff M. | 2017-08-17 | 192 | | Node | Brett L. | 2017-08-09 | 193 | | PHP | Sergey A. | 2017-07-26 | 194 | | Python | Matt I. | 2017-08-25 | 195 | | Go | Brett L. | 2017-08-09 | 196 | | C | Sergey A. | 2017-07-26 | 197 | -------------------------------------------------------------------------------- /rfc/0026-ketama-hashing.md: -------------------------------------------------------------------------------- 1 | # Meta 2 | - RFC Name: Ketama Hashing 3 | - RFC ID: 26 4 | - Start Date: 2016-12-21 5 | - Owner: Mike Goldsmith 6 | - Current Status: Accepted 7 | 8 | # Summary 9 | When storing documents in a Memcached bucket, the SDKs use a Ketama hashing algorithm to generate a ring of servers that can be used in to locate a server given a document key to create a robust and evenly distributed list of servers in the cluster. 10 | 11 | # Hashed Values 12 | The server hashes are MD5 hashes constructed using a server's IP, port and a repetition value, in the form: 13 | 14 | :- 15 | 16 | For example: 17 | ``` 18 | 127.0.0.1:8091-0 19 | 127.0.0.1:8091-1 20 | 127.0.0.1:8091-2 21 | ``` 22 | 23 | # Repetitions 24 | 40 hashes for each server in the cluster with the data server are generated. 25 | 26 | # Verification Process 27 | Add unit test to ensure a four node cluster generates the correct hashes that point to the correct server. The following JSON file contains a list of hash and hostname combinations that each SDK is expected to produce. The SDK should read the contents of the file and compare the generated hashes. 28 | 29 | Hostnames: 30 | ``` 31 | 192.168.1.101:11210 32 | 192.168.1.102:11210 33 | 192.168.1.103:11210 34 | 192.168.1.104:11210 35 | ``` 36 | 37 | [Expected Hashes](ketama-hashes.json) 38 | 39 | # Code Examples 40 | ## .NET 41 | ```csharp 42 | using (var md5 = MD5.Create()) 43 | { 44 | foreach (var server in _servers.Values.Where(x => x.IsDataNode)) 45 | { 46 | for (var rep = 0; rep < 40; rep++) 47 | { 48 | var bytes = Encoding.UTF8.GetBytes($"{server.EndPoint}-{rep}"); 49 | var hash = md5.ComputeHash(bytes); 50 | for (var j = 0; j < 4; j++) 51 | { 52 | var key = ((long) (hash[3 + j * 4] & 0xFF) << 24) 53 | | ((long) (hash[2 + j * 4] & 0xFF) << 16) 54 | | ((long) (hash[1 + j * 4] & 0xFF) << 8) 55 | | (uint) (hash[0 + j * 4] & 0xFF); 56 | 57 | _buckets[key] = server; 58 | } 59 | } 60 | } 61 | } 62 | ``` 63 | 64 | ## Java 65 | ```java 66 | private void populateKetamaNodes() { 67 | for (NodeInfo node : nodes()) { 68 | if (!node.services().containsKey(ServiceType.BINARY)) { 69 | continue; 70 | } 71 | 72 | for (int i = 0; i < 40; i++) { 73 | MessageDigest md5; 74 | try { 75 | md5 = MessageDigest.getInstance("MD5"); 76 | md5.update(env.memcachedHashingStrategy().hash(node, i).getBytes(CharsetUtil.UTF_8)); 77 | byte[] digest = md5.digest(); 78 | for (int j = 0; j < 4; j++) { 79 | Long key = ((long) (digest[3 + j * 4] & 0xFF) << 24) 80 | | ((long) (digest[2 + j * 4] & 0xFF) << 16) 81 | | ((long) (digest[1 + j * 4] & 0xFF) << 8) 82 | | (digest[j * 4] & 0xFF); 83 | ketamaNodes.put(key, node); 84 | } 85 | } catch (NoSuchAlgorithmException e) { 86 | throw new IllegalStateException("Could not populate ketama nodes.", e); 87 | } 88 | } 89 | } 90 | } 91 | ``` 92 | 93 | ## libcouchbase 94 | ```c 95 | static int 96 | update_ketama(lcbvb_CONFIG *cfg) 97 | { 98 | char host[MAX_AUTHORITY_SIZE+10] = ""; 99 | int nhost; 100 | unsigned pp, hh, ss, nn; 101 | unsigned char digest[16]; 102 | lcbvb_CONTINUUM *new_continuum, *old_continuum; 103 | 104 | qsort(cfg->servers, cfg->ndatasrv, sizeof(*cfg->servers), server_cmp); 105 | 106 | new_continuum = calloc(160 * cfg->ndatasrv, sizeof(*new_continuum)); 107 | /* 40 hashes, 4 numbers per hash = 160 points per server */ 108 | for (ss = 0, pp = 0; ss < cfg->ndatasrv; ++ss) { 109 | /* we can add more points to server which have more memory */ 110 | for (hh = 0; hh < 40; ++hh) { 111 | lcbvb_SERVER *srv = cfg->servers + ss; 112 | nhost = snprintf(host, MAX_AUTHORITY_SIZE+10, "%s-%u", srv->authority, hh); 113 | vb__hash_md5(host, nhost, digest); 114 | for (nn = 0; nn < 4; ++nn, ++pp) { 115 | new_continuum[pp].index = ss; 116 | new_continuum[pp].point = ((uint32_t) (digest[3 + nn * 4] & 0xFF) << 24) 117 | | ((uint32_t) (digest[2 + nn * 4] & 0xFF) << 16) 118 | | ((uint32_t) (digest[1 + nn * 4] & 0xFF) << 8) 119 | | (digest[0 + nn * 4] & 0xFF); 120 | } 121 | } 122 | } 123 | 124 | qsort(new_continuum, pp, sizeof *new_continuum, continuum_item_cmp); 125 | old_continuum = cfg->continuum; 126 | cfg->continuum = new_continuum; 127 | cfg->ncontinuum = pp; 128 | free(old_continuum); 129 | return 1; 130 | } 131 | ``` 132 | 133 | ## Go 134 | ```go 135 | func newKetamaContinuum(serverList []string) *ketamaContinuum { 136 | continuum := ketamaContinuum{} 137 | 138 | // Libcouchbase presorts this. Might not strictly be required.. 139 | sort.Strings(serverList) 140 | 141 | for ss, authority := range serverList { 142 | // 160 points per server 143 | for hh := 0; hh < 40; hh++ { 144 | hostkey := []byte(fmt.Sprintf("%s-%d", authority, hh)) 145 | digest := md5.Sum(hostkey) 146 | 147 | for nn := 0; nn < 4; nn++ { 148 | 149 | var d1 = uint32(digest[3+nn*4]&0xff) << 24 150 | var d2 = uint32(digest[2+nn*4]&0xff) << 16 151 | var d3 = uint32(digest[1+nn*4]&0xff) << 8 152 | var d4 = uint32(digest[0+nn*4] & 0xff) 153 | var point = d1 | d2 | d3 | d4 154 | 155 | continuum.entries = append(continuum.entries, routeKetamaContinuum{ 156 | point: point, 157 | index: uint32(ss), 158 | }) 159 | } 160 | } 161 | } 162 | 163 | sort.Sort(ketamaSorter{continuum.entries}) 164 | 165 | return &continuum 166 | } 167 | ``` 168 | 169 | # Signoff 170 | 171 | | Language | Representative | Date | 172 | | - | - | - | 173 | | C | Mark Nunberg | 21st March 2017 | 174 | | Go | Brett Lawson | 22nd Marcg 2017 | 175 | | Java | Michael Nitschinger | 21st March 2017 | 176 | | .NET | Jeff Morris | 23rd March 2017 | 177 | | NodeJS | Brett Lawson | 22d March 2017 | 178 | | PHP | Sergey Avseyev | 27th March 2017 | 179 | | Python | Mark Nunberg | 21st March 2017 | 180 | -------------------------------------------------------------------------------- /rfc/0028-enhanced_error_messages.md: -------------------------------------------------------------------------------- 1 | # Meta 2 | - RFC Name: Enhanced Error Messages 3 | - RFC ID: 0028 4 | - Start Date: 2017-04-24 5 | - Owner: Brett Lawson 6 | - Current Status: ACCEPTED 7 | 8 | # Summary 9 | As of Couchbase Server 5.0.0, the server will be improving its error handling behaviours to enable the inclusion of additional, per-operation contextual information. This will take the form of JSON-formatted error objects included in the response body of server error responses. This change will not affect operations which already customize the error response body (multi-mutation subdocument operations when status code is `SUCCESS (0x00)` or `SUBDOC_MULTI_PATH_FAILURE (0xCC)`, and `NOT_MY_VBUCKET` error responses for other operations). 10 | 11 | The primary driver of this change is to enable correlation of security-sensitive authentication error context between the client and the server. This change will also enable the server to provide further detail to developers on the reason an operation has failed, beyond what is available with status codes alone. 12 | 13 | # Changelog 14 | - April 24 2017: Initial Draft 15 | 16 | # Description 17 | The server will be switching to a new error response format (we no longer will receive text such as `not_found`). These responses will be encoded in the following format: `{ "error" : { "context" : "textual context information", "ref" : "error reference to be found in the server logs" }}`. Each component of this error body will be optional, and if none of these optional fields are available, the error response will be blank. In addition, all JSON error responses will be tagged with the JSON data type bit to signal the client that the error response is parseable JSON. Note that in the case of no error body being returned (due to no additional error details being available), the JSON bit will NOT be set. 18 | 19 | Should a client wish to log any particular error, they are are required to log all components of this enhanced error information. This log message MUST include the the error message as defined by the client as well as the context and ref fields from the error response body if they are defined. If the client does not have an error message defined for the status code received, it must include the status code directly along with any enhanced error information; clients may also optionally include the error code even if an error message is known. Clients MUST NOT include the response body within the log messages or client error objects if the the JSON bit is not set, or if the response body is empty. 20 | 21 | In the case of SDKs which have an error wrapping object (Java and .NET Exception, Go error structures, etc…), the client MUST, at minimum include the additional context within the error message that would be displayed if the developer were to print the object directly. This string can be generated by concatenating all information that is available about an error, IE: "Client Error Message (Context: Server Error Context, Ref #: Server Error Reference)". Additionally, clients with error wrapping objects SHOULD include the additional information as individual, independent fields. 22 | 23 | In the case of clients which do not have error wrapping objects, the clients may optionally include methods to fetch the additional information provided by the server for the most recent operation that failed (similar to `errno()`). This information is not required to be persisted any longer than what is reasonable for the client (i.e.: libcouchbase may only allow fetching the latest error information within the context of the error callback for the specific operation, or only until the next operation is dispatched or received). 24 | 25 | # Language Specifics 26 | ## .NET 27 | .NET creates a custom TemporaryLockFailureException if the response status is "TemporaryFailure" (0x0086) and the response body contains "lock_error". For backward compatibility, the response body must always be checked to see if the string is found, whether a direct string or in JSON. 28 | 29 | # Questions 30 | - (Brett) Won't the server context string being concatenated with the client-side error message cause a duplication of information in the final message? 31 | No. The server context string will never duplicate the information that is already provided by the error code, and will only include additional contextual information which is not represented by the error code itself. If no additional context information is available beyond what is represented by the error code, the server will simply omit that field. 32 | 33 | - (Brett) Won't this change mean that our error messages change between versions? 34 | The full error strings may change, but the initial client-message component will continue to be the same as it was before, and the enhanced error information will be concatenated to the end. 35 | 36 | - (Mark) Won't this mean that we will receive garbage or useless error contexts from older server versions? 37 | This specification states that we will only be utilizing error responses that are marked with the JSON bit, and that all other error responses (except in the case of already-customized handlers as indicated above) will be ignored and assumed not to convey any enhanced contextual information. Additionally, it's only expected to be used in situations where an error reply is not the most common path, so the additional memory usage won't be the bulk of the operations and meets a clear need for the users. 38 | 39 | - (Brett) Will this not cause older clients which used the response body to change their error messages, potentially causing backwards-compatibility issues? 40 | A: The only SDK affected by this RFC which is known to include the server-side response body in the error message is the .NET SDK, which will be treating this as a bug and will be correcting it to be in line with this SDK. 41 | 42 | - (Mark) How does this affect or get affected by the inclusion of KV Error Map features? 43 | This RFC is tangential to the KV Error Map RFC. The only overlap that exists being that the 'client error message' as described above, will come from the KV Error Map rather than from a static map in the client. 44 | 45 | - (Michael) any reason this is not behind a HELLO? 46 | The user-visible changes of this RFC are already behind a bit (the JSON bit). I'm not sure what benefit would be provided by putting it behind another flag. 47 | 48 | - (Matt) Should the error messages be documented as being volatile? 49 | Yes 50 | 51 | # Signoff 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62 | 63 | 64 | 65 | 66 | 67 | 68 | 69 | 70 | 71 | 72 | 73 | 74 | 75 | 76 | 77 | 78 | 79 | 80 | 81 | 82 | 83 | 84 | 85 | 86 | 87 | 88 | 89 | 90 | 91 | 92 | 93 | 94 |
LanguageRepresentativeDate
JavaMichael N.2017-05-05
CMark N.2017-04-26
PythonMark N.2017-04-26
.NETJeff M.2017-04-22
node.jsBrett L.2017-04-22
PHPSergey A.2017-05-10
GoBrett L.2017-04-22
95 | -------------------------------------------------------------------------------- /rfc/0050-sdk3-datastructures.md: -------------------------------------------------------------------------------- 1 | # Meta 2 | 3 | - RFC Name: SDK3 Datastructures 4 | - RFC ID: 0050-sdk3-datastructures 5 | - Start Date: July 28, 2019 6 | - Owner: Brett Lawson \ 7 | - Current Status: ACCEPTED 8 | - Relates To 9 | - [SDK-RFC#8](rfc/0008-datastructures.md) (Datastructures) 10 | - [Original Google Drive Doc](https://docs.google.com/document/d/1mKk20ScVE8ssF2DvqZTe9xIUvOUanJ7LOARFiCJPkQ0) 11 | 12 | # Motivation 13 | 14 | A subset of Couchbase's user-base desires easy drop-in replacement for various other technologies or drop-in persistent replacements for various concurrent data-structures available in the SDKs.  In order to enable competitive representation, and continue with the feature-set already enabled by SDK 2, the decision was made to persist our data-structure support into SDK 3. 15 | 16 | # Summary 17 | 18 | This RFC describes information about higher level data-structures which should be implemented in all RFCs.  As opposed to the typical SDK 3 RFC which provides a specific interface which should be implemented, this RFC specifically targets providing a specific feature set with the implementation being driven by what is idiomatic to the specific SDK platform. 19 | 20 | # Technical Details 21 | 22 | To provide data-structures within the framework of SDK 3.0, methods should be included within the Collections interface which enable you to instantiate a concurrent data-structure object according to what is available in each language.  In Java you might expect this to be interfaces from the java.util.concurrent namespace, in .NET this might be interfaces from the System.Collections.Concurrent namespace, etc...  The remainder of this document shall be dedicated to describing the functionality which must be provided by each SDK.  Note that SDKs are only responsible for implementing (and should only implement) interfaces which are designed for concurrent use and must not block by default.  Additionally, data structures are only supported at the document level, and cannot be nested.  Lastly, all data-structures should be created lazily (only when they are first used) and all retrieval methods should make the assumption that a missing document simply indicates an empty structure. 23 | 24 | ### Error Handling 25 | 26 | Exceptions for datastructures fall into two categories: 27 | 28 | - Exceptions raised by the SDK that cannot be mapped into the language specific exception types based on the languages data-structures implementation. 29 | - Exceptions which are covered by the language's data-structures implementation. 30 | 31 | For Couchbase-Specific exceptions these errors need to be documented: 32 | 33 | - TimeoutException (#1) 34 | - CouchbaseException (#0) 35 | 36 | All other documented or undocumented errors are implementation specific because they heavily depend on the language-specific interface/type they implement. 37 | 38 | ### Lists 39 | 40 | The server-side representation of a list is a JSON array.  An instance of this type is returned via a non-blocking, no-network call to ICollection::list(key).  The following is a list of operations which must be supported upon lists. 41 | 42 | #### Iteration 43 | 44 | Implemented by fetching the entire list and yielding a result per value in the list: 45 | 46 | ```typescript 47 | items = mc_get($doc) 48 | for item in items: 49 |   yield item 50 | ``` 51 | 52 | #### Indexed Access 53 | 54 | Implemented via a subdocument element access: 55 | 56 | ```typescript 57 | return mc_lookupin_get($doc, "[" + $index + "]"); 58 | ``` 59 | 60 | #### Indexed Removal 61 | 62 | Implemented via a subdocument element removal: 63 | 64 | ```typescript 65 | mc_mutatein_remove($doc, "[" + $index + "]"); 66 | ``` 67 | 68 | #### Append 69 | 70 | Implemented via a subdocument array append: 71 | 72 | ```typescript 73 | mc_mutatein_array_append($doc, $value); 74 | ``` 75 | 76 | #### Prepend 77 | 78 | Implemented via a subdocument array prepend: 79 | 80 | ```typescript 81 | mc_mutatein_array_prepend($doc, $value); 82 | ``` 83 | 84 | #### IndexOf 85 | 86 | Implemented by fetching the whole document and finding the appropriate item (note that depending on language specifics, this should support specifying a starting index): 87 | 88 | ```typescript 89 | items = mc_get($doc) 90 | 91 | for i, item in items:\ 92 |   if i < $startingIndex: continue 93 |   if item === $value: 94 |     return i 95 | return -1 96 | ``` 97 | 98 | #### Size 99 | 100 | Implemented via a subdocument get count: 101 | 102 | ```typescript 103 | mc_lookupin_get_count($doc); 104 | ``` 105 | 106 | #### Clear 107 | 108 | Implemented via a document remove: 109 | 110 | ```typescript 111 | mc_remove($doc); 112 | ``` 113 | 114 | ### Maps 115 | 116 | The server-side representation of a map is a JSON object.  An instance of this type is returned via a non-blocking, no-network call to ICollection::map(key).  The following is a list of operations which must be supported for maps. 117 | 118 | #### Iteration 119 | 120 | Implemented by fetching the entire list and yielding a compound object per value in the map: 121 | 122 | ```typescript 123 | items = mc_get($key) 124 | for k, v in items: 125 |   yield {k, v} 126 | ``` 127 | 128 | #### Keyed Access 129 | 130 | Implemented using a subdocument get operation: 131 | 132 | ```typescript 133 | return mc_lookupin_get($doc, $key); 134 | ``` 135 | 136 | #### Keyed Insertion 137 | 138 | Implemented using a subdocument set operation: 139 | 140 | ```typescript 141 | mc_mutatein_mapset($doc, $key, $value); 142 | ``` 143 | 144 | #### Keyed Removal 145 | 146 | Implemented using a subdocument removal operation: 147 | 148 | ```typescript 149 | mc_mutatein_mapremove($doc, $key); 150 | ``` 151 | 152 | #### Exists 153 | 154 | Implemented by using a subdocument exists operation: 155 | 156 | ```typescript 157 | return mc_lookupin_exists($doc, $key); 158 | ``` 159 | 160 | #### Size 161 | 162 | Implemented via a subdocument get count operation: 163 | 164 | ```typescript 165 | mc_lookupin_get_count($doc); 166 | ``` 167 | 168 | #### Keys 169 | 170 | Implemented by fetching the entire document and extracting the keys: 171 | 172 | ```typescript 173 | items = mc_get($key)\ 174 | return KEYS_OF(items) 175 | ``` 176 | 177 | #### Values 178 | 179 | Implemented by fetching the entire document and extracting the values: 180 | 181 | ```typescript 182 | items = mc_get($key)\ 183 | return VALUES_OF(items) 184 | ``` 185 | 186 | #### Clear 187 | 188 | Implemented via a document remove: 189 | 190 | ```typescript 191 | mc_remove($doc); 192 | ``` 193 | 194 | ### Sets 195 | 196 | The server-side representation of a map is an unsorted JSON list.  An instance of this type is returned via a non-blocking, no-network call to ICollection::set(key).  The following is a list of operations which must be supported for sets. 197 | 198 | Note: Sets are restricted to containing primitive types only due to server-side comparison restrictions. 199 | 200 | #### Iteration 201 | 202 | See implementation of List::Iteration 203 | 204 | #### Add 205 | 206 | Implemented using a subdocument array add unique operation: 207 | 208 | ```typescript 209 | mc_mutatein_array_add_unique($doc, $value); 210 | ``` 211 | 212 | #### Remove 213 | 214 | Implemented by fetching the entire document, identifying the item to remove, and then performing a CAS based removal of the singular entry, retrying on CAS failures: 215 | 216 | ```typescript 217 | while !timedOut(): 218 |   items = mc_get($doc) 219 |   for i, item in items: 220 |     if item === $value: 221 |       try: 222 |         mc_mutatein_remove($doc, "[" + $i + "]", cas: cas) 223 |       catch CasMismatch: 224 |         continue 225 |       catch ...: 226 |         throw 227 |       return 228 | throw TimeoutException() 229 | ``` 230 | 231 | #### Values 232 | 233 | Implemented by fetching the entire document and returning it (as it should already be an array): 234 | 235 | ```typescript 236 | return mc_get($key); 237 | ``` 238 | 239 | #### Contains 240 | 241 | Implemented by fetching the entire document and searching for the requested value: 242 | 243 | ```typescript 244 | items = mc_get($doc) 245 | for i, item in items: 246 |   if item === $value: 247 |     return true 248 | return false 249 | ``` 250 | 251 | #### Size 252 | 253 | See implementation of List::Size 254 | 255 | #### Clear 256 | 257 | Implemented via a document remove: 258 | 259 | ```typescript 260 | mc_remove($doc); 261 | ``` 262 | 263 | ### Queues 264 | 265 | The server-side representation of a queue is a JSON list.  An instance of this type is returned via a non-blocking, no-network call to ICollection::queue(key).  Queues are implemented similarly to lists, but include a pop method for removing an item from the queue. Queues are a FIFO structure.  The following is a list of operations which must be supported for queues. 266 | 267 | #### Iteration 268 | 269 | See implementation of List::Iteration 270 | 271 | #### Push 272 | 273 | See implementation of List::Prepend 274 | 275 | #### Pop 276 | 277 | Implemented by performing a subdocument get on "[-1]", and then performing a CAS-based subdocument removal with a path of "[-1]", retrying on CAS failures: 278 | 279 | ```typescript 280 | while !timedOut(): 281 |   value, cas = mc_lookupin_get($doc, "[-1]") 282 |   try: 283 |     mc_mutatein_remove($doc, "[-1]", cas: cas) 284 |   catch CasMismatch: 285 |     continue 286 |   catch ...: 287 |     throw 288 |   return 289 | throw TimeoutException() 290 | ``` 291 | 292 | #### Size 293 | 294 | See implementation of List::Size 295 | 296 | #### Clear 297 | 298 | Implemented via a document remove: 299 | 300 | ```typescript 301 | mc_remove($doc); 302 | ``` 303 | 304 | # Changelog 305 | 306 | - July 28, 2019 - Revision #1 (by Brett Lawson) 307 | 308 | - Initial Draft 309 | 310 | - Sept 11, 2019 - Revision #2 (by Brett Lawson) 311 | 312 | - Added wording to describe how instances of the interfaces described in this document are accessed from the SDK API. 313 | - Clarified the continue/break levels for some of the retry loops. 314 | - Corrected an error in the example implementation for Set::values() 315 | - Added missing Keyed Access and Keyed Removal to Maps. 316 | - Updated example code for operations which must do CAS-retries to include a 16 attempt maximum rather than indefinite attempts. 317 | - Corrected some minor typos. 318 | - Removed useless questions section. 319 | 320 | - Sept 19, 2019 - Revision #3 (by Brett Lawson) 321 | 322 | - Add wording to clarify that data-structures should be created lazily and that a missing document should not error, but rather indicates an empty structure. 323 | - Added Clear() method to all data-structures which removes the underlying document which effectively clears the structure. 324 | 325 | - Nov 18, 2019 - Revision #4 (by Brett Lawson) 326 | 327 | - Added correct pseudo-code to demonstrate the appropriate retry behaviour during CAS operations. 328 | - Added wording to clarify that the Queue data-structure is a FIFO structure. 329 | - Added more specific error handling details. 330 | - Added support for specifying a starting index to the indexOf method of List. 331 | 332 | - Nov 18, 2019 (by Brett Lawson) 333 | 334 | - Moved RFC to REVIEW state. 335 | 336 | - April 30, 2020 (by Brett Lawson) 337 | 338 | - Moved RFC to ACCEPTED state. 339 | 340 | - Sept 17, 2021 (by Brett Lawson) 341 | 342 | - Converted to Markdown 343 | 344 | # Signoff 345 | 346 | | Language | Team Member | Signoff Date | Revision | 347 | | ---------- | ------------------- | ------------ | -------- | 348 | | Node.js | Brett Lawson | 2020-04-16 | 4 | 349 | | Go | Charles Dixon | 2020-04-22 | 4 | 350 | | Connectors | David Nault | 2020-04-29 | 4 | 351 | | PHP | Sergey Avseyev | 2020-04-22 | 4 | 352 | | Python | Ellis Breen | 2020-04-29 | 4 | 353 | | Scala | Graham Pople | 2020-04-30 | 4 | 354 | | .NET | Jeffry Morris | 2020-04-22 | 4 | 355 | | Java | Michael Nitschinger | 2020-04-16 | 4 | 356 | | C | Sergey Avseyev | 2020-04-22 | 4 | 357 | | Ruby | Sergey Avseyev | 2020-04-22 | 4 | 358 | -------------------------------------------------------------------------------- /rfc/0051-sdk3-views.md: -------------------------------------------------------------------------------- 1 | # Meta 2 | 3 | * RFC Name: SDK3 Views 4 | * RFC ID: 0051-sdk3-views 5 | * Start Date: 2019-09-18 6 | * Owner: Sergey Avseyev 7 | * Current Status: ACCEPTED 8 | * Relates To: 9 | [0054-sdk3-management-apis](0054-sdk3-management-apis.md), 10 | [0058-sdk3-error-handling](0058-sdk3-error-handling.md), 11 | [0059-sdk3-foundation](0059-sdk3-foundation.md) 12 | 13 | # Summary 14 | 15 | This RFC is part of the bigger SDK 3.0 RFC and describes the Views APIs in detail. 16 | 17 | # Technical Details 18 | 19 | API entry endpoint: 20 | 21 | ``` 22 | interface IBucket { 23 | ... 24 | ViewResult ViewQuery(string designDocument, string viewName, [ViewOptions options]); 25 | ... 26 | } 27 | ``` 28 | 29 | ## IBucket::ViewQuery 30 | 31 | The `IBucket::ViewQuery` method enables a user to query a view index and receive the results in a streaming manner. The SDK should perform 32 | this by executing a HTTP GET request against the views service. 33 | 34 | ### Signature 35 | 36 | ``` 37 | ViewResult ViewQuery(string designDocument, string viewName, [ViewOptions options]); 38 | ``` 39 | 40 | ### Parameters 41 | 42 | * Required 43 | 44 | * `designDocument`: `string` 45 | 46 | Specifies the name of the design document containing the view which is to be queries. 47 | 48 | * `viewName`: `string` 49 | 50 | Specifies the name of the view to query. 51 | 52 | * Optional (part of `ViewOptions`) 53 | 54 | * `includeDocuments` (`bool`) = undefined (`false`) 55 | 56 | If specified, the SDK must fetch the document for every key in the result set and expose it via ViewRow.Document. This is not 57 | mandatory to implement, and rather optimization for synchronous-only SDKs, that don't have background IO worker. 58 | 59 | * `scanConsistency` (`ViewScanConsistency`) = undefined (`NotBounded`) 60 | 61 | Specifies the level of consistency for the query. Sent within the HTTP request query as `stale` and is encoded as: 62 | * `RequestPlus`: `false` 63 | * `UpdateAfter`: `update_after` 64 | * `NotBounded`: `ok` 65 | 66 | * `skip` (`uint32`) = undefined (`0`) 67 | 68 | Specifies the number of results to skip from the start of the result set. Sent within the HTTP request query as `skip` and is encoded 69 | as a decimal string (`14`). 70 | 71 | * `limit` (`uint32`) = undefined (unlimited) 72 | 73 | Specifies the maximum number of results to return. Sent within the HTTP request query as `limit` and is encoded as a decimal string 74 | (`5`). 75 | 76 | * `startKey` (`JsonValue`) = undefined 77 | 78 | Specifies the key to skip too before beginning to return results. Sent within the HTTP request query as `startkey` and is encoded as a 79 | JSON string (`{}`). 80 | 81 | * `endKey` (`JsonValue`) = undefined 82 | 83 | Specifies the key to stop returning results at. Sent within the HTTP request query as `endkey` and is encoded as a JSON string (`{}`, 84 | `"forum_post_42"`). 85 | 86 | * `startKeyDocId` (`string`) = undefined 87 | 88 | Specifies the document id to start returning results at within a number of results should startKey have multiple entries within the 89 | index. Sent within the HTTP request query as `startkey_docid` and is encoded as direct string (`hello`). 90 | 91 | * `endKeyDocId` (`string`) = undefined 92 | 93 | Specifies the document id to stop returning results at within a number of results should endKey have multiple entries within the 94 | index. Sent within the HTTP request query as `endkey_docid` and is encoded as direct string (`hello`). 95 | 96 | * `inclusiveEnd` (`boolean`) = undefined (`false`) 97 | 98 | Specifies whether the endKey/endKeyDocId values above should be inclusive or exclusive. Sent within the HTTP request query as 99 | `inclusive_end` and is encoded as a boolean string (`true` or `false`). 100 | 101 | * `group` (`boolean`) = undefined (`false`) 102 | 103 | Specifies whether to enable grouping of the results. Sent within the HTTP request query as `group` and is encoded as a boolean string 104 | (`true` or `false`). 105 | 106 | * `groupLevel` (`uint32`) = undefined (`unlimited`) 107 | 108 | Specifies the depth within the key to group results. Sent within the HTTP request query as `group_level` and is encoded as a decimal 109 | string (`14`). 110 | 111 | * `key` (`JsonValue`) = undefined 112 | 113 | Specifies a specific key to fetch from the index. Sent within the HTTP request query as `key`. 114 | 115 | * `keys` (`[]JsonValue`) = undefined 116 | 117 | Specifies a specific set of keys to fetch from the index. Sent within the HTTP request query as `keys`. 118 | 119 | * `order` (`ViewOrdering`) = undefined (`Descending`) 120 | 121 | Specifies the order of the results that should be returned. Sent within the HTTP request query as `descending` and is encoded as: 122 | * `Ascending`: `false` 123 | * `Descending`: `true` 124 | 125 | * `reduce` (`boolean`) = undefined (`false`) 126 | 127 | Specifies whether to enable the reduction function associated with this particular query index. Sent within the HTTP request query as 128 | `reduce` encoded as a boolean string (`true` or `false`). 129 | 130 | * `onError` (`ViewErrorMode`) = undefined(`Stop`) 131 | 132 | Specifies the behaviour of the query engine should an error occur during the gathering of view index results which would result in 133 | only partial results being available. Sent within the HTTP request query as `on_error` and is encoded as: 134 | * `Continue`: `continue` 135 | * `Stop`: `stop` 136 | 137 | * `debug` (`boolean`) = undefined (`false`) 138 | 139 | Allows to return debug information as part of the view response. Sent within the HTTP request query as `debug` and is encoded as a 140 | boolean string (`true` or `false`). 141 | 142 | * `raw` (`string key, string value`) 143 | 144 | Allows the user to specify a custom parameter to be passed within the HTTP request's query string and should be URI encoded inside the 145 | SDK before being sent to the server. This is used as an escape hatch in case the user wishes to utilize an option which is not 146 | otherwise exposed. 147 | 148 | * `namespace` (`DesignDocumentNamespace`) = `Production` 149 | 150 | Specifies whether the SDK should prefix the design document name with a `"dev_"` prefix. See enum definition in 151 | [0054-sdk3-management-apis](0054-sdk3-management-apis.md). 152 | 153 | * `timeout` (`Duration`) = `$Cluster::viewOperationTimeout` 154 | 155 | Specifies how long to allow the operation to continue running before it is canceled (default value defined in 156 | [0059-sdk3-foundation](0059-sdk3-foundation.md)). 157 | 158 | * `serializer` (`JsonSerializer`) = `$Cluster::Serializer` 159 | 160 | Specifies the serializer which should be used for deserialization of keys and values which are read from the `ViewRow`s. 161 | `JsonSerializer` interface is defined in [0055-sdk3-transcoders-and-serializers](0055-sdk3-transcoders-and-serializers.md). 162 | 163 | ### Returns 164 | 165 | A ViewResult that maps the result of the view query to an object. 166 | 167 | ### Throws 168 | 169 | * Documented 170 | * `ViewNotFoundException` (#501) 171 | * `RequestTimeoutException` (#1) 172 | * `CouchbaseException` (#0) 173 | 174 | * Undocumented 175 | * `RequestCanceledException` (#2) 176 | * `InvalidArgumentException` (#3) 177 | * `ServiceNotAvailableException` (#4) 178 | * `InternalServerException` (#5) 179 | * `AuthenticationException` (#6) 180 | 181 | ## Return Types 182 | 183 | ### ViewResult 184 | 185 | Represents the resulting data from a view query. 186 | 187 | ``` 188 | struct ViewResult { 189 | Stream Rows(); 190 | Promise MetaData(); 191 | } 192 | ``` 193 | 194 | ### ViewRow 195 | 196 | Represents a single row of data returned from a view index. The `JsonSerializer` which is specified as part of the originating query should 197 | be used to perform decoding of the key and value contained within the row. Note that the `Document` field in this structure is only included 198 | if the `IncludeDocument` parameter is set on the view query. Similarly, the `Id` field is excluded from view query results if a reduction is 199 | being used. 200 | 201 | ``` 202 | struct ViewRow { 203 | Optional Id(); 204 | K Key(); 205 | V Value(); 206 | Optional Document(); 207 | } 208 | ``` 209 | 210 | ### ViewMetaData 211 | 212 | Represents the meta-data returned along with a view query. 213 | 214 | ``` 215 | struct ViewMetaData { 216 | uint64 TotalRows(); 217 | Optional debug(); 218 | } 219 | ``` 220 | 221 | # Changelog 222 | 223 | * Sept 18, 2019 - Revision #1 (by Sergey Avseyev) 224 | * Initial Draft 225 | 226 | * Sept 26, 2019 - Revision #2 (by Brett Lawson) 227 | * Expanded upon a number of behaviours to remove ambiguity. 228 | * Added missing Serializer option to ViewQuery options. 229 | * Added debug option to ViewQuery options and ViewMetaData. 230 | * Removed fullset from ViewQuery options 231 | * Renamed continueOnError to onError in ViewQuery options. 232 | * Reformatted the RFC 233 | * Converted to use DesignDocumentNamespace from management RFC #54 234 | 235 | * Oct 23, 2019 - Revision #3 (by Sergey Avseyev) 236 | * Added Document accessor for ViewRow 237 | * Added includeDocuments options 238 | 239 | * Jan 10, 2020 - Revision #4 (by Sergey Avseyev) 240 | * Clarified that ViewOptions#includeDocuments is optional to implement. 241 | 242 | * Feb 06, 2020 - Revision #5 (by Sergey Avseyev) 243 | * Added Optional to ViewRow Id and Document fields which are optional depending on various configuration options for the view query. Also clarified their behaviour in the description for ViewRow. 244 | 245 | * April 30, 2020 246 | * Moved RFC to ACCEPTED state. 247 | 248 | * August 27, 2020 249 | * Converted to Markdown. 250 | 251 | # Signoff 252 | 253 | | Language | Team Member | Signoff Date | Revision | 254 | |------------|---------------------|----------------|----------| 255 | | Node.js | Brett Lawson | April 16, 2020 | #5 | 256 | | Go | Charles Dixon | April 22, 2020 | #5 | 257 | | Connectors | David Nault | April 29, 2020 | #5 | 258 | | PHP | Sergey Avseyev | Sept 27, 2019 | #5 | 259 | | Python | Ellis Breen | April 29, 2020 | #5 | 260 | | Scala | Graham Pople | April 30, 2020 | #5 | 261 | | .NET | Jeffry Morris | April 22, 2020 | #5 | 262 | | Java | Michael Nitschinger | April 16, 2020 | #5 | 263 | | C | Sergey Avseyev | Sept 27, 2019 | #5 | 264 | | Ruby | Sergey Avseyev | Sept 27, 2019 | #5 | 265 | -------------------------------------------------------------------------------- /rfc/0055-serializers-transcoders.md: -------------------------------------------------------------------------------- 1 | # Meta 2 | 3 | * RFC Name: SDK3 Transcoders & Serializers 4 | * RFC ID: 0055-serializers-transcoders 5 | * Start Date: June 13, 2019 6 | * Owner(s): Brett Lawson, Jeffry Morris 7 | * Current Status: ACCEPTED 8 | 9 | ## Motivation 10 | 11 | The SDK team has provided a common system for converting SDK data-types to storable formats for Couchbase Server which is common across all SDKs and provides the ability for the customer to seamlessly transition their data between numerous languages. 12 | 13 | In certain cases, the transcoder or serializer behaviours may be altered by the user providing a custom implementation of `ITranscoder` and/or `IJsonSerializer`. For example: 14 | 15 | A user has stored its data in Couchbase in a binary format and wants it to remain binary as a storage format. 16 | A user has stored its data in Couchbase in a binary format and wants to read it as binary and write it as JSON. 17 | A user wishes to use a serializer other than the default serializer (Jackson or NewtonSoft.NET for example) and provide a custom implementation of serializer. 18 | etc... 19 | 20 | The first two cases are generally legacy/upgrade scenarios from customers or users who started with very old Couchbase servers before JSON was a storage type. The last is done to satisfy the needs of a customer or user for whatever reason does not want to use the default serializer (usually for performance reasons or because they wish to reuse the custom configuration which they've already implemented with an alternative JSON serialization library). 21 | 22 | ## Summary 23 | 24 | Most SDKs use Transcoders and Serializers when reading or writing data to and from Couchbase using the various Services (Search, KV, Views, etc). 25 | 26 | Transcoders handle the encoding and decoding of the documents when stored and retrieved from Couchbase Server using the [Common Flags Specification][SDK-RFC#20]. This is done so that documents can be written in one language's SDK and read universally in another and vice versa. The RFC defines the format types of JSON, non-JSON string, and raw binary. Transcoders also delegate the reading or writing of the data using serializers or native conversions from bytes to concrete types and vice versa. 27 | 28 | Serializers are used by Transcoders to handle converting the actual JSON bytes to and from native language objects (native data structures that mirror the JSON documents elements). These native objects are frequently very specific to a platform. 29 | 30 | ## Technical Details 31 | 32 | The class diagram below shows the two major interfaces `ITranscoder` and `IJsonSerializer`. In addition to these interfaces, specific core libraries for converting to & from bytes and primitive types may be used within an implementation. 33 | 34 | 35 | ![figure 1: general design](https://github.com/couchbaselabs/sdk-rfcs/blob/master/rfc/figures/rfc55-uml1.png) 36 | 37 | The serializer is used by the various non-binary operations within Couchbase to perform parsing of JSON data. This is used for example when parsing rows from N1QL, fields from Search or values from Views. 38 | 39 | The transcoder types are used by various CRUD operations within Couchbase to perform transcoding of the raw bytes stored within Couchbase to and from the native types of the specific SDK language. The transcoder is responsible for accepting a native type and generating the raw bytes for storage along with a set of flags used to specify the format of those bytes for other SDKs. Note that transcoders which perform serialization/deserialization of JSON types should have a IJsonSerializer property used to configure the JSON serialization behaviour of that Transcoder. 40 | 41 | ### Datatype Field Handling 42 | 43 | The SDK is responsible for parsing the flags returned by the transcoder and detecting the presence of the Common Flags JSON bit. If this bit is set, the SDK should also specify the JSON bit within the datatype field sent to the server. During the deserialization phase, the common flags are used exclusively, and the datatype flag is ignored. 44 | 45 | ## Default JsonSerializer 46 | 47 | The default JsonSerializer implementation within the SDK should handle passing types of T, all numeric types and strings through the standard JSON library included with the SDK. In the case of a byte array, the Serializer should instead pass the data through unchanged, enabling query service results to be fetched without performing serialization by passing a byte array as the output type T. 48 | 49 | ## Default Transcoders 50 | 51 | The SDK must implement a number of specific implementations of the `ITranscoder` interface, along with a default implementation of the IJsonSerializer interface. The JsonTranscoder implementation should be the default transcoder utilized by the SDK should no other option be specified by the user, the default serializer should take advantage of whatever JSON serialization already exists within the SDK. The following is a list of specific transcoder implementations which MUST be implemented by the SDK, along with information about their specific behaviours. Note that some knowledge of the specifics of [Common Flags][SDK-RFC#20] is assumed in the explanation of the transcoders below. 52 | 53 | ### LegacyTranscoder 54 | 55 | The LegacyTranscoder is intended to be a backwards-compatibility transcoder. This transcoder implements behaviour which matches the SDK 2.0 behaviours and allows customers migrating from these older SDKs who did not make use of strictly JSON data to correctly cooperate with the newer SDK. Note that this transcoder should have a IJsonSerializer property used to customize the JSON serialization behaviour of the transcoder. 56 | 57 | * T -> JSON Value [Flags=JSON] 58 | * String -> String Bytes [Flags=String] 59 | * Number -> JSON Number [Flags=JSON] 60 | * Buffer -> Binary Bytes [Flags=Binary] 61 | 62 | ### JsonTranscoder (Default) 63 | 64 | The JsonTranscoder is intended to be the new default for SDK 3.0, this transcoder implements JSON transcoding alone in order to guarantee compatibility with the services integrated in Couchbase Server. In addition, the specifics of this implementation resolve a number of ambiguous scenarios with regards to handling pre-transcoded types, forcing the user to explicitly specify their meaning through (potentially) the selection of another transcoder type. Note that this transcoder should have a IJsonSerializer property used to custom the JSON serialization behaviour of the transcoder. 65 | 66 | * T -> JSON Value [Flags=JSON] 67 | * String -> JSON String [Flags=JSON] 68 | * Number -> JSON Number [Flags=JSON] 69 | * Binary -> Error 70 | 71 | #### RawJsonTranscoder 72 | 73 | The RawJsonTranscoder provides the ability for the user to explicitly specify that the data they are storing or retrieving is meant to be strictly JSON bytes. This enables the user to retrieve data from Couchbase and forward it across the network without incurring unnecessary parsing costs along with enabling the user to take JSON data which was received across the network and insert it into Couchbase without incurring parsing costs. Note that this transcoder does not accept an IJsonSerializer, and always performs straight pass through of the data to the server. 74 | 75 | * T -> Error 76 | * String -> JSON Bytes [Flags=JSON] 77 | * Number -> Error 78 | * Buffer -> JSON Bytes [Flags=JSON] 79 | 80 | #### RawStringTranscoder 81 | 82 | The RawStringTranscoder provides the ability for the user to explicitly store and retrieve raw string data to Couchbase. This transcoder is responsible for ensuring that the data is encoded as UTF-8 and masking the appropriate string flag. 83 | 84 | * T -> Error 85 | * String -> String Bytes [Flags=String] 86 | * Number -> Error 87 | * Buffer -> Error 88 | 89 | #### RawBinaryTranscoder 90 | 91 | The RawBinaryTranscoder provides the ability for the user to explicitly store and retrieve raw byte data to Couchbase. The transcoder does not perform any form of real transcoding, but rather passes the data through and assigns the appropriate binary flags. 92 | 93 | * T -> Error 94 | * String -> Error 95 | * Number -> Error 96 | * Buffer -> Binary Bytes [Flags=Binary] 97 | 98 | #### SDK Specific Transcoder 99 | 100 | A number of languages in the wild have custom serialization formats implicitly provided by the language. For instance, in the case of Java there is Java Serialized Object Format, in the case of PHP there is the PHP Serialized Object Format. In the case that a custom language specific serialization format exists for the implementing language, a custom transcoder which specifically acts against this format should be implemented. The SDK MUST ensure that this custom format encodes the flags data according to the appropriate rules specified by [Common Flags][SDK-RFC#20]. 101 | 102 | ### Language Specifics 103 | In languages such as Go, where there is a common language-level method of specifying an array of bytes that is specifically of JSON type (in Go this is the json.RawMessage type), the SDK is responsible for extending the LegacyTranscoder, JsonTranscoder and RawJsonTranscoder to support these types implicitly. Additionally, depending on the context of any SDK Specific Transcoder, these too should support these specialized types. 104 | 105 | #### Languages With Type Trait Capabilities 106 | (I'll use Scala terms here, but some languages have similar and can make use of this.) 107 | Scala gets a small tweak to permit it to take advantage of its advanced type capabilities. When app does: 108 | 109 | ```scala 110 | val result = collection.get("id", transcoder = JsonTranscoder()) 111 | result.contentAs[JsonObject]() 112 | ``` 113 | 114 | The contentAs method takes a Scala implicit parameter, a `JsonSerializer[T]` (technically a Type Trait). The 'implicit' part means the compiler is asked to find a `JsonSerializer[JsonObject]`, in various scopes. There's a number of these `JsonSerializable[T]` provided out of the box, providing support for several popular JSON libraries, plus Strings and byte arrays. For instance, a `JsonSerializer[io.circe.Json]` would use the Circe library to turn a byte array to and from the `io.circe.Json` type. The user is also free to add their own to support e.g. another JSON library. 115 | 116 | That compiler-discovered `JsonSerializer` is then provided to the Transcoder previously specified in the GetOptions block, if a) it's a `JsonTranscoder` (the only one that can accept a `JsonSerializer`) and b) the app didn't specify a serializer for the transcoder already. This is the only real change from the RFC - instead of the app having to choose and specify a serializer, it's compiler provided. 117 | 118 | #### Other examples: 119 | 120 | ```scala 121 | // Save byte array, unserialized, as Binary 122 | val imageData: Array[Byte] = // … 123 | coll.replace("id", imageData, transcoder = RawBinaryTranscoder()) 124 | 125 | // saves string, unserialized, with Json dataformat 126 | coll.replace("id", """{"hello":"world"}""", transcoder = RawJsonTranscoder()) 127 | 128 | // saves string, unserialized, with String dataformat 129 | coll.replace("id", """{"hello":"world"}""", transcoder = RawStringTranscoder()) 130 | 131 | // pick up JsonSerializer[String], which is a pass-through serializer, and default JsonTranscoder will save with Json dataformat 132 | coll.replace("id", """{"hello":"world"}""") 133 | ``` 134 | 135 | ### Custom Transcoders and Serializers 136 | 137 | The SDKs MUST ensure that the `ITranscoder` and `IJsonSerializer` types are able to be implemented by the user outside the scope of the default transcoders which are provided by the SDK. 138 | 139 | Note that while utilizing a custom Transcoder may affect the ability for a customer to share data among various SDKs, a correctly implemented custom transcoder will still specify flags data conforming to [Common Flags][SDK-RFC#20], and this case should be detectable by other SDKs to avoid attempting to parse potentially unparsable data and return an error instead. 140 | 141 | ## Changelog 142 | * June 13, 2019 - Revision #1 (by Jeffry Morris) 143 | * * Initial Draft 144 | * Sept 26, 2019 - Revision #2 (by Brett Lawson) 145 | * * Rewrote the RFC according to the newly specified transcoding behaviours from Sept 21, 2019 SDK3 Core meeting. Specifically, changing the transcoder behaviour to utilize individualized transcoders to explicitly specify the desired transcoding behaviour as opposed to utilizing a DataFormat abstraction. 146 | * * Added a number of clarifications with regards to the implementation of transcoders and their interaction with Common Flags and the Datatype field supported by the server. 147 | Clarified the behaviour of the JsonSerializer. 148 | * April 30, 2020 149 | * * Moved RFC to ACCEPTED state. 150 | 151 | |Language |Representative |Date |Revision | 152 | |:--- |:--- |:--- |:--- | 153 | |Node.js|Brett Lawson|Sept 26, 2019|#2 | 154 | |Go|Charles Dixon|April 22, 2020|#2 | 155 | |Connectors|David Nault|April 29, 2020 |#2 | 156 | |PHP|Sergey Avseyev|Sept 27, 2019|#2 | 157 | |Python|Ellis Breen|April 30, 2020|#2 | 158 | |Scala|Graham Pople |Oct 24, 2019|#2 | 159 | |Java|Michael Nitschinger|Sept 27, 2019|#2 | 160 | |C|Sergey Avseyev|Sept 27, 2019|#2 | 161 | |Ruby|Sergey Avseyev|Sept 27, 2019|#2 | 162 | 163 | 164 | [SDK-RFC#20]: /rfc/0020-common-flags.md 165 | -------------------------------------------------------------------------------- /rfc/0056-sdk3-query.md: -------------------------------------------------------------------------------- 1 | # Meta 2 | 3 | * RFC Name: SDK3 Query 4 | * RFC ID: 0056-sdk3-query 5 | * Start Date: 2019-09-27 6 | * Owner: Michael Nitschinger 7 | * Current Status: ACCEPTED 8 | * Relates To: 9 | [0054-sdk3-management-apis](0054-sdk3-management-apis.md), 10 | [0058-sdk3-error-handling](0058-sdk3-error-handling.md), 11 | [0059-sdk3-foundation](0059-sdk3-foundation.md) 12 | 13 | # Summary 14 | 15 | This RFC is part of the bigger SDK 3.0 RFC and describes the N1QL Query APIs in detail. 16 | 17 | # Technical Details 18 | 19 | ## API Entry Point 20 | 21 | The Query API is located at Cluster level: 22 | 23 | ``` 24 | interface ICluster { 25 | ... 26 | QueryResult Query(string statement, [QueryOptions options]); 27 | ... 28 | } 29 | ``` 30 | 31 | And at the Scope level: 32 | 33 | ``` 34 | interface IScope { 35 | ... 36 | QueryResult Query(string statement, [QueryOptions options]); 37 | ... 38 | } 39 | ``` 40 | 41 | ## ICluster#Query 42 | 43 | The `ICluster::Query` method enables a user to perform a N1QL query and receive the results in a streaming manner. The SDK should perform this by executing a HTTP GET request against the query service. 44 | 45 | **Signature:** 46 | 47 | ``` 48 | QueryResult Query(string statement, [QueryOptions options]); 49 | ``` 50 | 51 | **Parameters:** 52 | 53 | * Required: 54 | * statement : string 55 | * Specifies the N1QL query statement in string form (i.e. "select foo from bar") 56 | * Optional (part of QueryOptions): 57 | * scanConsistency(QueryScanConsistency) = undefined(NotBounded) 58 | * Specifies the level of consistency for the query. 59 | * Sent within the JSON payload as `scan_consistency` and is encoded as: 60 | * RequestPlus: `request_plus` 61 | * NotBounded: `not_bounded` 62 | * This option overrides any setting that was previously set on consistentWith 63 | * adhoc(boolean), 64 | * Specifies if the prepared statement logic should be executed internally. 65 | * This option is not sent over the wire but rather internally controls the prepared statement behavior as outlined in the "Enhanced Prepared Statements RFC" 66 | * clientContextId(String) = UUID(), 67 | * Specifies a context ID string which is mirrored back from the query engine on response 68 | * If not provided, the client must create and use a UUID instead. Sent as in the JSON payload as "client_context_id" as a JSON String `"foobar"` 69 | * consistentWith(MutationState) = undefined 70 | * Specifies custom scan consistency through "at_plus" with mutation state token vectors 71 | * On the wire represented as a custom scan_consistency together with a scan_vector. See the section on "ConsistentWith Handling" in this RFC for details 72 | * This option overrides any setting that was previously set on scanConsistency 73 | * maxParallelism(uint32) = undefined(0), 74 | * The maximum number of logical cores to use in parallel for this query 75 | * Sent in the JSON payload as "max_parallelism" as a JSON String i.e. `"1"`. 76 | * parameters(JsonObject | JsonArray) = undefined 77 | * Specifies positional or named parameters 78 | * For languages that do not support operator overloading, the alternative naming is positionalParameters(JsonArray) and namedParameters(JsonObject) 79 | * Sent in the JSON payload 80 | * For positional parameters as a json array under the "args" key 81 | * For named parameters directly in the JSON payload, but each named argument is prefixed with "$" if it doesn't already have the dollar prefix provided by the user 82 | * The payload can include both positional and named parameters 83 | * Setting JsonArray overrides any previously set _positional_ parameters 84 | * Setting JsonObject overrides any previously set _named_ parameters 85 | * pipelineBatch(uint32) = undefined 86 | * Specifies pipeline batching characteristics 87 | * Sent in the JSON payload as "pipeline_batch" as a JSON String 88 | * pipelineCap(uint32) = undefined 89 | * Specifies pipeline cap characteristics 90 | * Sent in the JSON payload as "pipeline_cap" as a JSON String 91 | * profile(QueryProfile) = undefined(Off) 92 | * Specifies the profiling level to enable 93 | * Sent within the JSON payload as `profile` and is encoded as: 94 | * Off: `"off"` 95 | * Phases: `"phases"` 96 | * Timings: `"timings"` 97 | * raw(String key, JsonValue value) 98 | * Specifies values with their key and value as presented in the map as part of the JSON payload 99 | * This is an escape hatch to support unknown commands and be forwards compatible 100 | * readonly(boolean) = undefined(false) 101 | * Allows to specify if the query is readonly 102 | * Sent within the JSON Payload as `readonly` as a JSON boolean 103 | * metrics(boolean) = false 104 | * Allows to enable the metrics at the end of the result in the metadata section 105 | * Sent within the JSON Payload as `metrics` as a JSON boolean 106 | * scanWait(Duration) = undefined 107 | * Specifies the maximum time wait for a scan. 108 | * Sent within the JSON payload as `scan_wait` encoded as a JSON String in go format! (so "1s" for example) 109 | * scanCap(uint32) = undefined 110 | * Specifies scan cap characteristics 111 | * Sent in the JSON payload as "scan_cap" as a JSON String 112 | * timeout(Duration) = $Cluster::queryOperationTimeout 113 | * Specifies how long to allow the operation to continue running before it is cancelled. 114 | * serializer(JsonSerializer) = $Cluster::Serializer 115 | * Specifies the serializer which should be used for deserialization of rows returned. 116 | * preserveExpiry(boolean) = undefined(false) 117 | * Specifies that the query engine should preserve expiration values set on any documents modified by this query. 118 | * Sent within the JSON Payload as `preserve_expiry` as a JSON boolean 119 | * Should be added at API stability level of uncommitted. 120 | * useReplica(bool) = undefined 121 | * Specifies that the query engine should use replica nodes for kv fetches if the active node is down. 122 | * Sent within the JSON payload as `use_replica` as the JSON string `"on"` or `"off"`, only if set by the user. 123 | * If the SDK does not have the ability to differentiate between not set and `false` then a different type should be used instead, suggested: 124 | * `enum QueryUseReplicaLevel { ON, OFF }` 125 | * When `useReplica` is set by the user the SDK should check the `clusterCapabilities` to ensure that there is a `n1ql` field containing a `readFromReplica` entry. 126 | * If this is not present then fail with a FeatureNotAvailableException. 127 | * If the query service does read from a replica then it will populate an entry in the `warning` field in the response payload. 128 | 129 | 130 | **Returns:** 131 | 132 | A `QueryResult` that maps the result of the N1QL query to an object. 133 | 134 | **Throws:** 135 | 136 | * Documented 137 | * RequestTimeoutException (#1) 138 | * CouchbaseException (#0) 139 | * Undocumented 140 | * CasMismatchException (#9) 141 | * RequestCanceledException (#2) 142 | * InvalidArgumentException (#3) 143 | * ServiceNotAvailableException (#4) 144 | * InternalServerException (#5) 145 | * AuthenticationException (#6) 146 | * PlanningFailedException (#201) 147 | * QueryIndexException (#202) 148 | * PreparedStatementException (#203) 149 | * ParsingFailedException (#8) 150 | * FeatureNotAvailableException (#15) 151 | 152 | ## IScope#Query 153 | 154 | The API and implementation are identical to `ICluster::Query`, except that all queries will be sent with a `query_context` parameter of ``"default:`bucket-name`.`scope-name`"``. 155 | 156 | ## QueryProfile 157 | 158 | ``` 159 | enum QueryProfile { 160 | Off, 161 | Phases, 162 | Timings 163 | } 164 | ``` 165 | 166 | In the JSON payload, the field name is `profile`. 167 | 168 | | Identifier | Wire Format | Description | 169 | |------------|--------------|-----------------| 170 | | Off | `"off"` | No profiling information is returned from the cluster. This is the default. Not sent on the wire if set. | 171 | | Phases | `"phases"` | Profiling information about the various phases is included. | 172 | | Timings | `"timings"` | In addition to the phases, detailed timing information is available. | 173 | 174 | 175 | ## QueryScanConsistency 176 | 177 | ``` 178 | enum QueryScanConsistency { 179 | NotBounded, 180 | RequestPlus 181 | } 182 | ``` 183 | 184 | In the JSON payload, the field name is `scan_consistency`. 185 | 186 | | Identifier | Wire Format | Description | 187 | |-------------|------------------|-----------------| 188 | | NotBounded | `"not_bounded"` | No scan consistency is used. This is the default and not sent over the wire when set. | 189 | | RequestPlus | `"request_plus"` | The indexer will grab the highest seqnos at query time and wait until indexed before returning. | 190 | 191 | ## Raw Params 192 | 193 | Raw params are the escape hatch for options which are not directly exposed and also future-proofs the API. The API must only accept valid JSON values and/or encoded it into their JSON representations. 194 | 195 | ## QueryResult 196 | 197 | The QueryResult is returned if the response arrived successfully and contains both the associated metadata and the actual rows. 198 | 199 | ``` 200 | struct QueryResult { 201 | Stream Rows(); 202 | Promise MetaData(); 203 | } 204 | ``` 205 | 206 | Depending on the language best practices, rows() returns a list of entities, but how they are decoded is language-dependent. 207 | 208 | ``` 209 | struct QueryMetaData { 210 | String RequestId(); 211 | String ClientContextId(); 212 | QueryStatus Status(); 213 | Optional Signature(); 214 | List Warnings(); 215 | QueryMetrics Metrics(); 216 | Optional Profile(); 217 | } 218 | ``` 219 | 220 | NOTE: If present, the signature value returned by the server is usually a JSON Object or a JSON String, but the server may return any JSON type (Object, Array, String, Number, Boolean). 221 | 222 | Everything but the signature and the profile should always be present. Even when the Metrics are disabled in the options, a default implementation should be provided. The reasoning behind this is that it will be enabled 99% of the time and more and having every user to check for non-existence is more error prone. 223 | 224 | ``` 225 | struct QueryMetrics { 226 | Duration elapsedTime() 227 | Duration executionTime() 228 | uint64 sortCount() 229 | uint64 resultCount() 230 | uint64 resultSize() 231 | uint64 mutationCount() 232 | uint64 errorCount() 233 | uint64 warningCount() 234 | } 235 | ``` 236 | 237 | The status needs to be decoded from the wire representation, which is in all cases the lowercase version of the enum. So "success" on the wire is turning into SUCCESS. 238 | 239 | ``` 240 | enum QueryStatus { 241 | Running, 242 | Success, 243 | Errors, 244 | Completed, 245 | Stopped, 246 | Timeout, 247 | Closed, 248 | Fatal, 249 | Aborted, 250 | Unknown 251 | } 252 | ``` 253 | 254 | A QueryWarning is always a tuple of code and message: 255 | 256 | ``` 257 | class QueryWarning { 258 | int32 code() 259 | String message() 260 | } 261 | ``` 262 | 263 | # Changelog 264 | 265 | * Sept 12, 2019 - Revision #1 266 | * Initial Draft 267 | * Sept 27, 2019 - Revision #2 (by Michael Nitschinger) 268 | * Major refactoring and alignment with the analytics & view rfc 269 | * Sept 27, 2019 - Revision #3 (by Michael Nitschinger) 270 | * Make QueryMetrics optional in QueryMetaData, because now they are disabled by default 271 | * Sept 30, 2019 - Revision #4 (by Michael Nitschinger) 272 | * Clarify that max_parallelism, pipeline_cap, pipeline_batch and scan_cap need to be sent over the wire as Strings not Numbers! 273 | * Oct 16, 2019 - Revision #5 (by Michael Nitschinger) 274 | * QueryStatus enum values are converted from uppercase to camel case 275 | * April 30, 2020 276 | * Moved RFC to ACCEPTED state. 277 | * January 5, 2022 278 | * Added preserveExpiry to QueryOptions. 279 | * June 21, 2023 - Revision #6 (by Charles Dixon) 280 | * Added useReplica to QueryOptions. 281 | * Dec 11, 2023 - Revision #7 (by David Nault) 282 | * Change type of QueryMetadata.signature() from JsonObject to JsonValue. 283 | * Feb 6, 2025 - Revision #8 (by Dimitris Christodoulou) 284 | * Add IScope#Query API. 285 | * April 30, 2025 - Revision #9 (by Dimitris Christodoulou) 286 | * Allow setting both positional and named parameters. 287 | 288 | # Signoff 289 | 290 | | Language | Team Member | Signoff Date | Revision | 291 | |------------|----------------|---------------|----------| 292 | | Node.js | Jared Casey | May 1, 2025 | #9 | 293 | | Go | Charles Dixon | July 6, 2023 | #6 | 294 | | Connectors | David Nault | May 1, 2025 | #9 | 295 | | PHP | Sergey Avseyev | May 7, 2025 | #9 | 296 | | Python | Jared Casey | May 1, 2025 | #9 | 297 | | Scala | Graham Pople | May 2, 2025 | #9 | 298 | | .NET | Jeffry Morris | May 8, 2025 | #9 | 299 | | Java | David Nault | May 1, 2025 | #9 | 300 | | Kotlin | David Nault | May 1, 2025 | #9 | 301 | | C | Sergey Avseyev | May 7, 2025 | #9 | 302 | | C++ | Sergey Avseyev | May 7, 2025 | #9 | 303 | | Ruby | Sergey Avseyev | May 7, 2025 | #9 | 304 | -------------------------------------------------------------------------------- /rfc/0057-sdk3-analytics.md: -------------------------------------------------------------------------------- 1 | # Meta 2 | 3 | * RFC Name: SDK3 Analytics 4 | * RFC ID: 0057-sdk3-analytics 5 | * Start Date: 2019-09-27 6 | * Owner: Michael Nitschinger 7 | * Current Status: ACCEPTED 8 | * Relates To: 9 | [0054-sdk3-management-apis](0054-sdk3-management-apis.md), 10 | [0058-sdk3-error-handling](0058-sdk3-error-handling.md), 11 | [0059-sdk3-foundation](0059-sdk3-foundation.md) 12 | 13 | # Summary 14 | 15 | This RFC is part of the bigger SDK 3.0 RFC and describes the Analytics Query APIs in detail. 16 | 17 | # Technical Details 18 | 19 | ## API Entry Point 20 | 21 | ``` 22 | interface ICluster { 23 | ... 24 | IAnalyticsResult AnalyticsQuery(string statement, [AnalyticsOptions options]); 25 | ... 26 | } 27 | ``` 28 | 29 | ## ICluster#AnalyticsQuery 30 | 31 | The ICluster::AnalyticsQuery method enables a user to perform an analytics query and receive the results in a streaming manner. The SDK should perform this by executing a HTTP GET request against the analytics service. 32 | 33 | **Signature:** 34 | 35 | ``` 36 | IAnalyticsResult AnalyticsQuery(string statement, [AnalyticsOptions options]); 37 | ``` 38 | 39 | **Parameters:** 40 | 41 | * Required: 42 | * statement : string 43 | * Specifies the Analytics statement in string form (i.e. "select foo from bar") 44 | * Optional (part of QueryOptions): 45 | * scanConsistency(AnalyticsScanConsistency) = undefined(NotBounded) 46 | * Specifies the level of consistency for the query. 47 | * Sent within the JSON payload as `scan_consistency` and is encoded as: 48 | * RequestPlus: `request_plus` 49 | * NotBounded: `not_bounded` 50 | * clientContextId(String) = UUID(), 51 | * Specifies a context ID string which is mirrored back from the query engine on response 52 | * If not provided, the client must create and use a UUID instead. Sent as in the JSON payload as "client_context_id" as a JSON String `"foobar"` 53 | * raw(String key, JsonValue value) = undefined 54 | * Specifies values with their key and value as presented as part of the JSON payload 55 | * This is an escape hatch to support unknown commands and be forwards compatible 56 | * readonly(boolean) = undefined(false) 57 | * Allows to specify if the query is readonly 58 | * Sent within the JSON Payload as `readonly` as a JSON boolean 59 | * parameters(JsonObject | JsonArray) = undefined 60 | * Specifies positional or named parameters 61 | * For languages that do not support operator overloading, the alternative naming is positionalParameters(JsonArray) and namedParameters(JsonObject) 62 | * Sent in the JSON payload 63 | * For positional parameters as a JSON array under the "args" key 64 | * For named parameters directly in the JSON payload, but each named argument is prefixed with "$" if it doesn't already have the dollar prefix provided by the user 65 | * The payload can include both positional and named parameters 66 | * Setting JsonArray overrides any previously set _positional_ parameters 67 | * Setting JsonObject overrides any previously set _named_ parameters 68 | * priority(boolean) = undefined(0) 69 | * Allows to give certain requests higher priority than others. 70 | * Sent on the wire in the HTTP header as "Analytics-Priority" a JSON Number. See the section on Priority in this RFC for more details how it should be encoded. 71 | * timeout(Duration) = $Cluster::analyticsOperationTimeout 72 | * SSpecifies how long to allow the operation to continue running before it is cancelled. 73 | * serializer(JsonSerializer) = $Cluster::Serializer 74 | * SSpecifies the serializer which should be used for deserialization of rows returned. 75 | 76 | **Returns:** 77 | 78 | An `IAnalyticsResult` that maps the result of the analytics query to an object. 79 | 80 | **Throws:** 81 | 82 | * Documented 83 | * RequestTimeoutException (#1) 84 | * CouchbaseException (#0) 85 | * Undocumented 86 | * RequestCanceledException (#2) 87 | * InvalidArgumentException (#3) 88 | * ServiceNotAvailableException (#4) 89 | * InternalServerException (#5) 90 | * AuthenticationException (#6) 91 | * CompilationFailedException (#301) 92 | * JobQueueFullException (#302) 93 | * DatasetNotFound (#303) 94 | * ParsingFailedException (#8) 95 | 96 | ## Named and Positional Parameter Handling 97 | 98 | **Note:** named and positional parameters must not be set at the same time. If both are set the SDK must fail immediately with an IllegalArgumentException or similar. 99 | 100 | Named parameters are applied with a prefix "$" to the JSON payload, positional ones as an array in the "args" element. 101 | 102 | ## AnalyticsScanConsistency 103 | 104 | ``` 105 | enum AnalyticsScanConsistency { 106 | NotBounded, 107 | RequestPlus 108 | } 109 | ``` 110 | 111 | In the JSON payload, the field name is `scan_consistency`. 112 | 113 | | Identifier | Wire Format | Description | 114 | |-------------|------------------|-----------------| 115 | | NotBounded | `"not_bounded"` | No scan consistency is used. This is the default and not sent over the wire when set. | 116 | | RequestPlus | `"request_plus"` | The indexer will grab the highest seqnos at query time and wait until indexed before returning. | 117 | 118 | ## Raw Params 119 | 120 | Raw params are the escape hatch for options which are not directly exposed and also future-proofs the API. The API must only accept valid JSON values and/or encoded it into their JSON representations. 121 | 122 | ## Priority 123 | 124 | The server allows the client to specify priority, but it is a numeric value on the wire. So a higher priority means a lower value (i.e. -1 over 0). To make it simpler for a user, this has been mapped to a boolean on the `QueryOptions` API. 125 | 126 | So if priority is set to true, it must be written as numeric -1 on the wire. 127 | 128 | Also it is important that the priority is NOT part of the json blob, but rather set as a http header! 129 | 130 | * Header name: `"Analytics-Priority"` 131 | * Header value: not present if 0 (false), set to -1 if true 132 | 133 | ## AnalyticsResult 134 | 135 | The IAnalyticsResult is returned if the response arrived successfully and contains both the associated metadata and the actual rows. 136 | 137 | ``` 138 | struct AnalyticsResult { 139 | Stream Rows(); 140 | Promise MetaData(); 141 | } 142 | ``` 143 | 144 | Depending on the language best practices, rows() returns a list of entities, but how they are decoded is language-dependent. 145 | 146 | ``` 147 | struct AnalyticsMetaData { 148 | String RequestId(); 149 | String ClientContextId(); 150 | AnalyticsStatus Status(); 151 | Optional Signature(); 152 | List Warnings(); 153 | AnalyticsMetrics Metrics(); 154 | } 155 | ``` 156 | 157 | Only the signature is optional, since it might not be present all the time. 158 | 159 | 160 | ``` 161 | struct AnalyticsMetrics { 162 | Duration elapsedTime() 163 | Duration executionTime() 164 | uint64 resultCount() 165 | uint64 resultSize() 166 | uint64 errorCount() 167 | uint64 processedObjects() 168 | uint64 warningCount() 169 | } 170 | ``` 171 | 172 | The status needs to be decoded from the wire representation, which is in all cases the lowercase version of the enum. So "success" on the wire is turning into SUCCESS. 173 | 174 | ``` 175 | enum AnalyticsStatus { 176 | RUNNING, 177 | SUCCESS, 178 | TIMEOUT, 179 | FAILED, 180 | FATAL, 181 | UNKNOWN 182 | } 183 | ``` 184 | 185 | A AnalyticsWarning is always a tuple of code and message: 186 | 187 | ``` 188 | class AnalyticsWarning { 189 | int32 code() 190 | String message() 191 | } 192 | ``` 193 | 194 | # Changelog 195 | 196 | * Sept 27, 2019 - Revision #1 (by Michael Nitschinger) 197 | * Initial Draft 198 | * April 30, 2020 199 | * Moved RFC to ACCEPTED state. 200 | * May 20, 2024 - Revision #2 (by Dimitris Christodoulou) 201 | * Correct `AnalyticsStatus` enum values. 202 | * April 30, 2025 - Revision #3 (by Dimitris Christodoulou) 203 | * Allow setting both positional and named parameters. 204 | 205 | # Signoff 206 | 207 | | Language | Team Member | Signoff Date | Revision | 208 | |------------|---------------------|-----------------|----------| 209 | | Node.js | Jared Casey | May 1, 2025 | #3 | 210 | | Go | Charles Dixon | April 22, 2020 | #1 | 211 | | Connectors | David Nault | May 1, 2025 | #3 | 212 | | PHP | Sergey Avseyev | May 7, 2025 | #3 | 213 | | Python | Jared Casey | May 1, 2025 | #3 | 214 | | Scala | Graham Pople | May 2, 2025 | #3 | 215 | | .NET | Jeffry Morris | May 8, 2025 | #3 | 216 | | Java | David Nault | May 1, 2025 | #3 | 217 | | Kotlin | David Nault | May 1, 2025 | #3 | 218 | | C | Sergey Avseyev | May 7, 2025 | #3 | 219 | | C++ | Sergey Avseyev | May 7, 2025 | #3 | 220 | | Ruby | Sergey Avseyev | May 7, 2025 | #3 | 221 | -------------------------------------------------------------------------------- /rfc/0061-sdk3-diagnostics.md: -------------------------------------------------------------------------------- 1 | # Meta 2 | 3 | - RFC Name: SDK3 Diagnostics 4 | - RFC ID: 0061-sdk3-diagnostics 5 | - Start Date: November 12, 2019 6 | - Owner: Michael Nitschinger \ 7 | - Current Status: ACCEPTED 8 | - Relates To: 9 | - SDK-RFC#59 (SDK3 Foundation) 10 | - [Original Google Drive Doc](https://docs.google.com/document/d/1Lw3nuYVtRbXYIujeCxakcyYQWFs2OAj4Borb75eNe5Y) 11 | 12 | # Motivation 13 | 14 | This SDK3 RFC describes the various ping and diagnostic operations available throughout the SDK. 15 | 16 | # Summary 17 | 18 | The various diagnostic options are available at multiple API levels throughout the SDK. 19 | 20 | ```java 21 | enum ClusterState { 22 | Online, // all nodes and their sockets are reachable 23 | Degraded, // at least one socket per service is reachable 24 | Offline, // not even one socket per service is reachable 25 | } 26 | 27 | enum EndpointState { 28 | Disconnected, // the endpoint socket is not reachable 29 | Connecting, // currently connecting (includes auth,...) 30 | Connected, // connected and ready 31 | Disconnecting, // disconnecting (after being connected) 32 | } 33 | ``` 34 | 35 | ## Cluster-Level 36 | 37 | ```java 38 | class Cluster { 39 | 40 | ... 41 | 42 | void waitUntilReady(Duration timeout, [WaitUntilReadyOptions options]) 43 | 44 | DiagnosticsResult diagnostics([DiagnosticsOptions options]) 45 | PingResult ping([PingOptions options]) 46 | } 47 | ``` 48 | 49 | ## Bucket-Level 50 | 51 | ```java 52 | class Bucket { 53 | 54 | ... 55 | 56 | void waitUntilReady(Duration timeout, [WaitUntilReadyOptions options]) 57 | 58 |   PingResult ping([PingOptions options]) 59 | } 60 | ``` 61 | 62 | ## The Diagnostics Command 63 | 64 | Creates diagnostic report that can be used to determine the healthfulness of the cluster. It does not proactively perform any I/O against the network. 65 | 66 | Signature 67 | 68 | ```typescript 69 | DiagnosticsResult Diagnostics([DiagnosticsOptions options]); 70 | ``` 71 | 72 | Parameters 73 | 74 | - Required 75 | - None 76 | - Optional 77 | - reportId - an optional string name for the generated diagnostics report. 78 | 79 | Returns 80 | 81 | - A DiagnosticsReport object detailing the current status of the SDK. 82 | 83 | Throws 84 | 85 | - `CouchbaseException` if anything unexpected comes up (like invalid arguments), but since it is doing I/O usually only validation errors will be thrown 86 | 87 | #### DiagnosticsResult JSON Output 88 | 89 | The SDK Must provide a method on the DiagnosticsResult to export it to JSON into this format: 90 | 91 | ```json 92 | { 93 | "version": 2, // This is a uint, and is always 2 94 | "id": "0xdeadbeef", 95 | "sdk": "gocbcore/v7.0.2", 96 | "services": { 97 | "search": [ 98 | { 99 | "id": "0x1415F11", 100 | "last_activity_us": 1182000, 101 | "remote": "centos7-lx1.home.ingenthron.org:8094", 102 | "local": "127.0.0.1:54669", 103 | "state": "reconnecting", 104 | "details": "RECONNECTING, backoff for 4096ms from Fri Sep  1 00:03:44 PDT 2017" 105 | } 106 | ], 107 | "kv": [ 108 | { 109 | "id": "0x1415F12", 110 | "last_activity_us": 1182000, 111 | "remote": "centos7-lx1.home.ingenthron.org:11210", 112 | "local": "127.0.0.1:54670", 113 | "state": "connected", 114 | "namespace": "bucketname" 115 | } 116 | ], 117 | "query": [ 118 | { 119 | "id": "0x1415F13", 120 | "last_activity_us": 1182000, 121 | "remote": "centos7-lx1.home.ingenthron.org:8093", 122 | "local": "127.0.0.1:54671", 123 | "state": "connected" 124 | }, 125 | { 126 | "id": "0x1415F14", 127 | "last_activity_us": 1182000, 128 | "remote": "centos7-lx2.home.ingenthron.org:8095", 129 | "local": "127.0.0.1:54682", 130 | "state": "disconnected" 131 | } 132 | ], 133 | "analytics": [ 134 | { 135 | "id": "0x1415F15", 136 | "last_activity_us": 1182000, 137 | "remote": "centos7-lx1.home.ingenthron.org:8095", 138 | "local": "127.0.0.1:54675", 139 | "state": "connected" 140 | } 141 | ], 142 | "views": [ 143 | { 144 | "id": "0x1415F16", 145 | "last_activity_us": 1182000, 146 | "remote": "centos7-lx1.home.ingenthron.org:8092", 147 | "local": "127.0.0.1:54672", 148 | "state": "connected" 149 | } 150 | ] 151 | } 152 | } 153 | ``` 154 | 155 | ## The waitUntilReady Command 156 | 157 | Waits until a desired cluster state by default ("online") is reached or times out. 158 | 159 | This command lives both at the Cluster and at the Bucket level. At the cluster level it waits until cluster-level resources are ready (query, search,... and gcccp connections if available). At the bucket level it ALSO includes bucket-level resources (view connections and bucket-scoped kv connections). 160 | 161 | Signature 162 | 163 | ```java 164 | void waitUntilReady(Duration timeout, [WaitUntilReadyOptions options]); 165 | ``` 166 | 167 | Parameters 168 | 169 | - Required 170 | - Timeout: The timeout duration the user is willing to wait 171 | - Optional 172 | - desiredState - ClusterState the desired state (default ONLINE). 173 | - serviceTypes - the set of service types to check, if not provided will use all services from the config 174 | 175 | Throws 176 | 177 | - `CouchbaseException` if anything unexpected comes up (like invalid arguments) 178 | - `UnambiguousTimeoutException` if the expected state cannot be reached in the given timeout 179 | 180 | `waitUntilReady` will perform a ping if it has not seen activity on a specific service. If there has been activity it will use this information to determine readiness (i.e. by using diagnostics). See the `ClusterState` enum that defines the semantics when to return for the desired state. 181 | 182 | Optionally, the SDK may ensure only a single call through synchronization or a lock, and/or return the same object to await on multiple calls. 183 | 184 | Note: It is a hard error if OFFLINE is passed in, but keeping it for future use (i.e. returning a cluster state somewhere). In addition, it is also present at any async or reactive APIs with their equivalent Async return types. 185 | 186 | For now, no checks are added to the Collection level. 187 | 188 | ## The ping Command 189 | 190 | Actively performs I/O by application-level pinging services and returning their pinged status. This API is available on the Cluster and on the Bucket level. 191 | 192 | Signature 193 | 194 | ```java 195 | PingResult ping([PingOptions options]); 196 | ``` 197 | 198 | Parameters 199 | 200 | - Required 201 | - None 202 | - Optional 203 | - reportId - an optional string name for the generated ping report 204 | - timeout - the timeout for all ping commands (if no timeout is chosen, the individual timeouts will be used as configured on the config) 205 | - serviceTypes - the set of service types to check, if not provided will use all services from the config 206 | 207 | Returns 208 | A PingResult object representing the result of each ping performed. 209 | 210 | Throws 211 | 212 | - `CouchbaseException` if anything unexpected comes up (like invalid arguments) (note that timeouts must not be propagated but will be surfaced as individual errors in the report) 213 | 214 | #### PingResult 215 | 216 | ```java 217 | interface PingResult { 218 | String Id(); 219 | Int16 Version(); // VERSION IS 2 220 | String Sdk(); 221 | Map> Endpoints(); 222 | } 223 | 224 | interface EndpointPingReport { 225 | ServiceType Type(); 226 | String Id(); 227 | String Local(); 228 | String Remote(); 229 | PingState State(); 230 | Optional Error(); // if ping state is error, contains the error message what appened 231 | Optional Namespace(); 232 | Duration Latency(); 233 | } 234 | 235 | enum PingState { 236 | Ok, 237 | Timeout, 238 | Error 239 | } 240 | ``` 241 | 242 | #### PingResult JSON Output 243 | 244 | The SDK Must provide a method on the PingResult to export it to JSON into this format: 245 | 246 | ```json 247 | { 248 | "version": 2, 249 | "id": "0xdeadbeef", 250 | "config_rev": 53, 251 | "sdk": "gocbcore/v7.0.2", 252 | "services": { 253 | "sSearch": [ 254 | { 255 | "id": "0x1415F11", 256 | "latency_us": 877909, 257 | "remote": "centos7-lx1.home.ingenthron.org:8094", 258 | "local": "127.0.0.1:54669", 259 | "state": "ok" 260 | } 261 | ], 262 | "kKv": [ 263 | { 264 | "id": "0x1415F12", 265 | "latency_us": 1182000, 266 | "remote": "centos7-lx1.home.ingenthron.org:11210", 267 | "local": "127.0.0.1:54670", 268 | "state": "ok", 269 | "namespace": "bucketname" 270 | } 271 | ], 272 | "query": [ 273 | { 274 | "id": "0x1415F14", 275 | "latency_us": 2213, 276 | "remote": "centos7-lx2.home.ingenthron.org:8095", 277 | "local": "127.0.0.1:54682", 278 | "state": "timeout" 279 | } 280 | ], 281 | "analytics": [ 282 | { 283 | "id": "0x1415F15", 284 | "latency_us": 2213, 285 | "remote": "centos7-lx1.home.ingenthron.org:8095", 286 | "local": "127.0.0.1:54675", 287 | "state": "error", 288 | "error": "endpoint returned HTTP code 500!" 289 | } 290 | ], 291 | "views": [ 292 | { 293 | "id": "0x1415F16", 294 | "latency_us": 45585, 295 | "remote": "centos7-lx1.home.ingenthron.org:8092", 296 | "local": "127.0.0.1:54672", 297 | "state": "ok" 298 | } 299 | ] 300 | } 301 | } 302 | ``` 303 | 304 | ## Enumerations 305 | 306 | ### ServiceType (type: any) 307 | 308 | - KV (json: "kv") 309 | - Query (json: "query") 310 | - Analytics (json: "analytics") 311 | - Search (json: "search") 312 | - Views (json: "views") 313 | - Manager (json: "mgmt") 314 | 315 | # Changelog 316 | 317 | - Nov 12, 2019 - Revision #1 (by Michael Nitschinger) 318 | 319 | - Initial Draft 320 | 321 | - Nov 18, 2019 - Revision #2 (by Michael Nitschinger) 322 | 323 | - Added waitUntilReady 324 | 325 | - Dec 12, 2019 - Revision #3 (by Michael Nitschinger) 326 | 327 | - Make timeout mandatory on waitUntilReady 328 | 329 | - Dec 16, 2019 - Revision #4 (by Michael Nitschinger) 330 | 331 | - Renamed Services() to Endpoints() for consistency because it is called EndpointDiagnostics 332 | - Changed the Key of the endpoints() map from String to ServiceType - since we define this enum already it is much easier for the user to get it that way rather than a very generic String type. Also since we can have many endpoints per service type, the API has been changed to a list. Result:\ 333 | Before Map\ 334 | After Map> 335 | - Added EndpointState enum similarly to ClusterState so it can be observed at the individual level too (and we need to turn it into a string anyways for the JSON export) 336 | - Moved "scope" to "namespace" because scope is now an overloaded term with collections 337 | - Changed lastActivity simple int64 to Duration to align with the other RFCs on how to represent a time duration. Also changed to an optional, since on the diagnostics API there might have not been activity on that endpoint yet. 338 | 339 | - Dec 17, 2019 - Revision #5 (by Michael Nitschinger) 340 | 341 | - Removed ping as its own command and is subsumed by the ping option on the diagnostics command 342 | - Added a separate command section for waitUntilReady 343 | 344 | - Dec 18, 2019 - Revision #6 (by Michael Nitschinger) 345 | 346 | - Removed ping option again from diagnostics 347 | - Re-added and clarified on the ping command 348 | - Reorganized the rfc and cleanup 349 | 350 | - Dec 21 2019 - Revision #7 (by Michael Nitschinger) 351 | 352 | - Added options to ping 353 | - Moved ping result back to a List since it can still return a list of EndpointPingReports if you have multiple nodes 354 | 355 | - April 16, 2020 - Revision #8 (by Michael Nitschinger) 356 | 357 | - Removed version from DiagnosticResult, made it clear in the JSON output that value type is of uint and always 2. Note: if you've already implemented it, you can deprecate it and comment here. 358 | - Added ServiceType enumeration 359 | 360 | - April 30, 2020 (by Michael Nitschinger) 361 | 362 | - Moved RFC to ACCEPTED state. 363 | 364 | - Sept 17, 2021 (by Brett Lawson) 365 | 366 | - Converted to Markdown 367 | 368 | # Signoff 369 | 370 | | Language | Team Member | Signoff Date | Revision | 371 | | ---------- | ------------------- | ------------ | -------- | 372 | | Node.js | Brett Lawson | 2020-04-16 | 8 | 373 | | Go | Charles Dixon | 2020-04-22 | 8 | 374 | | Connectors | David Nault | 2020-04-29 | 8 | 375 | | PHP | Sergey Avseyev | 2020-04-22 | 8 | 376 | | Python | Ellis Breen | 2020-04-29 | 8 | 377 | | Scala | Graham Pople | 2020-04-30 | 8 | 378 | | .NET | Jeffry Morris | 2020-04-23 | 8 | 379 | | Java | Michael Nitschinger | 2020-04-16 | 8 | 380 | | C | Sergey Avseyev | 2020-04-22 | 8 | 381 | | Ruby | Sergey Avseyev | 2020-04-22 | 8 | 382 | -------------------------------------------------------------------------------- /rfc/0069-kv-error-map-v2.md: -------------------------------------------------------------------------------- 1 | # Meta 2 | 3 | - RFC Name: KV Error Map V2 4 | - RFC ID: 0069-kv-error-map-v2 5 | - Start Date: 2021-09-17 6 | - Owner: Brett Lawson 7 | - Current Status: DRAFT 8 | - Supersedes 9 | - SDK-RFC#13 (KV Error Map) 10 | 11 | # Summary 12 | 13 | This proposal outlines the SDK handling of error categories and attributes defined by Couchbase Server 5.0 and above (through [kv-engine](https://github.com/couchbase/kv_engine)), including the changes introduced by Couchbase Server 7.0 for Error Map V2. 14 | 15 | # Motivation 16 | 17 | Prior to the introduction of Error Map, error codes were explicitly defined, or occasionally were defined as ranges of values. In many cases, new server versions would introduce new error codes which older clients were unaware of. This would lead to clients responding to error situations in unexpected ways, and in some cases leading to clients which were unable to make forwards progress. Consequently, error code changes were difficult to introduce to the server, in spite of the potential to significantly improve behaviour (for example, it is much more helpful to return an error code `LOCKED` if an item is locked, than a `TMPFAIL` which was done previously). Starting in Couchbase Server 5.0, clients are able to request a mapping of the error codes that the server will use when communicating with us, along with various meta-data about those errors and how they should be handled. Starting in Couchbase Server 7.5, this error map has been enhanced with some additional guidelines and rules to ensure its continued usefulness. 18 | 19 | # General Design 20 | 21 | The server exposes the error map as a simple JSON blob that contains the mappings of known error codes to their attributes. The client requests this error map from the server during the negotiation phase (after `HELLO`). When the server responds with an error code that the client otherwise does not know how to handle, it refers to the error map and determines (semi-heuristically) the best course of action. 22 | 23 | ## Control Flow 24 | 25 | 1. The client must first advertise that it supports the KV Error Map feature by including the `XERROR (0x07)` feature in the HELLO negotiation command. This tells the server that it may send newer error codes to the client without risking confusing it. It also implies that the client will retrieve the error code map from the server. 26 | 2. Once the client has sent the `XERROR` feature and receive indication that the server also supports it (by receiving `XERROR` included in the server's list of features), it sends a `CMD_GET_ERROR_MAP (0xfe)`, indicating the maximum format version of the error map the client understands. There are currently two supported versions: 1 (Couchbase Server 5.0+) and 2 (Couchbase Server 7.5+). Clients should always request the highest version that they support, though the server may respond with a lower version and this lower version must still be parsed and used. 27 | 3. The server will reply with a JSON error map. The client must store this error map on a per node basis. The version and revision are contained in the error map, as well as the actual code-to-attribute mappings. The version is used to indicate new error attributes, whereas the revision is used to indicate updated error codes. 28 | 4. When the client receives an unrecognized error code, it can consult the error map (as below). 29 | 30 | ## Negotiation 31 | 32 | The client should first indicate that it supports the KV Error Map feature by sending the `XERROR (0x07)` feature flag in `HELLO`. If the server responds back with `XERROR` in the list of features, then proceed to the next step. Otherwise, the server doesn't support extended error codes. 33 | 34 | _Note:_ `HELLO` should be done as the first step, even before auth, so that auth can return more detailed error information. This is known to break certain old and unsupported server versions, so if possible an escape hatch like a flag should be introduced which performs `HELLO` after authentication to keep them working. 35 | 36 | **The XERROR bit should be sent by default.** Since the client is under authority when to consult the error map, more information is always good but the clients need to ensure that backwards compatibility is not broken. So in some cases newer error codes or attributes need to be mapped back to old behavior when encountered to make sure the semantics of the SDK don't change with different server versions. 37 | 38 | ## Getting the Error Map 39 | 40 | To get the error map, send the `CMD_GET_ERROR_MAP (0xfe)` to the server. This should be done immediately after the client has received acknowledgment that the server actually supports extended error codes, as the server is now free to send any error code it wants in reply. 41 | 42 | The `GET_ERROR_MAP` command requires a 2 byte payload, which is the network-encoded maximum version number the client can parse. At the time of this RFC, there are two supported versions. Clients should always request the highest version they support. The server should send back a reply with the JSON error map in the payload. Note that the error map payload version returned from the server may not be the same as was requested in the case of an older server version. These older versions must still be parsed and used. 43 | 44 | ## Parsing the Error Map 45 | 46 | The error map's format is defined in the [memcached Github repository](https://github.com/couchbase/kv_engine/blob/master/docs/ErrorMap.md), and will not be repeated here again. The client should parse the error map and ensure that all error attributes are understood and accounted for. If the error map seems corrupted in any way, it should ignore it, and renegotiate with the server (or reconnect) without the `XERROR` feature. 47 | 48 | Once the error map has been received, it should be stored in a per-node database. Different nodes may wish to dictate how different errors behave (for example, in respect to different cluster versions and/or retries). 49 | 50 | ## Handling Error Attributes 51 | 52 | Error attributes beyond the retry attributes (discussed further down), which is utilized to indicate that a client must use the retry semantics of the particular error are not in scope for this document. A future RFC will provide details on additional attributes that may be added, and the behaviors they will provide. If in doubt if an attribute should be used, the SDK should use a conservative approach and favor backwards compatibility. 53 | 54 | # Error Attributes 55 | 56 | Error Categories act as the driving force behind the utility of the error map. Categories define the basic characteristics of an error condition so that even if the SDK does not understand a specific error code, it may determine the next course of action based on the error's defined attributes. 57 | 58 | Attributes are the most important part of the error map as they convey how an error code should be handled. An error code may have more than a single attribute. 59 | 60 | - `item-only`: This attribute means that the error is related to a constraint failure regarding the item itself, i.e. the item does not exist, already exists, or its current value makes the current operation impossible. Retrying the operation when the item's value or status has changed may succeed. 61 | - `invalid-input`: This attribute means that a user's input was invalid because it violates the semantics of the operation, or exceeds some predefined limit. 62 | - `fetch-config`: The client's cluster map may be outdated and requires updating. The client should obtain a newer configuration. 63 | - `conn-state-invalidated`: The current connection is no longer valid. The client must reconnect to the server. Note that the presence of other attributes may indicate an alternate remedy to fixing the connection without a disconnect, but without special remedial action a disconnect is needed. 64 | - `auth`: The operation failed because the client failed to authenticate or is not authorized to perform this operation. Note that this error in itself does not mean the connection is invalid, unless conn-state-invalidated is also present. 65 | - `special-handling`: This error code must be handled specially. If it is not handled, the connection must be dropped. 66 | - `support`: The operation is not supported, possibly because the of server version, bucket type, or current user. 67 | - `temp`: This error is transient. Note that this does not mean the error is retriable. 68 | - `internal`: This is an internal error in the server. 69 | - `retry-now`: The operation may be retried immediately. 70 | - `retry-later`: The operation may be retried after some time. 71 | - `subdoc`: The error is related to the subdocument subsystem. 72 | - `dcp`: The error is related to the DCP subsystem. 73 | - `rate-limit`: The error is related to rate limiting being applied to the current connection or user. 74 | 75 | The most up-to-date list of attributes can to be found under https://github.com/couchbase/memcached/blob/master/docs/ErrorMap.md#error-attributes. 76 | 77 | It is important to note that a single error code may contain more than one attribute. Some attributes are informational by nature, while others may affect the flow of the SDK, and yet others may affect the flow of the application. Additionally, to support clean forwards compatibility, the SDKs must be designed such that unrecognized attributes are silently ignored and behaviours defined for any of the known attributes are still applied. 78 | 79 | It is important to reiterate while the attributes are important for unknown error codes to decide what to do next, the SDKs must favor backwards compatibility for known status codes / responses, and the error map attributes should only be used to decide in the case that an unknown status code is encountered. 80 | 81 | # Retry Intervals 82 | 83 | While the error map includes various values which can be used for the specification of client retry behaviour, clients should not currently make use of these values instead opting to use the configured retry strategy in the SDK. 84 | 85 | # Processing Chain 86 | 87 | Making use of the error map consists of the SDK consistently and intelligently determining the best course of action as well as the best error to report for any error code depending on its specific attributes. 88 | 89 | ## Processing Order Example 90 | 91 | Note that this RFC does not specify exactly how each SDK should handle all the attributes (to follow in a new RFC). Here are some examples that might apply: 92 | 93 | 1. _Special-handling._ This error must be specially handled by the client. The connection must be dropped and reinitialized if it can't handle the error properly. 94 | 2. Fetch-config: Hint that the client may wish to get a new cluster map. This attribute does not affect the processing of the given error, but may help avoid receiving such an error in the future. 95 | 3. Temporary: Informational attribute that may be relayed to the user -- the operation may be retried at some point in the future without any modification 96 | 4. Item-only: Informational attribute that may be relayed to the user 97 | 5. Auth: AuthZ or AuthN error (usually the former). Informational attribute that may be relayed to the user. 98 | 6. Conn-state-invalidated: Client must drop the connection and reconnect. This may be ignored if the client otherwise knows how to handle the error (e.g. `AUTH_STALE`). 99 | 100 | # Reserved Ranges 101 | 102 | While the error map allows the server to technically send us any error within the range of 0-65536, and clients must handled any error code within this range, the range 0xff00-0xffff is reserved for testing/proxy use and will never be included in an error map dispatched by the server. This should have no impact on the SDK implementation. 103 | 104 | # Language Specifics 105 | 106 | Currently there are no known language specifics for this RFC. 107 | 108 | # Questions 109 | 110 | **(Unknown) Should HELLO or AUTH be done first?** 111 | We can HELLO at any time. HELLO should be done first, as a result. Note that in many cases, many of our initial bootstrap operations can be pipelined together as a single stream of commands. Note: older servers may not be happy with HELLO first and disconnect us, this is clarified in the RFC above now. 112 | 113 | **(Matt) How are mixed version clusters handled? 4.6 and 5.0 nodes? 5.0 and 5.1 nodes?** 114 | The client will keep track of the error map on a per-node basis so that each node has the highest version of the error map it can support. In practice its expected that the changes are additive so its going to be a minor detail (mn). 115 | 116 | **(Unknown) Precedence of SDK settings over cluster settings?** 117 | SDK settings must always take precedence and only unknown response status codes should be looked up. The SDK is the authority. 118 | 119 | **(Unknown) How does the client request a compatible version?** 120 | At the protocol level, the client can just write the version maximum version it desires as part of the "key". 121 | 122 | **(Unknown) How is logging handled for each of these? Predefined? Part of the response? Goal, I think, is common behaviors.** 123 | Logging is determined based on the general SDK logging facilities. Errors are normally somewhere at WARN while others might be DEBUG. The SDK should just hook this up into the usual logging flow. 124 | 125 | **(Unknown) What happens if an outer operation is specified by the user to be 300s, but the `MaxDuration` of the retry specification specifies something different?** 126 | The shorter time between the two options should always take precedence. Note that the retry specification can specify a `MaxDuration` of 0 to indicate to follow the client behavior. 127 | 128 | **(Brett) Which error map should the client be using when it processes the error map for sub-document operations? Is it safe to use the highest possible version?** 129 | The client should utilize the highest revision error map that it has on a per node basis, in the same manner that it processes error maps for all other kinds of status codes. 130 | 131 | **(Brett) What happens if the error code changes between retry behaviors?** 132 | If the error code changes between retry behaviors, than the new retry behavior should be utilized, or the operation failed immediately in the case of a non-retriable error. Note that the retry count should be reset when a new, different error code is received. 133 | 134 | **(Brett) What should the strategy be if the client receiving the error map cannot handle that retry possibility or behavior attribute?** 135 | Bubble the error back to the user. We should never end up in this kind of situation though, because the version exchanged in the GET_ERROR_MAP guarantees the server will not give the client a retry strategy that it does not understand. In any case, the unknown attribute should be ignored. 136 | 137 | # Changelog 138 | 139 | - Sept 17, 2021 - Revision \#1 (by Brett Lawson) 140 | 141 | - Copied 0013-kv-error-map which covers Error Map V1. 142 | 143 | - Sept 17, 2021 - Revision \#2 (by Brett Lawson) 144 | 145 | - Updated 'Summary' section to indicate specific coverage of v2 error maps. 146 | - Updatec 'Control Flow' to cover the new version number that should be used when requesting error maps. 147 | - Updated 'Getting the Error Map' to cover the new version number. 148 | - Updated 'Error Attributes' section with the newly introduced `rate-limit` attribute. 149 | - Updated 'Error Attributes' section to clarify that unrecognized attributes must be ignored by the SDK. 150 | - Updated 'Error Attributes' section to clarify that attributes should only be used to decide SDK behaviour 151 | when unknown status codes are received, and SDK-defined behaviour should always be used for known status codes. 152 | -------------------------------------------------------------------------------- /rfc/0071-http-client.md: -------------------------------------------------------------------------------- 1 | # RFC-0071 HTTP Client 2 | 3 | ## Meta 4 | 5 | * RFC Name: HTTP Client 6 | * RFC ID: 0071-http-client 7 | * Start Date: 2022-01-19 8 | * Owner: David Nault 9 | * Current Status: Draft 10 | 11 | ## Summary 12 | 13 | Specification for a public SDK 3 API for making custom HTTP requests to Couchbase Server. 14 | 15 | ## Motivation 16 | 17 | Couchbase Server has a well-documented public HTTP interface. 18 | The SDK 3 Management APIs provide abstractions over much of the HTTP interface, but some advanced options are not available in the Management APIs. 19 | To cover those gaps, we'd like to make it easier for developers to make custom HTTP requests. 20 | 21 | An HTTP client built into the SDK relieves developers of these burdens: 22 | 23 | * **Routing the request to an appropriate Couchbase Server node.** 24 | In a deployment that uses multidimensional scaling, some services are only available on certain nodes. 25 | The SDK already has a map of where the services are running, and uses this information for request routing. 26 | * **Establishing a secure connection.** 27 | The SDK already knows which TLS certificates to trust. 28 | * **Presenting user credentials.** 29 | The SDK already knows the username and password, or in the case of mutual TLS, the client certificate. 30 | 31 | In other words, the SDKs are good at dispatching secure, authenticated HTTP requests to Couchbase Server. 32 | The goal is to expose this capability in a way that's easy and safe for developers to use. 33 | 34 | ## Limitations 35 | 36 | This is not a general-purpose HTTP client; it can only talk to Couchbase Server. 37 | 38 | The HTTP client is not intended for making requests that return streaming responses (like executing a N1QL query, for example). 39 | The entire response is expected to fit comfortably in memory. 40 | 41 | ## Requirements 42 | 43 | A compliant implementation has the following capabilities: 44 | 45 | 46 | * The client inherits credentials and transport security settings from the `Cluster`. 47 | 48 | 49 | * A user specifies which Couchbase service should handle the request, and the client dispatches the request to an appropriate host and port. 50 | 51 | 52 | * The supported HTTP verbs are: GET, POST, PUT, and DELETE. 53 | 54 | 55 | * All requests have a path. 56 | 57 | 58 | * GET requests may have query parameters. 59 | The user should be able to specify query parameters separately from the path (so the client can automatically URL-encode them). 60 | 61 | 62 | * POST and PUT requests may have a body with content type `application/json` or `application/x-www-form-urlencoded`. 63 | The client automatically sets the "Content-Type" header to the appropriate value. 64 | 65 | 66 | * A user can specify additional HTTP request headers. 67 | 68 | 69 | * A user can specify query strings and form bodies with duplicate names. For example, `color=green&color=blue`. 70 | 71 | 72 | * The client automatically URL-encodes names and values in query parameters and form bodies, unless the user goes out of their way to bypass the encoding. 73 | An "escape hatch" allows users to provide a pre-encoded query string or form body. 74 | 75 | 76 | * The client provides a way to replace placeholders (`{}`) in a path with URL-encoded arguments. 77 | 78 | 79 | * The user can access the HTTP status code and content of the server response. 80 | 81 | ### Accessing the HTTP client 82 | 83 | The HTTP client is available via the `Cluster`. 84 | `Cluster` has a new `httpClient()` method that returns a `CouchbaseHttpClient`. 85 | 86 | ```java 87 | class Cluster { 88 | ... 89 | 90 | /** 91 | * Returns an HTTP client configured to talk to this cluster. 92 | */ 93 | CouchbaseHttpClient httpClient() 94 | } 95 | ``` 96 | 97 | ## Reference Design 98 | 99 | This section offers recommendations for implementing the HTTP client. 100 | Implementers should adhere to the reference design where it makes sense, and diverge where the idioms of a specific language or SDK provide a more natural solution. 101 | 102 | Except where noted, ***all properties should be considered implementation details and hidden from the user*** (made `private` or equivalent). 103 | 104 | ### HttpTarget 105 | 106 | Immutable value object. 107 | Specifies where to dispatch the request. 108 | 109 | #### Properties 110 | 111 | * `serviceType` Enum. Identifies the Couchbase service (MANAGER, QUERY, etc.) that should service the request. 112 | This is the SDK's existing ServiceType enum, if it has one. 113 | 114 | NOTE: Since HttpTarget has only one property, you might wonder why we don't just use the existing ServiceType enum instead. 115 | One reason is to prevent users from thinking they can make HTTP requests to the KV service. 116 | Also, a future enhancement might let a user target a specific node, and the node identifier would likely be part of the HttpTarget. 117 | Also, the Views service requires the bucket name so the SDK can route the request to a node hosting active partitions for the bucket. 118 | Even though we're not supporting the Views service, it's an example of how additional context may be required to target future services. 119 | 120 | #### Factory methods / Constructors 121 | 122 | 1. `HttpTarget.analytics()` 123 | 2. `HttpTarget.backup()` 124 | 3. `HttpTarget.eventing()` 125 | 4. `HttpTarget.manager()` 126 | 5. `HttpTarget.query()` 127 | 6. `HttpTarget.search()` 128 | 129 | ### HttpPath 130 | 131 | Immutable value object. 132 | Represents the "path" component of a URI. 133 | 134 | #### Factory methods / Constructors 135 | 136 | 1. `HttpPath.of(String template, String... args)` 137 | 138 | This method has two parameters: a template string, and a varargs array (or List) of argument strings. 139 | 140 | Each `{}` placeholder in the template string is replaced with the URL-encoded form of the corresponding additional argument. 141 | For example: 142 | ``` 143 | HttpPath.of("/foo/{}/bar/{}", "hello world", "a/b") 144 | ``` 145 | creates the path `/foo/hello%20world/bar/a%2Fb` 146 | 147 | If the number of placeholders in the template string does not match the number of additional arguments, this method throws IllegalArgumentException (or equivalent). 148 | 149 | **NOTE:** At the implementor's discretion, a path may be represented as a string instead of an `HttpPath`. 150 | In that case, the implementation should provide a static `formatPath(String template, String... args)` method that performs the placeholder substitution. 151 | 152 | ### NameValuePairs 153 | 154 | Immutable value object. 155 | Represents a query string or form data. 156 | 157 | #### Properties 158 | 159 | * `urlEncoded` String. The URL-encoded form of the query string or form data (not including the leading `?`) 160 | 161 | #### Factory methods / Constructors 162 | 163 | 1. `NameValuePairs.of(Map)` 164 | Takes a map from name to value. 165 | The value may be any type that is convertible to a String (or just String, depending on the language). 166 | URL-encodes the names and values before joining them to initialize `urlEncoded`. 167 | The order of the encoded parameters is determined by the map's iteration order. 168 | 169 | ```java 170 | NameValuePairs nvp = NameValuePairs.of( 171 | Map.of( 172 | "friendly greeting", "hello world", 173 | "number", 3 174 | ) 175 | ) 176 | 177 | // friendly%20greeting=hello%20world&number=3 178 | ``` 179 | 180 | 2. `NameValuePairs.of(List>)` 181 | Takes a list of pairs (name -> value). 182 | Languages with varargs may provide an additional overload that accepts a variable number of pairs. 183 | The value may be any type that is convertible to a String (or just String, depending on the language). 184 | URL-encodes the names and values before joining them to initialize `urlEncoded`. 185 | The order of the encoded parameters is the same as their order in the given list. 186 | 187 | ```java 188 | NameValuePairs nvp = NameValuePairs.of( 189 | List.of( 190 | Pair.of("friendly greeting", "hello world"), 191 | Pair.of("friendly greeting", "Olá"), 192 | Pair.of("number", 3), 193 | ) 194 | ) 195 | 196 | // friendly%20greeting=hello%20world&friendly%20greeting=Ol%C3%A1&number=3 197 | ``` 198 | 199 | 3. `NameValuePairs.ofPreEncoded(String)` 200 | This is the "escape hatch" where a developer can supply their own pre-encoded string. 201 | ```java 202 | NameValuePairs nvp = NameValuePairs.ofPreEncoded( 203 | "Garbage in, garbage out!" 204 | ) 205 | 206 | // Garbage in, garbage out! 207 | ``` 208 | 209 | ### HttpBody 210 | 211 | Immutable value object. 212 | Represents the body of a POST or PUT request. 213 | 214 | #### Properties 215 | 216 | * `contentType` String. 217 | Either "application/json" or "application/x-www-form-urlencoded". 218 | 219 | * `content` Byte array. The payload to post. 220 | 221 | #### Factory methods / Constructors 222 | 223 | 1. `HttpBody.json(ByteArray)` 224 | Creates an HTTP body with content type "application/json" and the given content. 225 | 226 | 2. `HttpBody.json(String)` 227 | Creates an HTTP body with content type "application/json". 228 | Content is the given string encoded as UTF-8. 229 | 230 | 3. `HttpBody.form(NameValuePairs)` 231 | Creates an HTTP body with content type "application/x-www-form-urlencoded". 232 | Content is the `urlEncoded` property encoded as UTF-8. 233 | 234 | As a convenience, implementations may add `form` overloads whose signatures match the `NameValuePair` constructors and create the NameValuePair internally before delegating to `form(NameValuePair)`. 235 | 236 | ### HttpResponse 237 | 238 | Immutable value object. 239 | Represents the result of an HTTP request. 240 | 241 | #### Properties 242 | 243 | These properties are public / visible to the user. 244 | 245 | * `statusCode` int. 246 | The HTTP status code returned by Couchbase Server. 247 | 248 | * `contentType` String (Optional) 249 | The value of the Content-Type header, if present in the response from Couchbase Server. 250 | 251 | * `content` Byte array. 252 | The content of the HTTP response. 253 | Possibly zero-length, but never null. 254 | 255 | If the implementation does not have access to the response headers, it may omit the `contentType` property. 256 | 257 | As a convenience, implementations may add a `contentAsString()` method that interprets the content as a UTF-8 String. 258 | 259 | ### CouchbaseHttpClient 260 | 261 | Instances must be thread-safe. 262 | 263 | #### Factory methods / Constructors 264 | 265 | Users do not create instances directly. 266 | `Cluster` has a new instance method `httpClient()` that returns a CouchbaseHttpClient. 267 | 268 | #### Other methods 269 | 270 | Instance methods of CouchbaseHttpClient correspond to the supported HTTP verbs. 271 | 272 | ```java 273 | class CouchbaseHttpClient { 274 | HttpResponse get( 275 | HttpTarget target, 276 | HttpPath path, 277 | NameValuePairs queryString, // optional 278 | List
headers // optional 279 | ); 280 | 281 | HttpResponse post( 282 | HttpTarget target, 283 | HttpPath path, 284 | HttpBody body, // optional 285 | List
headers // optional 286 | ); 287 | 288 | HttpResponse put( 289 | HttpTarget target, 290 | HttpPath path, 291 | HttpBody body, // optional 292 | List
headers // optional 293 | ); 294 | 295 | HttpResponse delete( 296 | HttpTarget target, 297 | HttpPath path, 298 | List
headers // optional 299 | ); 300 | } 301 | ``` 302 | 303 | A `Header` is just a name and a value, both strings. 304 | These are additional headers the client will include in the HTTP request. 305 | The documentation for this option should make it clear the user is not responsible for setting the "Content-Type" header. 306 | 307 | In addition to the parameters explicitly mentioned above, every method also has the optional parameters common to all SDK 3 management requests. 308 | The set of common parameters may vary by implementation, but typically includes "timeout", "retry strategy", and "parent span". 309 | 310 | If the implementation uses the notion of idempotency to control whether a request may be retried, GET requests are considered idempotent, and all other requests are non-idempotent. 311 | 312 | ### Implementation notes 313 | 314 | If the user provides a path that does not start with a slash, an implementation should prepend a slash to make it valid. 315 | 316 | If the user provides a path that contains the query string delimiter (`?`), an implementation should use `&` instead of `?` as the delimiter when appending the `queryString` value to the path. 317 | This allows the user to completely opt out of the query string parameter by encoding the query string as part of the path. 318 | 319 | For example, the following should be valid: 320 | 321 | ``` 322 | val response = httpClient.get( 323 | target = HttpTarget.manager(), 324 | path = HttpPath.of("foo/bar?color={}", red), 325 | queryString = NameValuePairs.of("magicWord" to "xyzzy"), 326 | ) 327 | ``` 328 | 329 | and should result in a path + query string of: 330 | 331 | /foo/bar?color=red&magicWord=xyzzy 332 | 333 | #### Asynchronous requests 334 | 335 | An implementation may provide multiple flavors of CouchbaseHttpClient to support blocking vs. asynchronous APIs, following the same patterns used elsewhere in the SDK 3 APIs. 336 | 337 | 338 | ## Changelog 339 | * Jan 20, 2022 - Revision #1 (by David Nault) 340 | * Initial Draft 341 | * Jan 21, 2022 - Revision #2 (by David Nault) 342 | * Added `HttpTarget.backup()` 343 | * Added `HttpResponse.contentAsString()` 344 | * Jan 24, 2022 - Revision #3 (by David Nault) 345 | * Replaced "CommonOptions" with a note that all HTTP client methods have the same optional parameters common to all SDK 3 management requests. 346 | * Added a note that GET requests should be considered idempotent, and all others are non-idempotent. 347 | * Added `HttpPath` for representing paths. An implementation may still use a String instead. 348 | * Feb 9, 2022 - Revision #4 (by David Nault) 349 | * Removed "views" HTTP target because views are on the way out. 350 | * Removed NodeIdentifier because it was under-specified and specific to the Java SDK. 351 | * Removed the "nodeIdentifier" and "bucketName" properties from HttpTarget. 352 | * Added "headers" option to all HTTP client methods, for specifying custom HTTP request headers. 353 | * Clarified that the client should automatically set the "Content-Type" header based on the HttpBody type. 354 | 355 | 356 | ## Signoff 357 | 358 | | Language | Team Member | Signoff Date | Revision | 359 | |--------------|--------------------|--------------|----------| 360 | | C | Sergey Avseyev | | | 361 | | Go | Charles Dixon | | | 362 | | Java | Michael Nitchinger | | | 363 | | .NET | Jeffry Morris | | | 364 | | Node.js | Brett Lawson | | | 365 | | PHP | Sergey Avseyev | | | 366 | | Python | Jared Casey | | | 367 | | Ruby | Sergey Avseyev | | | 368 | | Scala | Graham Pople | | | 369 | -------------------------------------------------------------------------------- /rfc/0074-config-profiles.md: -------------------------------------------------------------------------------- 1 | # RFC-0074 Configuration Profiles 2 | 3 | ## Meta 4 | 5 | * RFC Name: Configuration Profiles 6 | * RFC ID: 0074-config-profiles 7 | * Start Date: 2022-08-24 8 | * Owner: Michael Reiche 9 | * Current Status: Draft 10 | 11 | ## Summary 12 | 13 | Specification for Configuration Profiles with configuration properties having different values than defaults. 14 | From https://issues.couchbase.com/browse/CBD-5045 15 | 16 | ## Motivation 17 | 18 | The Default configuration properties are set following the Configuration specified in SDK 3 Foundation. https://github.com/couchbaselabs/sdk-rfcs/blob/master/rfc/0059-sdk3-foundation.md 19 | * Configuration property values different from the defaults would be more appropriate for some environments. 20 | * It is difficult to determine the values of the configuration properties when troubleshooting. 21 | 22 | 23 | ## Limitations 24 | 25 | The Configuration Profiles may property values may be an improvement upon the defaults for some cases. 26 | They are not intended to be the optimimum values for all cases. 27 | 28 | ## Requirements 29 | 30 | We want to introduce a feature that allows our SDKs to be quickly configured for some specific use-cases. The first profile we will create will be for "wan-development" environments and will change the default timeouts to work in high latency environments. However, the solution should be reusable to allow us to introduce additional profiles in the future (e.g., lambda/elixir may benefit from certain config defaults). 31 | The solution must allow config values to be overwritten by values that are associated with a config profile. Some SDKs may be more flexible than others, but the bare minimum implementation must support the ability to overwrite the config values when a config profile is applied. 32 | Also, it should be possible to enable logging of the values of the configuration properties that are being used by the SDK. 33 | 34 | ### Backwards compatibility 35 | The existing mechanism and configuration property values will remain unchanged. 36 | 37 | ### New Configuration Profiles 38 | * Sets of configuration property values will be defined by Configuration Profiles and referenced by a string name. 39 | * Not all properties need to be specified in a profile. 40 | * Properties that are specified in the configuration profile will overwrite properties in the environment object producing an environment object with the result. 41 | * Properties that are not specified will be copied as-is. 42 | 43 | 44 | ### New Configuration Profile Mechanism 45 | * A new Configurations Profile mechanism will allow new/different Configuration Profiles to be applied over existing configuration property values resulting in a new set of configuration property values (referred to as ClusterEnvironment in some SDKs). 46 | * Subsequent Configuration Profiles can be applied to the new set of configuration property values. 47 | * Configuration property values can be modified individually at any point before or after application of a Configuration Profile. 48 | 49 | 50 | ## Reference Design 51 | 52 | This section offers recommendations for implementing. 53 | Implementers should adhere to the reference design where it makes sense, and diverge where the idioms of a specific language or SDK provide a more natural solution. 54 | 55 | ### Generic Behavior 56 | 57 | ``` 58 | // Use the existing API to construct the environment (or environment builder depending on SDK) 59 | ClusterEnvironment originalEnv = …; 60 | 61 | // Optionally modify the resulting environment (or builder) with existing API 62 | originalEnv.setXXXX(...) 63 | // Optionally apply a configuration profile to the original environment (or environment builder) 64 | ClusterEnvironment newEnv = applyProfile(originalEnv, "development"); 65 | 66 | // Optionally modify the resulting environment (or builder) with existing API 67 | newEnv.setXXXX(...) 68 | 69 | // Use the environment as needed 70 | Cluster cluster = Cluster.connect( … newEnv … ); 71 | ``` 72 | 73 | ### Implementation Suggestions 74 | #### Java (uses a builder) 75 | 76 | ```java 77 | ClusterEnvironment clusterEnv = ClusterEnvironment.builder() 78 | .ioConfig(iocfg -> iocfg.numKvConnections(MY_NUM_KV_CONNECTIONS)) ← modify ioconfig 79 | .applyProfile("development") ← properties defined in "development" will overwrite 80 | .ioConfig(iocfg -> iocfg.maxHttpConnections(MY_MAX_HTTP_CONNECTIONS)) ← modify ioconfig 81 | .build(); 82 | ``` 83 | 84 | #### SDKs that use a struct 85 | 86 | ``` 87 | ClusterEnvironment originalEnv = … 88 | newOptions.ioConfig.numKvConnections = MY_NUM_KV_CONNECTIONS; 89 | ClusterEnvironment newEnv = applyProfile(originalEnv, 'development') 90 | newOptions.ioConfig.maxHttpConnections = MY_MAX_HTTP_CONNECTIONS 91 | Cluster cluster = cluster.connect( … someEnv …) 92 | ``` 93 | 94 | **Note:** it won't be possible to implement profiles by representing the profile with the same data structure as the environment (or builder) and merging the profile onto the existing one - unless there is a way to represent the properties as not being assigned a value (for java, some of the configuration values are primitives (i.e. int) and would need to use a special value to mean unassigned (such as -1)). Otherwise every property of the profile would appear to be assigned, and it would completely overwrite the initial configuration. 95 | 96 | ## Logging 97 | The configuration properties used by the SDK when connecting to the server should be logged to facilitate trouble-shooting. This does not need to be in a standard format. The Java SDK already does this as shown in the following sample: 98 | 99 | ``` 100 | INFO - {"timestamp":"2022-08-17 04:15:57.143","severity":"INFO","thread":"cb-events","class":"com.couchbase.core", 101 | "crId":"","context":{"svcId":"NRTOVIGO","svcBld":"1","svcNm":"ordnrt_vigomtdata_v1_task"}, 102 | "msg":"[com.couchbase.core][CoreCreatedEvent] {"clientVersion":"3.3.0","clientGitHash":"${buildNumber}", 103 | "coreVersion":"2.3.0","coreGitHash":"${buildNumber}", 104 | "userAgent":"couchbase-java/3.3.0 (Linux 4.14.232-177.418.amzn2.x86_64 amd64; OpenJDK 64-Bit Server VM 13.0.6+5-Debian-1)", 105 | "maxNumRequestsInRetry":32768,"ioEnvironment":{"nativeIoEnabled":false,"eventLoopThreadCount":8, 106 | "eventLoopGroups":["NioEventLoopGroup"]},"ioConfig":{"captureTraffic":[],"mutationTokensEnabled":true... 107 | ``` 108 | 109 | ## Profiles 110 | Configurable Properties - from SDK 3 Foundation RFC https://github.com/couchbaselabs/sdk-rfcs/blob/master/rfc/0059-sdk3-foundation.md. Note that only properties that are changed in the profile are included in this list (timeouts). 111 | 112 | | name | type | default (SDK3 Foundation)| wan-development | 113 | |---------------------------|----------------|--------------------------|-----------------| 114 | | KVConnectTimeout | Duration | 10s | 20s | 115 | | KVTimeout | Duration | 2.5s | 20s | 116 | | KVDurableTimeout | Duration | 10s | 20s | 117 | | ViewTimeout | Duration | 75s | 120s | 118 | | QueryTimeout | Duration | 75s | 120s | 119 | | AnalyticsTimeout | Duration | 75s | 120s | 120 | | SearchTimeout | Duration | 75s | 120s | 121 | | ManagementTimeout | Duration | 75s | 120s | 122 | 123 | ## Future Extensions 124 | Possibly maintain profiles in an external source (classpath resource, file etc) so that the values can be modified without changing code, or that new profiles can be created. 125 | For spring-boot applications, there is already a number of mechanisms to externalize properties which an application could leverage if so inclined - https://docs.spring.io/spring-boot/docs/3.0.0-M4/reference/html/features.html#features.external-config ] 126 | 127 | 128 | ## Changelog 129 | * Sep 19, 2022 - Revision #2 (by Michael Nitschinger) 130 | * Formatting Polish, Signoff 131 | * Aug 24, 2022 - Revision #1 (by Michael Reiche) 132 | * Version Draft 133 | 134 | ## Signoff 135 | 136 | | Language | Team Member | Signoff Date | Revision | 137 | |--------------|--------------------|--------------|----------| 138 | | C | Sergey Avseyev | 2022-09-02 | #1 | 139 | | Connectors | David Nault | 2022-09-01 | #1 | 140 | | Go | Charles Dixon | | | 141 | | Java | Michael Nitchinger | 2022-09-19 | #2 | 142 | | .NET | Jeffry Morris | 2022-09-08 | #1 | 143 | | Node.js | Brett Lawson | | | 144 | | PHP | Sergey Avseyev | 2022-09-02 | #1 | 145 | | Python | Jared Casey | 2022-09-02 | #1 | 146 | | Ruby | Sergey Avseyev | 2022-09-02 | #1 | 147 | | Scala | Graham Pople | 2022-09-08 | #1 | 148 | 149 | 150 | 151 | -------------------------------------------------------------------------------- /rfc/0076-subdoc-replica-reads.md: -------------------------------------------------------------------------------- 1 | # Meta 2 | 3 | | Field | Value | 4 | |----------------|----------------------------| 5 | | RFC Name | Sub-Document Replica Reads | 6 | | RFC ID | 76 | 7 | | Start Date | 2023-06-20 | 8 | | Owner | Charles Dixon | 9 | | Current Status | DRAFT | 10 | | Revision | #1 | 11 | 12 | # Summary 13 | [MB-23162](https://issues.couchbase.com/browse/MB-23162) adds support for performing subdocument read operations on replica servers. 14 | Replica read operations are performed against replica servers with a new ReplicaRead protocol flag. 15 | it also includes a new, informational, HELLO flag to be negotiated. 16 | 17 | This RFC specifies the SDK work needed to support this feature. 18 | All SDKs are required to implement this feature. 19 | 20 | # Motivation 21 | SDKs require a way to expose this new ReplicaRead feature in a way that is idiomatic to the SDK API. 22 | 23 | # Changes 24 | 25 | 1. Add two new functions to the `Collection` interface - `LookupInAnyReplica` and `LookupInAllReplicas`. 26 | 2. Add a new `LookupInReplicaResult` result type. 27 | 3. Add two new options types - `LookupInAnyReplicaOptions` and `LookupInAllReplicasOptions`. 28 | 4. Support the new REPLICA_READ (0x1c) HELLO flag. 29 | 5. When the above functions are used send `LookupIn` requests to replica nodes specifying the REPLICA_READ protocol flag (0x20). 30 | 31 | ## LookupInAllReplicas 32 | 33 | Gets a stream of document data from the server using LookupIn, leveraging both the active and all available replicas; the caller can choose which replica they wish to take and cancel the stream at any time. 34 | This MUST fetch from both the active and all replicas due to failover situations that can arise. 35 | In addition, the SDK MUST broadcast the requests simultaneously to ensure the application receives data as quickly as possible. 36 | 37 | ``` 38 | Stream LookupInAllReplicas(string id, LookupInSpec[] specs, [LookupInAllReplicasOptions]) 39 | ``` 40 | 41 | Parameters 42 | 43 | - Required 44 | 45 | - Id: string - the primary key. 46 | - LookupInSpec - an array of fetch operations, see [LookupIn](./0053-sdk3-crud.md#LookupIn) 47 | 48 | - Optional 49 | - Timeout or timeoutMillis (int/duration) - the time allowed for the operation to be terminated. This is controlled by the client. 50 | - Serializer - a custom serializer for converting the memcached JSON data to a native type. 51 | - ParentSpan - the parent span for any top level spans created during this operation. 52 | - RetryStrategy - the strategy to apply for retries. 53 | 54 | Returns 55 | 56 | A stream of replicas as a ILookupInReplicaResult object 57 | 58 | Throws 59 | 60 | - Documented 61 | 62 | - RequestTimeoutException (#1) 63 | - CouchbaseException (#0) 64 | - FeatureNotAvailableException (#15) 65 | 66 | - Undocumented 67 | - InvalidArgumentException (#3) 68 | 69 | ## LookupInAnyReplica 70 | 71 | Gets a document for a given id, leveraging both the active and all available replicas. 72 | This method follows the same semantics of LookupInAllReplicas (including the fetch from ACTIVE), but returns the first response as opposed to returning all responses. 73 | 74 | Signature 75 | 76 | ``` 77 | ILookupInReplicaResult LookupInAnyReplica(string id, LookupInSpec[] specs, [LookupInAnyReplicaOptions]) 78 | ``` 79 | 80 | Parameters 81 | 82 | - Required 83 | 84 | - Id: string - the primary key. 85 | - LookupInSpec - an array of fetch operations, see [LookupIn](./0053-sdk3-crud.md#LookupIn) 86 | 87 | - Optional 88 | 89 | - Timeout or timeoutMillis (int/duration) - the time allowed for the operation to be terminated. This is controlled by the client. 90 | - Serializer - a custom serializer for converting the memcached JSON data to a native type. 91 | - ParentSpan - the parent span for any top level spans created during this operation. 92 | - RetryStrategy - the strategy to apply for retries. 93 | 94 | Returns 95 | 96 | - A ILookupInReplicaResult wrapping a ILookupInResult object. 97 | 98 | Throws 99 | 100 | NOTE: The following exceptions are thrown at the "top" level. The errors that are thrown on a per-field basis are covered in [LookupIn](./0053-sdk3-crud.md#LookupIn). 101 | 102 | - Throws 103 | - Documented 104 | - DocumentUnretrievableException (#125) 105 | - The situation where we got responses from all of them (most likely: key not found) but none of them were SUCCESS so we ended up not returning anything (empty stream from lookupin all replicas). 106 | - RequestTimeoutException (#1) 107 | - CouchbaseException (#0) 108 | - FeatureNotAvailableException (#15) 109 | - Undocumented 110 | - InvalidArgumentException (#3) 111 | 112 | ## ILookupInReplicaResult 113 | 114 | The ILookupInReplicaResult interface is the return type for Sub-Document read operations from replicas. 115 | 116 | ```csharp 117 | public interface ILookupInReplicaResult : ILookupInResult { 118 | T ContentAs(int lookupIndex); 119 | Boolean Exists(int lookupIndex); 120 | Boolean IsReplica(); 121 | } 122 | ``` 123 | 124 | See [LookupIn](./0053-sdk3-crud.md#LookupIn) for information on `Exists` and `ContentAs`. 125 | 126 | - IsReplica 127 | - Returns true if the read was from a replica node. 128 | 129 | ## BucketCapabilities Check 130 | The `bucketCapabilities` field within the bucket config object must contain `subdoc.ReplicaRead` in order to use the `ReadReplica` document flag. 131 | 132 | It will only contain this field when ns_server has determined that all nodes have been upgraded to 7.5 or above. 133 | 134 | ## HELLO 135 | SDKs must implement and send a new HELLO flag with the code 0x1c. 136 | This HELLO flag is informative and does not actually enable/disable the feature, there is no reason to expose a way to disable this flag. 137 | 138 | Only Couchbase Server 7.5 and above will negotiate it. 139 | However, it can be negotiated by a 7.5 memcached node, when the cluster overall is not fully upgraded to 7.5. 140 | 141 | Hence, the reliable capability check is the “bucketCapabilities” one. 142 | At the point this is returned, all memcached nodes should be upgraded to 7.5 or above. 143 | This renders the HELLO check largely unnecessary (the HELLO doesn’t activate anything in memcached either, it is purely informational) but can be useful in debugging. 144 | 145 | ## Sending the request 146 | At the point just before sending the request, if the “bucketCapabilities” check fails (see above), then the SDK should raise a FeatureNotAvailableException. 147 | 148 | The check is done on each request, to support clusters being online-upgraded to 7.5. 149 | 150 | To perform ReplicaReads the SDK sends a normal LookupIn request to the relevant replica node and sets the top-level document flag to enable flag 0x20. 151 | Of course, this should be OR-ed with other such flags. 152 | 153 | ## Observability updates 154 | 155 | ### New top level span/metric names 156 | 157 | | Operation type | Identifier | 158 | |---------------------|------------------------| 159 | | LookupInAnyReplica | lookup_in_any_replica | 160 | | LookupInAllReplicas | lookup_in_all_replicas | 161 | 162 | Each of these operations must also include N inner `lookup_in` spans per replica/active read. 163 | 164 | # Changelog 165 | * June 21 2023 - Revision #1 166 | * Initial Draft 167 | 168 | # Signoff 169 | 170 | | Language | Team Member | Signoff Date | Revision | 171 | |-------------|----------------|--------------|----------| 172 | | .NET | Jeffry Morris | 2023-07-25 | #1 | 173 | | C/C++ | Sergey Avseyev | 2023-06-27 | #1 | 174 | | Go | Charles Dixon | 2023-05-21 | #1 | 175 | | Java/Kotlin | David Nault | 2023-07-06 | #1 | 176 | | Node.js | Jared Casey | 2023-07-19 | #1 | 177 | | PHP | Sergey Avseyev | 2023-06-27 | #1 | 178 | | Python | Jared Casey | 2023-07-19 | #1 | 179 | | Ruby | Sergey Avseyev | 2023-06-27 | #1 | 180 | | Scala | Graham Pople | 2023-07-04 | #1 | 181 | 182 | # Reference 183 | [KV patchset](https://review.couchbase.org/c/kv_engine/+/186113) 184 | -------------------------------------------------------------------------------- /rfc/figures/0058-aggregate-exception.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/couchbaselabs/sdk-rfcs/1ae1204667fb63e77371f75481de3d9457a89ee9/rfc/figures/0058-aggregate-exception.png -------------------------------------------------------------------------------- /rfc/figures/0058-general-design.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/couchbaselabs/sdk-rfcs/1ae1204667fb63e77371f75481de3d9457a89ee9/rfc/figures/0058-general-design.png -------------------------------------------------------------------------------- /rfc/figures/rfc30-fig1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/couchbaselabs/sdk-rfcs/1ae1204667fb63e77371f75481de3d9457a89ee9/rfc/figures/rfc30-fig1.png -------------------------------------------------------------------------------- /rfc/figures/rfc32-fig1.PNG: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/couchbaselabs/sdk-rfcs/1ae1204667fb63e77371f75481de3d9457a89ee9/rfc/figures/rfc32-fig1.PNG -------------------------------------------------------------------------------- /rfc/figures/rfc35-fig1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/couchbaselabs/sdk-rfcs/1ae1204667fb63e77371f75481de3d9457a89ee9/rfc/figures/rfc35-fig1.png -------------------------------------------------------------------------------- /rfc/figures/rfc35-fig2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/couchbaselabs/sdk-rfcs/1ae1204667fb63e77371f75481de3d9457a89ee9/rfc/figures/rfc35-fig2.png -------------------------------------------------------------------------------- /rfc/figures/rfc36-fig1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/couchbaselabs/sdk-rfcs/1ae1204667fb63e77371f75481de3d9457a89ee9/rfc/figures/rfc36-fig1.png -------------------------------------------------------------------------------- /rfc/figures/rfc55-uml1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/couchbaselabs/sdk-rfcs/1ae1204667fb63e77371f75481de3d9457a89ee9/rfc/figures/rfc55-uml1.png --------------------------------------------------------------------------------