├── .gitignore ├── text ├── EdgeDB-Execute.png ├── EdgeDB-Protocol-v0.png ├── EdgeDB-Protocol-v1.png ├── EdgeDB-TLS-Benchmark.png ├── 1016-extension-control.rst ├── 1019-index-fallback.rst ├── 1017-abstract-index.rst ├── 1020-triggers.rst ├── 0002-edgedb-release-process.rst ├── 1026-net-module.rst ├── 1010-global-vars.rst ├── 1018-extending-indexes.rst ├── 1022-freetypes.rst ├── 1002-optionality-qualifier.rst ├── 0001-rfc-process.rst ├── 1021-rewrites.rst ├── 1011-object-level-security.rst ├── 1012-range-types.rst ├── 1043-ext-auth-otc.rst ├── 1001-edgedb-server-control.rst ├── 1023-splats.rst ├── 1013-datetime-arithmetic.rst ├── 1005-edgedb-project.rst └── 1003-cli-naming.rst ├── .editorconfig ├── LICENSE-MIT ├── README.md ├── update_index.py └── LICENSE-APACHE /.gitignore: -------------------------------------------------------------------------------- 1 | /.vscode 2 | /.helix 3 | *~ 4 | -------------------------------------------------------------------------------- /text/EdgeDB-Execute.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/geldata/rfcs/HEAD/text/EdgeDB-Execute.png -------------------------------------------------------------------------------- /text/EdgeDB-Protocol-v0.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/geldata/rfcs/HEAD/text/EdgeDB-Protocol-v0.png -------------------------------------------------------------------------------- /text/EdgeDB-Protocol-v1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/geldata/rfcs/HEAD/text/EdgeDB-Protocol-v1.png -------------------------------------------------------------------------------- /text/EdgeDB-TLS-Benchmark.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/geldata/rfcs/HEAD/text/EdgeDB-TLS-Benchmark.png -------------------------------------------------------------------------------- /.editorconfig: -------------------------------------------------------------------------------- 1 | root = true 2 | 3 | [*] 4 | end_of_line = lf 5 | trim_trailing_whitespace = true 6 | insert_final_newline = true 7 | 8 | [text/*.rst] 9 | charset = "utf-8" 10 | indent_size = 4 11 | indent_style = space 12 | -------------------------------------------------------------------------------- /LICENSE-MIT: -------------------------------------------------------------------------------- 1 | Permission is hereby granted, free of charge, to any person obtaining a copy 2 | of this software and associated documentation files (the "Software"), to deal 3 | in the Software without restriction, including without limitation the rights 4 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 5 | copies of the Software, and to permit persons to whom the Software is 6 | furnished to do so, subject to the following conditions: 7 | 8 | The above copyright notice and this permission notice shall be included in all 9 | copies or substantial portions of the Software. 10 | 11 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 12 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 13 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 14 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 15 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 16 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 17 | SOFTWARE. 18 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # EdgeDB RFCs 2 | 3 | The purpose of this repository is to track *substantial* changes to EdgeDB. 4 | To read more on when and how to submit an RFC document, see RFC 1. 5 | 6 | ## List of RFCs 7 | [//]: # "NOTE: This section is auto-generated with update_index.py" 8 | 9 | ### Current Drafts 10 | * [RFC 1016 - Enabling and disabling extensions](./text/1016-extension-control.rst) 11 | * [RFC 1017 - Abstract index](./text/1017-abstract-index.rst) 12 | * [RFC 1018 - Extending indexes](./text/1018-extending-indexes.rst) 13 | * [RFC 1019 - Index fallback](./text/1019-index-fallback.rst) 14 | * [RFC 1022 - Typing free objects & simplifying SDL syntax](./text/1022-freetypes.rst) 15 | * [RFC 1026 - EdgeDB Networking Module](./text/1026-net-module.rst) 16 | * [RFC 1027 - Simplifying path resolution](./text/1027-no-factoring.rst) 17 | * [RFC 1029 - Role Based Access Control](./text/1029-rbac.rst) 18 | 19 | ### Accepted 20 | * [RFC 1 - The RFC Process](./text/0001-rfc-process.rst) (active) 21 | * [RFC 2 - The EdgeDB Release Process](./text/0002-edgedb-release-process.rst) (active) 22 | * [RFC 1000 - Migrations](./text/1000-migrations.rst) (accepted) 23 | * [RFC 1001 - CLI for installation and control of local EdgeDB server](./text/1001-edgedb-server-control.rst) (accepted) 24 | * [RFC 1002 - Optionality qualifier in properties and links](./text/1002-optionality-qualifier.rst) (final) 25 | * [RFC 1004 - Robust Client API](./text/1004-transactions-api.rst) (accepted) 26 | * [RFC 1005 - CLI and conventions for local projects using EdgeDB](./text/1005-edgedb-project.rst) (accepted) 27 | * [RFC 1006 - Simplified CLI Design](./text/1006-simplified-cli.rst) (accepted) 28 | * [RFC 1007 - Protocol v1](./text/1007-protocol-v1.rst) (accepted) 29 | * [RFC 1008 - TLS and ALPN Support](./text/1008-tls-and-alpn.rst) (accepted) 30 | * [RFC 1009 - GROUP](./text/1009-group.rst) (final) 31 | * [RFC 1010 - Global variables](./text/1010-global-vars.rst) (accepted) 32 | * [RFC 1011 - Object-Level Security](./text/1011-object-level-security.rst) (accepted) 33 | * [RFC 1012 - Range types](./text/1012-range-types.rst) (accepted) 34 | * [RFC 1013 - Date/time arithmetic](./text/1013-datetime-arithmetic.rst) (accepted) 35 | * [RFC 1015 - Full text search](./text/1015-full-text-search.rst) (accepted) 36 | * [RFC 1020 - Triggers](./text/1020-triggers.rst) (accepted) 37 | * [RFC 1021 - Mutation rewrites](./text/1021-rewrites.rst) (accepted) 38 | * [RFC 1023 - Adding Splats Syntax](./text/1023-splats.rst) (accepted) 39 | * [RFC 1025 - Database Branches](./text/1025-branches.rst) (accepted) 40 | * [RFC 1028 - CLI hooks](./text/1028-cli-hooks.rst) (accepted) 41 | 42 | ## License 43 | 44 | Licensed under either of 45 | 46 | * Apache License, Version 2.0, 47 | ([LICENSE-APACHE](./LICENSE-APACHE) or 48 | http://www.apache.org/licenses/LICENSE-2.0) 49 | * MIT license ([LICENSE-MIT](./LICENSE-MIT) or 50 | http://opensource.org/licenses/MIT) 51 | 52 | at your option. 53 | -------------------------------------------------------------------------------- /text/1016-extension-control.rst: -------------------------------------------------------------------------------- 1 | :: 2 | 3 | Status: Draft 4 | Type: Feature 5 | Created: 2022-10-07 6 | Authors: Victor Petrovykh 7 | 8 | =========================================== 9 | RFC 1016: Enabling and disabling extensions 10 | =========================================== 11 | 12 | This RFC proposes a generic mechanism for enabling and disabling extensions. 13 | 14 | 15 | Motivation 16 | ========== 17 | 18 | Extensions are part of the schema and affect functionality of the database, so 19 | adding or removing them can have wide-ranging effects on the applications 20 | using the database, not the least of which are the migrations necessary to 21 | make this change. The ability to load an extension, but keep it disabled when 22 | not needed is often desirable in development or testing environments. Loading 23 | the extension would register all the necessary schema changes, thus making 24 | sure that the enabled and disabled extension states are compatible with the 25 | same migration history. The only difference would be extension functionality, 26 | which could allow better flexibility when testing parts of the system that 27 | don't require the extension in question. 28 | 29 | 30 | Specification 31 | ============= 32 | 33 | Loaded extensions always export all their definitions, so that any symbol 34 | used in them is valid in EdgeQL or DDL regardless of whether the given 35 | extension is actually enabled or not. Disabled extensions may either produce 36 | errors when used or even silently perform an alternative [fallback] action. 37 | 38 | We want to introduce a special command to switch an extension on or off:: 39 | 40 | configure extension [enable | disable]; 41 | 42 | Potentially this might be under the umbrella of ``configure current 43 | database``. 44 | 45 | The configuration value could be reflected via a multi link in 46 | ``cfg::Config``:: 47 | 48 | type cfg::DatabaseConfig extending cfg::AbstractConfig { 49 | multi link extensions -> cfg::Extension; 50 | } 51 | 52 | type cfg::Extension extending cfg::ConfigObject { 53 | required property name -> str; 54 | required property is_enabled -> bool { 55 | default := true; 56 | } 57 | required property is_togglable -> bool { 58 | default := false; 59 | readonly := true; 60 | } 61 | } 62 | 63 | The ``is_togglable`` property indicates whether an extension can be toggled on 64 | and off. If it's ``false``, then attemting to toggle the extension state is an 65 | error. This value can only be set at the time of creating an extension and is 66 | immutable after that. 67 | 68 | The ``is_enabled`` property indicates whether a given extension is enabled or 69 | not and can potentially be changed by the special ``configure extension`` 70 | command as long as the extension is togglable. 71 | 72 | 73 | Backwards Compatibility 74 | ======================= 75 | 76 | There should not be any backwards compatibility issues. 77 | 78 | 79 | Security Implications 80 | ===================== 81 | 82 | Extensions potentially expose some data to a third-party service. Thus even 83 | after an extension is disabled this data may persist elsewhere even if it's no 84 | longer accessible in EdgeDB. 85 | 86 | 87 | Rejected Alternative Ideas 88 | ========================== 89 | 90 | We considered having arbitrary custom config values affecting whether an 91 | extension is enabled or disabled. In particular this was intended to provide 92 | more granular enabling/disabling for a given extension. It was rejected 93 | because the command to enable or disable an extension would then be basically 94 | a ``configure instance set`` or similar. Generally most ``configure`` commands 95 | don't have as far-reaching consequences as enabling an extension could. 96 | Extensions may require things like re-indexing the entire database or even 97 | reflect a large portion of it to some external service. These operations are 98 | significantly more constly than typical ``configure`` tweaks. 99 | 100 | We rejected ``alter extension set is_enabled := [true | false];`` 101 | as it appears to be a schema changing command, whereas we want to emphasize 102 | that it doesn't touch the schema. 103 | 104 | We rejected showing the extension status in the ``schema::Extension`` because 105 | this status is a special configuration rather than part of the schema. 106 | 107 | We rejected ideas of using one property ``is_enabled`` to indicate both 108 | whether an extension is enabled or disabled as well as whether it is togglable 109 | in principle. To do so we'd have to user the two ``bool`` values as well as 110 | ``{}`` to encode the extension state. This seems a bit more convoluted. -------------------------------------------------------------------------------- /text/1019-index-fallback.rst: -------------------------------------------------------------------------------- 1 | :: 2 | 3 | Status: Draft 4 | Type: Feature 5 | Created: 2022-12-15 6 | Authors: Victor Petrovykh 7 | 8 | ======================== 9 | RFC 1019: Index fallback 10 | ======================== 11 | 12 | This RFC proposes a mechanism for defining fallback options for indexes in 13 | case of extensions that provide index functionality are disabled. 14 | 15 | 16 | Motivation 17 | ========== 18 | 19 | We need a way to provide more than one type of index in EdgeDB. These 20 | alternative ``index`` definitions could be built-in or come from extensions. 21 | Different index-providing extensions may be enabled/disabled depending on 22 | whether EdgeDB is currently in development or production environemnt. We want 23 | to provide a fallback mechanism that allows to keep the same schema for these 24 | different environments. 25 | 26 | 27 | Specification 28 | ============= 29 | 30 | We want to provide a way of grouping several abstract indexes into one. The 31 | purpose of this is to provide fallback index implementations when the 32 | preferred one is not available. Such may be the case of full-text search 33 | indexes as the FTS functionality may be provided by a variety of backends. A 34 | particular FTS extension may be disabled in development environment, but 35 | enabled in production. This mechanism for providing fallback for disabled 36 | indexes allows to gracefully handle the differences in indexes without having 37 | to change large portions of the schema. The syntax for this mechanism is 38 | similar to defining a new abstract index and does not directly allow extending 39 | existing indexes. The point is to define a new abstract index in terms of 40 | existing implementations:: 41 | 42 | abstract index [ ( named only [] [, ...] ) ] 43 | using [ ( [ := ] [, ...] ) ] 44 | [?? ...] 45 | "{" 46 | [ ] 47 | [ ... ] 48 | "}" ; 49 | 50 | where argspec is: 51 | 52 | : [ ] [ = ] 53 | 54 | typequal is: 55 | 56 | [ { set of | optional } ] 57 | 58 | The ``using`` clause specifies the underlying implementation. It is possible 59 | to supply a ``??``-separated list of index implementations. The ``??`` 60 | operator here indicates that it's a kind of fallback mechanism. In that case 61 | the first implementation in the coalesced sequence that is enabled will 62 | actually be used, while other implementations will be ignored. Note that this 63 | ``??`` is not really an operator in this case, but a syntactical feature. 64 | 65 | The fallback abstract index may declare its own parameters which can them be 66 | referenced in the ``using`` clause by providing them as values for the 67 | underlying indexes. This is necessary because the underlying indexes may not 68 | have the same parameter names or might have parameters of the same name, but 69 | incompatible types and so a unifying layer is needed. 70 | 71 | This type of fallback abstract index can be referenced in a concrete index 72 | like any other abstract index. 73 | 74 | 75 | Examples 76 | -------- 77 | 78 | Here's an example of defining an FTS fallback index:: 79 | 80 | using extension fts; 81 | using extension es; 82 | 83 | module default { 84 | # defining an abstract FTS index that relies on 85 | # elasticsearch by default and falls back to 86 | # Postgres FTS implementation. 87 | abstract index my_fts 88 | using es::fts_index ?? fts::pg_index; 89 | 90 | type Post { 91 | required property title -> str; 92 | required property body -> str; 93 | 94 | index my_index on (.title); 95 | index my_index on (.body); 96 | } 97 | } 98 | 99 | 100 | Backwards Compatibility 101 | ======================= 102 | 103 | There should not be any backwards compatibility issues. 104 | 105 | 106 | Security Implications 107 | ===================== 108 | 109 | There are no security implications. 110 | 111 | 112 | Rejected Alternative Ideas 113 | ========================== 114 | 115 | We rejected the idea of using a comma-separated list for fallback options in 116 | favour of using a ``??`` to indicate more clearly the intent. The ``??`` 117 | operator already has the semantics of "use the first non-empty value", which 118 | is similar to what we mean when we provide a fallback list. The point is to 119 | have a syntax that signals to the user more clearly the purpose of this 120 | abstract index. This would also be consistent with some possible future where 121 | returning functions is possible and then similar constructs might even be part 122 | of other EdgeQL expressions that pick a function to be used. -------------------------------------------------------------------------------- /text/1017-abstract-index.rst: -------------------------------------------------------------------------------- 1 | :: 2 | 3 | Status: Draft 4 | Type: Feature 5 | Created: 2022-10-07 6 | Authors: Victor Petrovykh 7 | 8 | ======================== 9 | RFC 1017: Abstract index 10 | ======================== 11 | 12 | This RFC proposes formalizing a concept of an abstract index. This would be 13 | the mechanism by which Postgres indexes can be exposed and made available to 14 | the users. 15 | 16 | 17 | Motivation 18 | ========== 19 | 20 | We need a way to provide more than one type of index in EdgeDB. These 21 | alternative ``index`` definitions could be built-in or come from extensions. 22 | These ``abstract index`` definitions can then be referenced by name in the 23 | user schema when defining concrete indexes. 24 | 25 | 26 | Specification 27 | ============= 28 | 29 | The ``abstract index`` definition can follow a similar pattern to ``abstract 30 | constraint`` definition. It must appear as a top-level module element and 31 | serves as a definition of the index name and parameters. In its current state 32 | it would only be possible to define ``abstract index`` during the compilation 33 | of the standard library. Meanwhile the users can only refer to the existing 34 | abstract indexes when defining their concrete indexes in the user schema. 35 | 36 | The new abstract indexes can be defined as follows:: 37 | 38 | abstract index [ on ( , ... ), ... ] 39 | "{" 40 | [ ] 41 | [ using SQL ] 42 | [ ... ] 43 | "}" ; 44 | 45 | The index can be defined as targetting specific operations by providing a list 46 | of ``func-name`` identifies denoting function names or operators. This is 47 | meant to be similar to the operator classes in Postgres and can be used to 48 | infer the types for which the index is valid. A valid indexable type must be 49 | compatible with all of the provided operations. As a rule the type of the 50 | first parameter of the named function or operator determines the valid types 51 | for an index. Potentially several operator groupings may be provided. Each 52 | grouping then determines their own valid indexable types. Omitting the ``on`` 53 | clause implies that the index is generic enough to be valid for ``anytype``. 54 | Note that the underlying Postgres implementation actually takes care of the 55 | operator classes for a given index and typically this EdgeQL clause would not 56 | be reflected into the underlying implementation, but rather it is supposed to 57 | be consistent with it. 58 | 59 | Some of the details of the SQL index definition can be provided in the 60 | ``index-body``. Essentially this is what the ``CREATE INDEX`` command is going 61 | to be based on. The SQL body here is a template that uses ``__col__`` 62 | placeholder where the indexed column/expression would noramlly be, other than 63 | that it is intended to contain the rest of the ``CREATE INDEX`` SQL command. 64 | 65 | A concrete index can be defined inside any type in the following manner:: 66 | 67 | type "{" 68 | index [] on ( ) 69 | [ except ( ) ] 70 | [ "{" "}" ] ; 71 | "}" 72 | 73 | We add an optional ``name`` that can be provided here when some non-default 74 | index is used. This ``name`` refers to one of the existing abstract indexes. 75 | The rest of the concrete index definition stays as is. 76 | 77 | 78 | Exposed postgres indexes 79 | ------------------------ 80 | 81 | Postgres has the following built-in indexes: ``hash``, ``btree``, ``gin``, 82 | ``gist``, ``spgist``, and ``brin``. Some of these apply universally to all 83 | types and others are more specialized. We want to expose these indexes as part 84 | of a new ``pg`` module. Here's a breakdown of the exposed indexes: 85 | 86 | 1) ``pg::hash`` - applicable to ``anytype``, pretty much useful for exact 87 | matches, but not much else. It's generally smaller that more elaborate 88 | indexes. 89 | 2) ``pg::btree`` - applicable to ``anytype`` and the default index used (when 90 | no name is provided). 91 | 3) ``pg::gin`` - applicable to arrays, JSON, and tsvector, however we don't 92 | seem to use "contains" or "overlap" operators for anything other than 93 | arrays. It might later be useful for FTS functionallity (because of 94 | tsvector) once that gets introduced. 95 | 4) ``pg::gist`` - applicable to geometry types, tsquery, tsvector, and ranges. 96 | So currently we should expose it for ``range``. Eventually we may 97 | want to expose it for FTS functionality as well targetitng ``str``. 98 | 5) ``pg::spgist`` - applicable to geometry types, text, and ranges. So we 99 | should expose it for ``str`` and ``range``. 100 | 6) ``pg::brin`` - applicable to many types, but not all so ``anytype`` is not 101 | a good idea. Instead it is applicable to ``anyreal``, ``bytes``, ``str``, 102 | ``uuid``, ``datetime``, ``duration``, ``cal::local_datetime``, 103 | ``cal::local_date``, ``cal::local_time``, ``cal::date_duration``, 104 | ``cal::relative_duration``, ``range``. 105 | 106 | 107 | Examples 108 | -------- 109 | 110 | Here's an example of defining an default index and a ``spgist`` index:: 111 | 112 | module default { 113 | type Post { 114 | required property title -> str; 115 | required property body -> str; 116 | 117 | index on (.title); 118 | index pg::spgist on (.body); 119 | } 120 | } 121 | 122 | For the purposes of the standard library the following definitions might 123 | apply:: 124 | 125 | # this one is applicable to anytype, so no `on` clause is needed 126 | CREATE ABSTRACT INDEX pg::hash { 127 | USING SQL $$ 128 | USING hash ((__col__)) 129 | $$; 130 | } 131 | 132 | # this is applicable to ranges 133 | CREATE ABSTRACT INDEX pg::gist on (std::overlaps) { 134 | USING SQL $$ 135 | USING gist ((__col__)) 136 | $$; 137 | } 138 | 139 | 140 | Backwards Compatibility 141 | ======================= 142 | 143 | There should not be any backwards compatibility issues. 144 | 145 | 146 | Security Implications 147 | ===================== 148 | 149 | There are no security implications. 150 | 151 | 152 | Rejected Alternative Ideas 153 | ========================== 154 | 155 | We rejected the idea that the abstract indexes should specify the valid types 156 | directly. This seems to contradict both how Postgres defines index 157 | applicability and the notion that indexes should work for any type that 158 | supports certain operations, since those operations are basically relevant for 159 | how the indexes are built. -------------------------------------------------------------------------------- /update_index.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3.8 2 | 3 | from __future__ import annotations 4 | from typing import * # NoQA 5 | 6 | import dataclasses 7 | import os 8 | import pathlib 9 | import re 10 | 11 | 12 | Status = Literal[ 13 | "active", 14 | "inactive", 15 | "final", 16 | "accepted", 17 | "rejected", 18 | "deferred", 19 | "draft", 20 | ] 21 | 22 | 23 | CURRENT_DIR: Final = pathlib.Path(__file__).parent 24 | RFC_FILENAME = re.compile(r"^(?P\d\d\d\d)-(?P[a-z0-9_-]+)\.rst$") 25 | RFC_TITLE = re.compile(r"^RFC (?P\d+)(: | - )(?P.+)$") 26 | HEADER_FIELD = re.compile(r"^ +(?P<name>[^:]+): (?P<value>.*)$") 27 | HEADING = re.compile(r"^[=-~]+$") 28 | AUTOGENERATED = "NOTE: This section is auto-generated with update_index.py" 29 | ACCEPTED_STATUSES: Set[Status] = {"active", "inactive", "final", "accepted"} 30 | STATUSES: Set[Status] = ACCEPTED_STATUSES | { 31 | "rejected", "deferred", "draft", "withdrawn"} 32 | 33 | 34 | @dataclasses.dataclass 35 | class RFC: 36 | number: int 37 | title: str 38 | status: Status 39 | filename: str 40 | 41 | @classmethod 42 | def from_file(cls, path: pathlib.Path) -> RFC: 43 | number = 0 44 | title = "unknown" 45 | status = "unknown" 46 | filename = path.name 47 | expected_length = 0 48 | 49 | states = [ 50 | "before_preamble", 51 | "in_empty_line", 52 | "in_preamble", 53 | "before_title", 54 | "after_title", 55 | ] 56 | with path.open() as file: 57 | for line in file: 58 | line = line.rstrip() 59 | if states[0] == "before_preamble": 60 | if line == "::": 61 | states.pop(0) 62 | elif states[0] == "in_empty_line": 63 | if not line: 64 | states.pop(0) 65 | elif states[0] == "in_preamble": 66 | if m := HEADER_FIELD.match(line): 67 | if m.group("name") == "Status": 68 | status = m.group("value").lower() 69 | else: 70 | states.pop(0) 71 | elif states[0] == "before_title": 72 | if m := RFC_TITLE.match(line): 73 | number = int(m.group("number")) 74 | title = m.group("title") 75 | expected_length = len(line) 76 | elif HEADING.match(line): 77 | if expected_length == 0: 78 | # pre-heading overline 79 | continue 80 | 81 | if len(line) == expected_length: 82 | states.pop(0) 83 | else: 84 | # previous RFC title match, if any, was invalid 85 | number = 0 86 | title = "unknown" 87 | 88 | elif states[0] == "after_title": 89 | break 90 | 91 | if number == 0 or title == "unknown": 92 | raise LookupError( 93 | f"Invalid RFC document {path}: {number=} {title=}" 94 | ) 95 | 96 | if status in STATUSES: 97 | status = cast(Status, status) 98 | else: 99 | raise LookupError(f"Invalid RFC document {path}: {status=}") 100 | 101 | return RFC( 102 | number=number, title=title, status=status, filename=filename 103 | ) 104 | 105 | 106 | def list_rfcs(directory: pathlib.Path) -> Dict[Status, List[RFC]]: 107 | result: Dict[Status, List[RFC]] = {} 108 | for file in os.listdir(directory): 109 | if not RFC_FILENAME.match(file): 110 | continue 111 | 112 | rfc = RFC.from_file(directory / file) 113 | status = rfc.status 114 | if status in ACCEPTED_STATUSES: 115 | status = "accepted" 116 | result.setdefault(status, []).append(rfc) 117 | for rfcs in result.values(): 118 | rfcs.sort(key=lambda elem: elem.number) 119 | 120 | return result 121 | 122 | 123 | def patch_readme(readme: pathlib.Path, rfcs: Dict[Status, List[RFC]]) -> None: 124 | lines: List[str] = [] 125 | states = ["before_rfc_list", "in_rfc_list", "after_rfc_list"] 126 | with readme.open() as original: 127 | for line in original: 128 | if states[0] == "before_rfc_list": 129 | lines.append(line) 130 | if line.rstrip() == "## List of RFCs": 131 | states.pop(0) 132 | elif states[0] == "in_rfc_list": 133 | if line.startswith("## "): 134 | states.pop(0) 135 | lines.append(f'[//]: # "{AUTOGENERATED}"\n\n') 136 | if "draft" in rfcs: 137 | lines.append("### Current Drafts\n") 138 | for r in rfcs["draft"]: 139 | lines.append( 140 | f"* [RFC {r.number} - {r.title}]" 141 | f"(./text/{r.filename})\n" 142 | ) 143 | lines.append("\n") 144 | if "accepted" in rfcs: 145 | lines.append("### Accepted\n") 146 | for r in rfcs["accepted"]: 147 | lines.append( 148 | f"* [RFC {r.number} - {r.title}]" 149 | f"(./text/{r.filename}) ({r.status})\n" 150 | ) 151 | lines.append("\n") 152 | if "deferred" in rfcs: 153 | lines.append("### Deferred\n") 154 | for r in rfcs["deferred"]: 155 | lines.append( 156 | f"* [RFC {r.number} - {r.title}]" 157 | f"(./text/{r.filename})\n" 158 | ) 159 | lines.append("\n") 160 | if "rejected" in rfcs: 161 | lines.append("### Rejected\n") 162 | for r in rfcs["rejected"]: 163 | lines.append( 164 | f"* [RFC {r.number} - {r.title}]" 165 | f"(./text/{r.filename})\n" 166 | ) 167 | lines.append("\n") 168 | lines.append(line) 169 | elif states[0] == "after_rfc_list": 170 | lines.append(line) 171 | 172 | with readme.open("w") as new: 173 | for line in lines: 174 | new.write(line) 175 | 176 | 177 | def main() -> None: 178 | rfcs = list_rfcs(CURRENT_DIR / "text") 179 | patch_readme(CURRENT_DIR / "README.md", rfcs) 180 | 181 | 182 | if __name__ == "__main__": 183 | main() 184 | -------------------------------------------------------------------------------- /text/1020-triggers.rst: -------------------------------------------------------------------------------- 1 | :: 2 | 3 | Status: Accepted 4 | Type: Feature 5 | Created: 2022-12-15 6 | Authors: Michel J. Sullivan <sully@msully.net> 7 | 8 | ================== 9 | RFC 1020: Triggers 10 | ================== 11 | 12 | This RFC proposes a mechanism for triggers. 13 | 14 | Some text stolen from an earlier RFC by Elvis. 15 | 16 | Motivation 17 | ========== 18 | 19 | Triggers are useful for things such as generating audit logs and 20 | generating external events. 21 | 22 | Note 23 | ==== 24 | 25 | This version of the trigger RFC reflects a split between "triggers" 26 | and "mutation rewrites", which will come in another RFC. 27 | 28 | Specification 29 | ============= 30 | 31 | Synopsis:: 32 | 33 | type <type-name> "{" 34 | trigger <name> 35 | {after | after commit of} 36 | { insert | update | delete } [, ... ] 37 | for { each | all } 38 | do <expr> 39 | 40 | "}" 41 | 42 | (N.B: ``AFTER COMMIT OF`` will probably not be implemented in the first 43 | pass.) 44 | 45 | ``AFTER`` triggers are run after constraint and access policies have 46 | been checked. For ``insert`` and ``update``, the variable ``__new__`` 47 | refers to the new value of the modified object, and for ``update`` and 48 | ``delete``, the variable ``__old__`` refers to the old value. 49 | 50 | ``AFTER COMMIT OF`` triggers will run "shortly" after the containing 51 | transaction is committed. It has ``__new__`` variables, like ``AFTER`` 52 | triggers. The state of the database is indeterminate, since an 53 | potentially many queries may have executed after the triggering 54 | even. (This means that the ``__new__`` objects may no longer exist!) 55 | 56 | If ``FOR EACH`` is specified the ``<expr>`` is evaluated for 57 | *each* of the modified objects and the variables ``_old__`` and 58 | ``__new__`` refer to individual objects. If ``FOR ALL`` is specified, 59 | then the trigger is run once per query where more than one applicable 60 | object was modified, and ``__old__`` and ``__new__`` refer to the 61 | entire set of old and new modified objects. 62 | 63 | A failure in an ``AFTER`` trigger causes the query to fail. 64 | 65 | Trigger activation for ``AFTER`` triggers is chained but not recursive: 66 | triggers can activate other triggers, but a trigger cycle is an error. 67 | 68 | Triggers are fired off in "stages": first all triggers that were triggered 69 | by the initial query fire, then any triggers that were triggered in the 70 | first stage, and so on. 71 | 72 | It is an error if a trigger would need to fire in multiple stages. 73 | 74 | The overall state of the database during the trigger is from *after* 75 | the query. That is, if you access data not via ``__new__`` and 76 | ``__old__``, the data is from after the query. 77 | 78 | More generally, if there are multiple stages of triggers fired, the 79 | database state is from after the previous stage. 80 | 81 | Backwards Compatibility 82 | ======================= 83 | 84 | There should not be any backwards compatibility issues. 85 | 86 | Rationale 87 | ========= 88 | 89 | We support both ``FOR EACH`` and ``FOR ALL`` because for all is more 90 | powerful while for each is simpler and more ergonomic in common 91 | cases. (Especially for ``update``, when otherwise you would often need 92 | to join ``_old__`` and ``__new__``. 93 | 94 | 95 | Implementation considerations 96 | ============================= 97 | 98 | ``AFTER`` triggers will be implemented through a query rewrite mechanism 99 | in which we inline the triggers bodies into the query and run them on 100 | the modified objects. 101 | 102 | ``AFTER COMMIT OF`` is the most complex action to implement, because 103 | it requires two parts: 1) the query rewrite part that schedules the 104 | action by writing the affected data and the action expression into the 105 | job table, and 2) the runner task that pops the expression and input 106 | data from the job table and performs the action. 107 | 108 | As a result, we will probably skip it for the initial 3.0 implementation. 109 | 110 | 111 | Security Implications 112 | ===================== 113 | 114 | This whole feature is at least "security adjacent", since it may be 115 | used to implement things like audit logs and generalized constraints. 116 | 117 | The main direct security question is in the interaction with access 118 | policies. Triggers *can* see the object they are operating on, even 119 | if it would otherwise not be visible due to access policies. (Much 120 | like how a freshly inserted object is still returned from the query 121 | even if it would not be visible.) 122 | 123 | Otherwise, access policies *are* applied during trigger evaluation. 124 | 125 | 126 | Rejected Alternative Ideas 127 | ========================== 128 | 129 | Generalized policy based query rewrite 130 | -------------------------------------- 131 | A `previous version of this RFC 132 | <https://github.com/edgedb/rfcs/pull/50>`_ written by Elvis, combined 133 | triggers and access policies into one generic mechanims. We decided 134 | this was likely to be too complex, and that they should be split. 135 | 136 | ON SELECT 137 | --------- 138 | 139 | That RFC also called for being able to do ``DO AFTER`` actions on 140 | ``select``, giving as an example:: 141 | 142 | abstract object type LoggedAccess { 143 | policy on select 144 | do after (for obj in __result__ union ( 145 | insert AccessLog { 146 | object := obj, 147 | user_id := global user_id, 148 | time := datetime_current(), 149 | } 150 | )) 151 | } 152 | 153 | I have left this out of the proposal for now because it seems hard 154 | semantically to say what objects get ``select``ed. Presumbably 155 | ``select Obj filter .id = ...`` should only fire the policy once, 156 | but how about ``with W := (select Obj), select W filter .id = ...``. 157 | 158 | 159 | Having a BEFORE/AFTER split 160 | --------------------------- 161 | 162 | Another `previous version of this RFC 163 | <https://github.com/edgedb/rfcs/pull/70>`_, contained 164 | a distinction between ``BEFORE`` triggers and ``AFTER`` triggers. 165 | 166 | ``BEFORE`` triggers would be inlined into the query, would have access 167 | to ``__old__``, and could *not* modify objects that had already been 168 | modified. 169 | 170 | ``AFTER`` triggers would be run in a pipelined query, would not have 171 | access to ``__old__`` (and as such could not be used for ``DELETE``), 172 | and *could* modify objects that had already been modified in the 173 | original query. 174 | 175 | (The ``AFTER`` triggers proposed in this RFC are actually mostly the 176 | ``BEFORE`` triggers from the old RFC.) 177 | 178 | This approach is plausible, and makes it possible to do most of the 179 | things we wanted, but the differing limitations between the two is 180 | complicated and likely confusing. 181 | 182 | 183 | Implement using postgres triggers 184 | --------------------------------- 185 | 186 | There is a critical semantic problem in using postgres triggers, which 187 | is that postgres triggers only have access to the old state of the 188 | database and to the new rows. But in edgedb, the state of an object 189 | might be spread across multiple tables (for multi pointers), and so 190 | the full state of a new or updated object may be invisible to a 191 | postgres trigger. 192 | -------------------------------------------------------------------------------- /text/0002-edgedb-release-process.rst: -------------------------------------------------------------------------------- 1 | :: 2 | 3 | Status: Active 4 | Type: Process 5 | Created: 2020-10-08 6 | Authors: Łukasz Langa <lukasz@edgedb.com> 7 | RFC PR: `edgedb/rfcs#0022 <https://github.com/edgedb/rfcs/pull/22/>`_ 8 | 9 | ==================================== 10 | RFC 0002: The EdgeDB Release Process 11 | ==================================== 12 | 13 | 14 | Overview 15 | ======== 16 | 17 | This document describes how we're releasing a new version of EdgeDB. 18 | This includes both the CLI tool and the server, as well as announcements. 19 | 20 | 21 | Before we begin 22 | =============== 23 | 24 | In general, the order of releasing various components should be the 25 | following: bindings, CLI, server. 26 | 27 | This means that we want to make sure that the nightly bindings work 28 | with the old and the new server. Similarly, we want to make sure that 29 | the nightly CLI works with the old and new server. 30 | 31 | 1. Check with the team on the ``#engineering`` channel in Slack 32 | whether there are any release blockers. 33 | 34 | 2. Check if the latest nightly build of the CLI and of the server was 35 | green. 36 | 37 | 3. Perform the pre-release testing to ensure that the users have a 38 | smooth upgrading experience. 39 | 40 | 41 | Pre-release testing 42 | =================== 43 | 44 | The "bindings" in the checklist below refer to all of our currently 45 | supported bindings: 46 | 47 | - `Python <https://github.com/edgedb/edgedb-python>`_ bindings. 48 | - `JavaScript and TypeScript 49 | <https://github.com/edgedb/edgedb-js>`_ bindings. 50 | - `Go <https://github.com/edgedb/edgedb-go>`_ bindings. 51 | - `Deno <https://github.com/edgedb/edgedb-deno>`_ bindings. 52 | 53 | All of the binding tests need to be performed for each of our 54 | supported bindings. 55 | 56 | 1. Using an old version (stable release) of CLI and server, either 57 | initialize a new project or restore an existing one. 58 | 59 | 2. Update the bindings to the nightly version. 60 | 61 | 3. Try to connect and run some queries using the new bindings and an 62 | old (stable release) version of the server. 63 | 64 | 4. Update the CLI to the nightly version. 65 | 66 | 5. Try connecting to the old (stable release) server instances. Run 67 | queries. 68 | 69 | 6. Create a new project using the old server to run instances. Make 70 | sure the following sub-tasks are possible: 71 | 72 | - successfully initialize a new project 73 | - connect to it 74 | - run queries 75 | - perform at least one migration 76 | 77 | 7. Upgrade the server to the nightly build. 78 | 79 | 8. Try to connect to an existing (freshly upgraded) project via the 80 | CLI and run queries. 81 | 82 | 9. Try to connect and run some queries using the new bindings and the 83 | freshly upgraded project. 84 | 85 | 10. Try initializing a new project using the nightly server. Make sure 86 | the following sub-tasks are possible: 87 | 88 | - successfully initialize a new project 89 | - connect to it 90 | - run queries 91 | - perform at least one migration 92 | 93 | 11. Try to connect and run some queries using the new bindings and the 94 | freshly created project. 95 | 96 | 97 | Publishing the binaries 98 | ======================= 99 | 100 | CLI 101 | --- 102 | 103 | 1. Create a release branch if not exists (assuming you release 2.2.0): 104 | 105 | - ``git switch -c stable/2.2.x`` 106 | 107 | - update version if needed in ``Cargo.toml`` and run ``cargo check`` 108 | 109 | - commit (ideally with a name like "edgedb-cli 2.2.0") 110 | 111 | 2. Tag the commit: 112 | 113 | - ``git tag -s v2.2.0`` 114 | 115 | - for major releases the message can be something like:: 116 | 117 | v2.0.0 "Sagittarius" 118 | 119 | See changelog at 120 | https://github.com/edgedb/edgedb/blob/master/docs/changelog/2_x.rst 121 | 122 | for a minor/bugfix release no description is usually necessary 123 | 124 | - ``git push --atomic -u origin stable/2.2.x v2.2.0`` 125 | 126 | 3. Start the release flow by going to: 127 | 128 | - https://github.com/edgedb/edgedb-cli/actions/workflows/release.yml?query=workflow%3A%22Build%2C+Test%2C+and+Publish+a+Release%22 129 | - select the newly pushed tag from the dropdown and press "Run Workflow" 130 | 131 | 4. Update version in ``master`` branch to the next minor version: 132 | 133 | - ``git switch -c bump23`` 134 | 135 | - update ``Cargo.toml`` and run ``cargo check`` 136 | 137 | - commit (ideally with a name like "Bump main to 2.3.0-dev") 138 | 139 | - ``git push -u origin bump23`` 140 | 141 | - Create a PR from the newly created branch 142 | 143 | 144 | Server 145 | ------ 146 | 147 | .. note:: 148 | 149 | Only build the server after the CLI is published. 150 | 151 | 1. Create a release branch: 152 | 153 | - ``git switch -c releases/1.0a6`` 154 | 155 | 2. Tag the last commit: 156 | 157 | - ``git tag -s v1.0a6`` 158 | 159 | - the message can be something like:: 160 | 161 | v1.0a6 "Wolf 359" 162 | 163 | See changelog at 164 | https://github.com/edgedb/edgedb/blob/master/docs/changelog/1_0_a6.rst 165 | 166 | - ``git push --follow-tags`` 167 | 168 | 4. Start the release flow by going to: 169 | 170 | - https://github.com/edgedb/edgedb/actions/workflows/release.yml?query=workflow%3A%22Build+Test+and+Publish+a+Release%22 171 | - select the newly pushed release branch from the dropdown and press "Run Workflow" 172 | 173 | 5. When all is good, you can check out the tag locally, build EdgeDB 174 | and run ``edb gen-test-dumps`` to generate test dumps for the version 175 | you're releasing now. Commit them to **master**, not to the release 176 | branch, they're not needed there. 177 | 178 | 179 | External places to bump binaries at 180 | ----------------------------------- 181 | 182 | 1. Update tutorial.edgedb.com to run on the latest release. The package 183 | to update is edgedb-cloud/docker/embedded/, use the README there for 184 | update instructions. After uploading a new package to ECR, kick the 185 | Fargate job by running ``edbcloud fargate tutorial/t1 --force``. 186 | 187 | 2. Update Docker Hub. This should happen automatically during the server 188 | GitHub Action release build (debian-buster). 189 | 190 | 3. Update the DigitalOcean image. Instructions can be found in 191 | edgedb-deploy/digitalocean-1-click/README.md. 192 | 193 | 4. Update the Homebrew tap. 194 | 195 | The tap is auto-updating nightly. If you need to bump it faster, 196 | use the HTTP repository dispatch documented in the README of the 197 | tap. 198 | 199 | Alternatively, on an installed Homebrew the repository lives in 200 | ``/usr/local/Homebrew/Library/Taps/edgedb/homebrew-tap``. Go there 201 | and run ``./autoupdate.py``, commit and push the changes. 202 | 203 | 204 | 205 | 206 | Updating the Website 207 | ==================== 208 | 209 | The Downloads page 210 | ------------------ 211 | 212 | A number of places on the `Downloads <downloads_>`_ page refer to 213 | a particular version. In particular you want to update: 214 | 215 | * src/pages/download.jsx 216 | * src/pages/index.jsx 217 | * content/download/linux.centos.md 218 | * content/download/linux.ubuntu.* 219 | * content/download/linux.debian.* 220 | 221 | Announcement Blog Post 222 | ---------------------- 223 | 224 | Looking for a theme in the changelog is a good way to phrase the 225 | announcement blog post. Remember to give the post a fresh UUID. 226 | 227 | 228 | .. _downloads: https://edgedb.com/download 229 | -------------------------------------------------------------------------------- /text/1026-net-module.rst: -------------------------------------------------------------------------------- 1 | :: 2 | 3 | Status: Draft 4 | Type: Feature 5 | Created: 2024-08-14 6 | Authors: Scott Trinh <scott@edgedb.com> 7 | 8 | ================================== 9 | RFC 1026: EdgeDB Networking Module 10 | ================================== 11 | 12 | This RFC outlines the addition of a networking module to EdgeDB’s 13 | standard library, providing functionality for making HTTP requests and 14 | sending SMTP messages. 15 | 16 | Motivation 17 | ========== 18 | 19 | The EdgeDB Auth extension (ext::auth) currently sends emails directly to 20 | configured SMTP services at certain times in various authentication 21 | ceremonies and workflows: 22 | 23 | - Sign up with email (sends email verification email) 24 | - Password reset (sends password reset email) 25 | - Magic link sign up (sends magic link email) 26 | - Magic link sign in (sends magic link email) 27 | 28 | To allow full customization of emails and provide hooks into 29 | authentication events generally, we need to support sending webhook 30 | requests to the application directly. This would enable applications to 31 | trigger their own logic for sending fully customized and 32 | internationalized emails, SMS, log analytics, update usage billing 33 | records, etc. 34 | 35 | Specification 36 | ============= 37 | 38 | Module Name 39 | ----------- 40 | 41 | The new module will be named ``net``. 42 | 43 | Utility types 44 | ------------- 45 | 46 | Since requests are sent asynchronously, we need a way to track the status of the 47 | request. 48 | 49 | .. code-block:: edgeql 50 | 51 | scalar type net::RequestState extending std::enums<Pending, InProgress, Complete, Failed>; 52 | 53 | scalar type net::RequestFailureKind extending std::enums<NetworkError, Timeout>; 54 | 55 | ``net::RequestState`` represents the current asynchronous state of a request. 56 | The worker process will update the state of the request as it progresses through 57 | the various states. 58 | 59 | ``net::RequestFailureKind`` represents the types of failure that can occur when 60 | trying to send the request, not a failure response from the server. 61 | 62 | HTTP 63 | ---- 64 | 65 | This section outlines the types and functions of the ``net::http`` submodule. 66 | This submodule provides functionality for making HTTP requests. 67 | 68 | .. code-block:: edgeql 69 | 70 | abstract type net::http::Method extending std::enums<`GET`, POST, PUT, `DELETE`, PATCH, HEAD, OPTIONS>; 71 | 72 | type net::http::ScheduledRequest { 73 | required state: net::RequestState; 74 | required created_at: datetime; 75 | failure: tuple<kind: net::RequestFailureKind, message: str>; 76 | 77 | required url: str; 78 | required method: net::http::Method; 79 | required headers: array<tuple<name: str, value: str>>; 80 | required body: bytes; 81 | 82 | response: net::http::Response { 83 | constraint exclusive; 84 | }; 85 | }; 86 | 87 | type net::http::Response { 88 | required status: int16; 89 | required headers: array<tuple<name: str, value: str>>; 90 | body: bytes; 91 | }; 92 | 93 | function net::http::schedule_request( 94 | url: str, 95 | named only body: optional bytes, 96 | named only method: net::HttpMethod = net::HttpMethod::GET, 97 | named only headers: optional array<tuple<name: str, value: str>> 98 | ) -> net::http::ScheduledRequest; 99 | 100 | SMTP 101 | ---- 102 | 103 | This section outlines the types and functions of the ``net::smtp`` submodule. 104 | This submodule provides functionality for sending SMTP messages. 105 | 106 | .. code-block:: edgeql 107 | 108 | type net::smtp::ScheduledRequest { 109 | required state: net::RequestState; 110 | required created_at: datetime; 111 | failure: tuple<kind: net::RequestFailureKind, message: str>; 112 | 113 | required url: str; 114 | required from: multi str; 115 | required to: multi str; 116 | required subject: str; 117 | required text: optional str; 118 | required html: optional str; 119 | 120 | response: net::smtp::Response { 121 | constraint exclusive; 122 | }; 123 | }; 124 | 125 | type net::smtp::Response { 126 | required reply_code: int16; 127 | reply_message: str; 128 | }; 129 | 130 | function net::smtp::schedule_send( 131 | url: str, 132 | named only from: multi str, 133 | named only to: multi str, 134 | named only subject: str, 135 | named only text: optional str, 136 | named only html: optional str, 137 | ) -> net::smtp::ScheduledRequest; 138 | 139 | Implementation Details 140 | ---------------------- 141 | 142 | - Pending ``ScheduledRequest`` objects will be represent the queue of requests to be sent. 143 | - A Rust process will handle sending the requests. 144 | - Each protocol (HTTP, SMTP) will have its own pool of worker processes. 145 | - URLs will initially be represented as plain strings, with the possibility of adding type-checked URL support in the future. 146 | 147 | Examples 148 | ======== 149 | 150 | HTTP Request 151 | ------------ 152 | 153 | .. code:: edgeql 154 | 155 | with 156 | payload := '{"key": "value"}', 157 | request := ( 158 | select net::http::schedule_request( 159 | 'https://api.example.com/webhook', 160 | body := payload, 161 | method := net::http::Method::POST, 162 | headers := [("Content-Type", "application/json")], 163 | ) 164 | ) 165 | select request { 166 | id, 167 | state, 168 | created_at, 169 | url, 170 | }; 171 | 172 | SMTP Send 173 | --------- 174 | 175 | .. code:: edgeql 176 | 177 | with 178 | html_body := '<html><body><p>Hello, this is a test email.</p></body></html>', 179 | text_body := 'Hello, this is a test email.', 180 | request := ( 181 | select net::smtp::schedule_send( 182 | 'smtp://smtp.example.com:587', 183 | from := 'sender@example.com', 184 | to := {'recipient1@example.com', 'recipient2@example.com'}, 185 | subject := 'Test Email', 186 | html := html_body, 187 | text := text_body 188 | ) 189 | ) 190 | select request { 191 | id, 192 | state, 193 | created_at, 194 | url, 195 | }; 196 | 197 | Backwards Compatibility 198 | ======================= 199 | 200 | This RFC introduces new functionality and does not affect existing 201 | features. There are no backwards compatibility issues. 202 | 203 | Rejected Alternative Ideas 204 | ========================== 205 | 206 | 1. Using pg_net: While pg_net provides similar functionality, it was 207 | decided to implement our own solution for better control and 208 | integration with EdgeDB. This allows end users to more easily scale 209 | sending by scaling the EdgeDB server rather than scaling PostgreSQL. 210 | 2. Fully configurable queuing mechanism: For the initial implementation, 211 | a simple, built-in policy will be used instead of a fully 212 | configurable one to reduce complexity. 213 | 214 | Future Related Work 215 | =================== 216 | 217 | 1. Add support for more protocols (e.g., AMQP, ZeroMQ, SQS, FTP). 218 | 2. Implement fully type-checked URLs and standard library functions to 219 | assist in constructing correct URLs, and with quoting and 220 | concatenation. 221 | 3. Allow retrying through configuration at request creation time. 222 | 4. Integration with a future EdgeDB queuing module to gain a more 223 | sophisticated retry strategies, durability, and reliability. 224 | -------------------------------------------------------------------------------- /text/1010-global-vars.rst: -------------------------------------------------------------------------------- 1 | :: 2 | 3 | Status: Accepted 4 | Type: Feature 5 | Created: 2022-01-15 6 | Authors: Elvis Pranskevichus <elvis@edgedb.com> 7 | RFC PR: `edgedb/rfcs#0049 <https://github.com/edgedb/rfcs/pull/49>`_ 8 | 9 | ========================== 10 | RFC 1010: Global variables 11 | ========================== 12 | 13 | This RFC proposes adding a mechanism to declare and set global variables 14 | in a schema and set them on per-session basis. 15 | 16 | 17 | Motivation 18 | ========== 19 | 20 | The need for non-local data context frequently exists in programming, and the 21 | database is no exception. EdgeDB schemas offer a large variety of ways to 22 | express the data model with EdgeQL expressions, and the ability to parametrize 23 | these expressions with contextual per-session data will provide even more 24 | power. Consider the ubiquitous concept of authentication, where there is 25 | a notion of "current user" that is used to filter query results or make other 26 | decisions. Currently, such data must be passed as an explicit argument to 27 | queries and there is no way to pass such context to schema expressions at all. 28 | 29 | 30 | Specification 31 | ============= 32 | 33 | A global variable must be declared before it can be used in any expression. 34 | 35 | CREATE GLOBAL 36 | ------------- 37 | 38 | Define a new global variable. 39 | 40 | Required capabilities: DDL. 41 | 42 | Synopsis:: 43 | 44 | [ WITH <with-item> [, ...] ] 45 | CREATE [ REQUIRED ] [ MULTI | SINGLE] GLOBAL <name> -> <type> [ "{" 46 | <subcommand> ; [...] 47 | "}" ] ; 48 | 49 | # Computed global variable form: 50 | 51 | [ WITH <with-item> [, ...] ] 52 | CREATE [ REQUIRED ] GLOBAL <name> := <expr> ; 53 | 54 | # where <subcommand> is one of 55 | 56 | SET default := <value> 57 | RESET default 58 | RENAME TO <newname> 59 | USING <expr> 60 | RESET EXPRESSION 61 | CREATE ANNOTATION <annotation-name> := <value> 62 | ALTER ANNOTATION <annotation-name> := <value> 63 | DROP ANNOTATION <annotation-name> 64 | 65 | The ``<type>`` of non-computed globals must be a primitive type, i.e. it 66 | *must not* be an object type or a collection of object types. This is because 67 | object type variables will require some form of referential integrity checks 68 | like links do (i.e. the ``on target delete`` spec), which complicates the 69 | implementation considerably. Computed globals can have any type, and so it 70 | is possible to define an object type global in terms of its ``id``, e.g:: 71 | 72 | CREATE GLOBAL user_id -> uuid; 73 | CREATE GLOBAL user := (SELECT User FILTER .id = (GLOBAL user_id)); 74 | 75 | If a global is declared as ``REQUIRED``, it *must* provide a default value, or, 76 | if the global is computed, the expression must be non-optional. For example:: 77 | 78 | CREATE SCALAR TYPE Context EXTENDING enum<System, User>; 79 | CREATE REQUIRED GLOBAL context -> Context { 80 | SET default := Context.System; 81 | } 82 | 83 | # or 84 | 85 | CREATE REQUIRED GLOBAL user := assert_exists(SELECT User ...) 86 | 87 | Explicit cardinality is specified via the ``SINGLE`` or ``MULTI`` keyword, 88 | otherwise the cardinality is assumed to be ``SINGLE`` in non-computed globals, 89 | and is inferred from the expression in computed globals. 90 | 91 | 92 | ALTER GLOBAL 93 | ------------ 94 | 95 | Alter the definition of a global variable. 96 | 97 | Required capabilities: DDL. 98 | 99 | Synopsis:: 100 | 101 | [ WITH <with-item> [, ...] ] 102 | ALTER GLOBAL <name> 103 | "{" <subcommand>; [...] "}" ; 104 | 105 | # where <subcommand> is one of 106 | 107 | SET default := <value> 108 | RESET default 109 | RENAME TO <newname> 110 | USING <expr> 111 | RESET EXPRESSION 112 | SET REQUIRED 113 | DROP REQUIRED 114 | SET SINGLE 115 | SET MULTI 116 | CREATE ANNOTATION <annotation-name> := <value> 117 | ALTER ANNOTATION <annotation-name> := <value> 118 | DROP ANNOTATION <annotation-name> 119 | 120 | 121 | DROP GLOBAL 122 | ----------- 123 | 124 | Remove a global variable. 125 | 126 | Required capabilities: DDL. 127 | 128 | Synopsis:: 129 | 130 | [ WITH <with-item> [, ...] ] 131 | DROP GLOBAL <name> ; 132 | 133 | 134 | SET GLOBAL 135 | ---------- 136 | 137 | Set the value of a non-computed global variable *in the current session* 138 | by evaluating the given expression. 139 | 140 | Required capabilities: session config. 141 | 142 | Synopsis:: 143 | 144 | SET GLOBAL <name> := <expr> ; 145 | 146 | 147 | RESET GLOBAL 148 | ------------ 149 | 150 | Reset a non-computed global variable to its default value. 151 | 152 | Required capabilities: session config. 153 | 154 | Synopsis:: 155 | 156 | RESET GLOBAL <name> ; 157 | 158 | 159 | Referring to globals in queries 160 | =============================== 161 | 162 | The new ``GLOBAL <name>`` expression is used to refer to the value of the 163 | given global variable in queries. For example:: 164 | 165 | SELECT User FILTER .id = GLOBAL user_id 166 | 167 | The precedence of ``GLOBAL`` is similar to ``DETACHED`` and binds higher 168 | than most other operators, including the dot. 169 | 170 | References to globals are legal in regular EdgeQL queries (i.e. non-DDL) and 171 | in parts of schema definition that interpolate into EdgeQL queries: 172 | 173 | - defaults 174 | - expressions in computed properties and links 175 | - expression aliases 176 | 177 | 178 | Introspection 179 | ============= 180 | 181 | Globals can be introspected via a new entry in the introspection schema: 182 | ``schema::Global``. Likewise a ``DESCRIBE GLOBAL <foo>`` statement returns 183 | a textual DDL or SDL description of a global. 184 | 185 | 186 | Performance and scalability considerations 187 | ========================================== 188 | 189 | For scalability and HA reasons, the server must not maintain persistent session 190 | state, including the values of globals. This means that it is the 191 | responsibility of the _client_ to maintain this state and send it to the server 192 | with every query or script request in a context header. See 193 | `Protocol changes`_ below. 194 | 195 | Internally, references to non-computed globals are compiled as query arguments 196 | and are query cache friendly. 197 | 198 | 199 | Protocol changes 200 | ================ 201 | 202 | Regarding the protocol, there are two things to consider: client informing the 203 | server about the session state, and the server informing the client about 204 | the session state changes as a result of a ``SET/RESET GLOBAL`` statement. 205 | 206 | When sending a query or a script, a client _may_ specify the values of globals 207 | in the new "globals" header, encoded as a free object. 208 | 209 | When the server encounters a ``SET/RESET GLOBAL`` statement in a query or 210 | script message it *must* include the new values of the affected globals in 211 | the new "globals" header returned with the next ``CommandComplete`` response. 212 | 213 | 214 | Computed globals vs aliases 215 | =========================== 216 | 217 | While there is a lot of similarity between computed globals and aliases, 218 | they aren't the same and complement each other. Where an alias usually 219 | stands in place of a material set of objects (and this masquerades as an 220 | object type), a computed global stands in for a settable global. Additionally, 221 | the alias cardinality is usually ``multi``, whereas most globals would normally 222 | be ``single``. 223 | 224 | 225 | Backwards compatibility 226 | ======================= 227 | 228 | ``global`` is already a reserved keyword, no other new keywords are proposed 229 | in this RFC. 230 | 231 | Protocol changes require a protocol version bump. 232 | 233 | 234 | Security implications 235 | ===================== 236 | 237 | There are no security implications. 238 | -------------------------------------------------------------------------------- /text/1018-extending-indexes.rst: -------------------------------------------------------------------------------- 1 | :: 2 | 3 | Status: Draft 4 | Type: Feature 5 | Created: 2022-12-15 6 | Authors: Victor Petrovykh <victor@edgedb.com> 7 | 8 | =========================== 9 | RFC 1018: Extending indexes 10 | =========================== 11 | 12 | This RFC introduces a parametrized indexes and the possibility of extending 13 | indexes so that users may define their own abstract indexes with customized 14 | settings. 15 | 16 | 17 | Motivation 18 | ========== 19 | 20 | Some indexes are complex enought that they require tuning. We want to 21 | introduce index parameters defined in teh abstract indexes for this purpose. 22 | Additionally, it should be possible for users to define their own versions of 23 | existing abstract indexes to customize some of the parameter values. In 24 | particular this would allow slightly different flavours of the same basic 25 | index type to be used for different types or properties of the same type. 26 | 27 | 28 | Specification 29 | ============= 30 | 31 | The ``abstract index`` definition can follow a similar pattern to ``abstract 32 | constraint`` definition in regards to having parameters. 33 | 34 | One important distinction of the abstract index definitions is that new index 35 | parameters can only be introduced for a base index (i.e. an index that is not 36 | ``extending`` another index). Conversely any indexes that use ``extending`` 37 | clause cannot introduce any new parameters, instead they can provide specific 38 | values for parameters they inherit. This ensures that new index types 39 | can define any parameters they need, while making it possible for users to 40 | customize these generic indexes for specific purposes. 41 | 42 | The new indexes can be defined as follows:: 43 | 44 | abstract index <name> [ ( named only [<argspec>] [, ...] ) ] 45 | [ on ( <func-name>, ... ), ... ] 46 | "{" 47 | [ <annotation-declarations> ] 48 | [ using SQL <index-body> ] 49 | [ ... ] 50 | "}" ; 51 | 52 | where <argspec> is: 53 | 54 | <argname>: [ <typequal> ] <argtype> [ = <default-val> ] 55 | 56 | <typequal> is: 57 | 58 | [ { set of | optional } ] 59 | 60 | All the index paramters have to be ``named only`` so that they can be 61 | overridden individually when extending an abstract index. 62 | 63 | One way to customize indexes is by defining new abstract indexes extending 64 | some generic index and specifying concrete parameter values for them. In 65 | principle it would also be possible to combine several such abstract indexes 66 | defining different parameters as mixins for a single new abstract index. So 67 | extending provides a way of composition for different parameters combinations. 68 | 69 | The ``extending`` indexes definition:: 70 | 71 | abstract index <name> [ ( [<argname> := <const-expr>] [, ...] ) ] 72 | extending <index-name> [, ...] 73 | "{" 74 | [ annotation-declarations ] 75 | [ ... ] 76 | "}" ; 77 | 78 | The ``extending`` index can define specific constant values for any 79 | ``argname`` parameter that is inherited from the parent abstract indexes. This 80 | mechanism can be used to override the specific parameter values given by the 81 | parent indexes. Each of the parameter values tracks their inherited value 82 | independently of other parameters. This means that any given inheritance layer 83 | may potentially override a single parameter different from other layers and 84 | thus combine all these different parameter values. 85 | 86 | A concrete index definition may now reference these additional user-created abstract indexes just like any other abstract index. 87 | 88 | 89 | Composability 90 | ------------- 91 | 92 | One way of composing the indexes is by defining intermediate abstract indexes 93 | while setting the defaults for some of the parameters. This works similar to 94 | partial function closures in other languages and allows users to factor out 95 | some common index configuration. 96 | 97 | Each parameter should inherit the default value according to the same 98 | inheritance resolution strategy as is applied to types. Each parameter should 99 | track inherited default value independently from other index parameters. This 100 | allows greater flexibility for abstract index composition, while still 101 | preserving fairly intuitive expectations of defaults for the users. 102 | 103 | Sometimes the configuration cannot be neatly split between the parameters (in 104 | case we need to partially define an ``array`` or ``json``). For those cases 105 | the solution is to instead define constant valued aliases and compose them to 106 | form the desired index parameters of *concrete* indexes. We already track 107 | whether an expression is constant or not so that we can determine valid 108 | definitions of things like ``default``. This mechanism might need to be 109 | expanded a bit to handle some simple operators (like element access for arrays 110 | and json) in order to fully take advantage of this method of index parameter 111 | composition. 112 | 113 | 114 | Examples 115 | -------- 116 | 117 | Here's an example of a possible usage of abstract indexes to compose several 118 | FTS features as needed:: 119 | 120 | using extension es; 121 | 122 | module default { 123 | abstract index stopwords_mixin( 124 | filter := to_json('["english_stop"]') 125 | ) extending es::fts_index; 126 | 127 | abstract index lowercase_mixin( 128 | filter := to_json('["lowercase"]') 129 | ) extending es::fts_index; 130 | 131 | abstract index htmlfilter_mixin( 132 | char_filter := to_json('["html_strip"]') 133 | ) extending es::fts_index; 134 | 135 | # Composing the abstract indexes into what we will use. 136 | abstract index title_index extending htmlfilter_mixin, lowercase_mixin; 137 | abstract index body_index extending htmlfilter_mixin, stopwords_mixin; 138 | 139 | type Post { 140 | required property title -> str; 141 | required property body -> str; 142 | 143 | index title_index on (.title); 144 | index body_index on (.body); 145 | } 146 | } 147 | 148 | Example of using constants to compose configuration for FTS indexes:: 149 | 150 | using extension es; 151 | 152 | module default { 153 | alias eng_stop := to_json('["english_stop"]'); 154 | alias lowercase := to_json('["lowercase"]'); 155 | 156 | abstract index my_index( 157 | char_filter := to_json('["html_strip"]') 158 | ) extending es::fts_index; 159 | 160 | type Post { 161 | required property title -> str; 162 | required property body -> str; 163 | 164 | index my_index(filter := lowercase) on (.title); 165 | index my_index(filter := eng_stop ++ lowercase) on (.body); 166 | } 167 | } 168 | 169 | 170 | Backwards Compatibility 171 | ======================= 172 | 173 | There should not be any backwards compatibility issues. 174 | 175 | 176 | Security Implications 177 | ===================== 178 | 179 | There are no security implications. 180 | 181 | 182 | Rejected Alternative Ideas 183 | ========================== 184 | 185 | We rejected the idea that only parameters that aren't already given explicit 186 | values (via ``extending`` definition) can be specified here. It was intended 187 | to avoid accidentally breaking the general rules provided by the abstract 188 | indexes defining those parameters. In practice it could be far too restrictive 189 | and it would also change how *defaults* work. 190 | 191 | We rejected the idea of special syntax that would allow referring to inherited 192 | default values for parameters when defining the parameter expression. This was 193 | deemed to much of a cognitive burden by both introducing a new syntax and by 194 | having index parameter expressions follow slightly different rules than all 195 | other expressions. The idea was dropped in favour of inheriting default 196 | values. 197 | 198 | We rejected the idea of inheriting all parameters as a whole based on the 199 | index inheritance. The benefit was that of being a simpler inheritance 200 | mechanism, the drawback was that it made composition much harder. -------------------------------------------------------------------------------- /text/1022-freetypes.rst: -------------------------------------------------------------------------------- 1 | :: 2 | 3 | Status: Draft 4 | Type: Feature 5 | Created: 2023-01-23 6 | Authors: Yury Selivanov <yury@edgedb.com> 7 | 8 | 9 | ====================================================== 10 | RFC 1022: Typing free objects & simplifying SDL syntax 11 | ====================================================== 12 | 13 | Abstract 14 | ======== 15 | 16 | This RFC proposes enhancements to the EdgeDB type system allowing users to 17 | explicitly type free objects. Free object types can then be used to type 18 | function parameters or used to indicate types of query arguments. 19 | 20 | While discussing the necessary new syntax for typing free objects, this 21 | RFC proposes a set of adjustments to the current EdgeDB Schema Declaration 22 | Language syntax. 23 | 24 | 25 | Typing free objects 26 | =================== 27 | 28 | The proposed grammar is structured with the following requirements in mind: 29 | 30 | * The syntax must be "light" as it will appear frequently in EdgeQL 31 | queries inside cast (e.g. ``<type_expr>$foo``) constructs. 32 | 33 | * We want to automatically infer as many things as possible. 34 | 35 | * We want the syntax to play nice with the existing syntax for typing 36 | arrays and tuples. 37 | 38 | The grammar looks as follows:: 39 | 40 | type_name: 41 | # name of the type, such as "int64" or "default::User" 42 | 43 | type_expr: 44 | # type expressions valid in SDL, such as "A | B" or 45 | # "array<int64>" 46 | 47 | property_or_link: 48 | ["multi" | "single"] ["required" | "optional"] ["property" | "link"] \ 49 | name ":" type_name | type_expr | free_object_type 50 | 51 | free_object_type: 52 | "{" 53 | property_or_link 54 | [ "," property_or_link ] * 55 | "}" 56 | 57 | 58 | Key takeaways: 59 | 60 | * EdgeDB will infer whether a field is a property or a link, making 61 | "property" and "link" keywords optional. 62 | 63 | * Similar to SDL, all properties and links are *optional* and 64 | *single* by default. 65 | 66 | (Note that the syntax deviates from SDL: we use ``:`` instead of ``->`` and 67 | ``property`` / ``link`` modifier keywords are optional. To make the new free 68 | object types syntax compatible with SDL we also propose to update the SDL 69 | syntax in the below `Updates to SDL`_ section.) 70 | 71 | Examples: 72 | 73 | A free object as a query argument:: 74 | 75 | with 76 | data := <{a: int64, b: str}>$arg 77 | select 78 | data.a + <str>data.b; 79 | 80 | A more complex free object with nested data and JSON casts:: 81 | 82 | with 83 | data := <{ 84 | required pokemon: str, 85 | multi required locations: { 86 | required point: tuple<float64, float64>, 87 | street: str 88 | } 89 | }>json_array_unpack(<json>$arg) 90 | 91 | for item in data union ( 92 | insert Pokemon { 93 | name := item.pokemon, 94 | # ... 95 | } 96 | ) 97 | 98 | In user-defined functions:: 99 | 100 | function compute(pokemon: { 101 | required pokemon: str, 102 | multi required locations: { 103 | required point: tuple<float64, float64>, 104 | street: str 105 | } 106 | }) -> int64 using ( 107 | select # ... 108 | ) 109 | 110 | 111 | Updates to SDL 112 | ============== 113 | 114 | Summary: 115 | 116 | * Make the ``property`` and ``link`` keyword optional for all non-computed 117 | properties and links. The kind of a pointer can be inferred automatically 118 | in all situations. Users will be encouraged to explicitly specify the 119 | kind of a pointer for readability or enforcement purposes where necessary. 120 | 121 | * Replace the ``->`` symbol with ``:`` for type annotations of pointers. 122 | ``->`` will still be valid syntax, but we will update our documentation 123 | and output of introspection commands to use ``:``. 124 | 125 | * Keep the ``->`` symbol for function return type annotation (in other words 126 | we will not add support for using ``:`` in return type annotation). 127 | 128 | Motivation: 129 | 130 | * The SDL syntax of declaring object types and ad-hoc free object types 131 | has to look similar. But having ``->`` symbols inside EdgeQL's casts will 132 | negatively impact the readability of the language, e.g. compare 133 | ``select <{x: int64, y: int64}>`` vs ``select <{x -> int64, y -> int64}>``. 134 | 135 | * The history of the ``->`` syntax in SDL has its roots back in the time when 136 | SDL was a whitespace sensitive language. We could not use ``:`` because it 137 | used to denote a block, e.g.:: 138 | 139 | property foo -> int64: 140 | annotation title := 'the foo-est of foos' 141 | 142 | However, the modern SDL isn't whitespace sensitive and we can now use ``:``. 143 | 144 | * There's no strict requirement for forcing users to use ``property`` and 145 | ``link`` modifiers. We can always infer the kind of the pointer by its 146 | specified or inferred type. The intent mainly was to improve the readability 147 | of the schema with explicit modifiers. However, now it is clear that (a) we 148 | need a shorter syntax for inline object typing, and (b) our users wish 149 | that our SDL would have a more concise syntax. By making ``property`` and 150 | ``link`` optional our SDL syntax becomes closer to popular languages 151 | like TypeScript, ultimately improving the DX and the first impression of our 152 | SDL. 153 | 154 | Example: 155 | 156 | +--------------------------------------+--------------------------------------+ 157 | | Current syntax: | Proposed syntax: | 158 | +======================================+======================================+ 159 | |:: | :: | 160 | | | | 161 | | abstract type Content { | abstract type Content { | 162 | | required property title -> str; | required title: str; | 163 | | multi link actors -> Person { | multi actors: Person { | 164 | | property character_name -> str; | character_name: str; | 165 | | }; | }; | 166 | | } | } | 167 | | | | 168 | | type Movie extending Content { | type Movie extending Content { | 169 | | property release_year -> int32; | release_year: int32; | 170 | | } | } | 171 | | | | 172 | | type Show extending Content { | type Show extending Content { | 173 | | property num_seasons := | property num_seasons := | 174 | | count(.<show[is Season]); | count(.<show[is Season]); | 175 | | } | } | 176 | +--------------------------------------+--------------------------------------+ 177 | 178 | 179 | Implementation timeline 180 | ======================= 181 | 182 | Typing free objects sounds like an EdgeDB 4.0 feature, especially given that 183 | we enable tuples to be used as query arguments in 3.0. 184 | 185 | However, the proposed SDL changes (making ``property`` and ``link`` optional 186 | and replacing ``->`` with ``:``) are better to implement as early as possible, 187 | in other words in 3.0. Happily the proposed change should be backwards 188 | compatible. 189 | 190 | 191 | Future enhancements 192 | =================== 193 | 194 | Simplifying tuple/array type syntax 195 | ----------------------------------- 196 | 197 | We can potentially simplify tuple and array types syntax as follows: 198 | 199 | ===================================== ========================================= 200 | Current Future 201 | ===================================== ========================================= 202 | ``select <tuple<int64, str>>$0`` ``select <(int64, str)>$0`` 203 | ``select <array<tuple<int64>>>$0`` ``select <[(int64,)]>$0`` 204 | ``property foo -> tuple<int64, str>`` ``foo: (int64, str)`` 205 | ===================================== ========================================= 206 | 207 | 208 | Rejected ideas 209 | ============== 210 | 211 | Make "required" the default pointer type in free objects 212 | -------------------------------------------------------- 213 | 214 | This would draw nice parallels between tuples and free objects in EdgeQL 215 | queries, but would make it exceptionally hard to comprehend types in SDL files. 216 | 217 | 218 | Make property/link modifiers optional for computed fields 219 | --------------------------------------------------------- 220 | 221 | Computed pointers and fields like ``default`` both use the ``:=`` syntax. 222 | Making ``property`` and ``link`` modifiers optional for computeds would make 223 | them look like fields. 224 | -------------------------------------------------------------------------------- /text/1002-optionality-qualifier.rst: -------------------------------------------------------------------------------- 1 | :: 2 | 3 | Status: Final 4 | Type: Guideline 5 | Created: 2020-06-17 6 | RFC PR: `edgedb/rfcs#0001 <https://github.com/edgedb/rfcs/pull/1>`_ 7 | 8 | ======================================================= 9 | RFC 1002: Optionality qualifier in properties and links 10 | ======================================================= 11 | 12 | This RFC describes the reasons why DDL and SDL declarations for properties 13 | and links assume optionality by default. 14 | 15 | 16 | Motivation 17 | ========== 18 | 19 | The data definition language, and more importantly, the schema definition 20 | language, which users use to communicate with EdgeDB, need to avoid 21 | unnecessary verbosity. 22 | 23 | This requires some conventions and assumptions about what the user expects 24 | most of the time. Those assumptions shouldn't bar the user from changing 25 | their mind when the need arises. 26 | 27 | A default value has far reaching consequences which we cover in this 28 | document. 29 | 30 | 31 | Implementation 32 | ============== 33 | 34 | Since the discussion always applies to properties and links, we're referring 35 | to them simply as *pointers* below. 36 | 37 | Assume OPTIONAL qualifiers when no explicit qualifier is provided 38 | ----------------------------------------------------------------- 39 | 40 | This is the currently implemented behavior as of EdgeDB v1.0 alpha 3. 41 | 42 | An optional pointer is a form of incomplete data that allows the application 43 | to flexibly decide how to proceed. If the data is there, fine. If it isn't, 44 | the application can include user-level code to deal with that. 45 | 46 | Moreover, multi properties and multi links make the case of an empty set 47 | intuitively acceptable: since the cardinality allows for any number of 48 | elements, a state with an empty set is not surprising. 49 | 50 | Plenty of examples of existing databases and RPC systems assume optionality 51 | by default. Relational databases assume optionality unless a NOT NULL 52 | constraint is passed. Protocol buffers removed support for required fields 53 | altogether in version 3. JSON-based NoSQL databases like MongoDB assume 54 | optionality. 55 | 56 | Promoting an optional field into a required field doesn't create silent 57 | failures in pre-existing queries. 58 | 59 | Acknowledged downsides 60 | ~~~~~~~~~~~~~~~~~~~~~~ 61 | 62 | Developers might be failing to fill pointers at INSERT time due 63 | to forgetfulness at query creation time, even if the data could be 64 | provided. They might be harder to fill at a later stage. 65 | 66 | Optionality is a special case that the user code has to deal with. Empty sets 67 | in queries might leave to unexpected results, i.e. missing objects. An empty 68 | set is a special case of a default value. A better default might be possible 69 | for a given pointer. 70 | 71 | Assuming optionality makes it laborious for the user to change their mind 72 | later, i.e. alter a pointer to become required: 73 | 74 | 1. there might already be objects in the database with an empty set in 75 | the given pointer; and 76 | 2. client code written to insert data to the database or update data in 77 | it might become invalid if it depended on the optionality. 78 | 79 | The former issue may be addressed with a data migration prior to the 80 | schema migration. The latter problem is out of scope for EdgeDB but 81 | can be handled by migrating client code *first*. If the clients aren't 82 | under the database administrator's control, a reasonable choice in this 83 | case is to abort the attempt to make a pointer required. 84 | 85 | Additionally, optionality by default in pointers is inconsistent with 86 | non-optionality in function arguments. 87 | 88 | 89 | Rejected ideas 90 | ============== 91 | 92 | Assume REQUIRED qualifiers when no explicit qualifier is provided 93 | ----------------------------------------------------------------- 94 | 95 | An extreme form of optionality everywhere is equivalent to the database 96 | not having a true schema. Unless it's necessary to make a pointer optional, 97 | forcing pointers to be required ensures that whenever data *can* be specified, 98 | it won't be forgotten at INSERT or UPDATE time. 99 | 100 | Required pointers simplify type information. There's two particular problems 101 | on the client side with unexpected optionality: 102 | 103 | * lack of handling for that special case in user code, and 104 | 105 | * surprising effects of EdgeQL queries that omit objects with unexpected 106 | empty sets in pointers used in the query. 107 | 108 | However, since many pointers in real-world schemas will be declared as OPTIONAL 109 | regardless of the default so the downsides of optionality wouldn't be 110 | solved by using REQUIRED as the default qualifier. 111 | 112 | Assuming that all pointers are required unless otherwise specified makes it 113 | easier to migrate into optionality later. Client code that was forced to 114 | provide values at INSERT time before will continue working and there are no 115 | incompatible empty sets in the database prior to the migration. However, 116 | see "Downsides" below. 117 | 118 | Downsides 119 | ~~~~~~~~~ 120 | 121 | The main disadvantage at time of writing is that the current behavior as of 122 | EdgeDB v1.0a3 is the opposite: pointers without an explicit qualifier are 123 | assumed optional. This poses a `Migration challenge`_. 124 | 125 | Required pointers have to be always specified at INSERT time, making 126 | queries more verbose to write and data more costly to send over the wire. 127 | This can be dealt with by providing default values with SET DEFAULT which 128 | is a form of optionality with an explicit fallback value which doesn't fall 129 | outside of the declared type. 130 | 131 | Required pointers by default make it more likely that users would have to 132 | make required-to-optional pointer migrations in the future. Those hide 133 | a specific gotcha: pre-existing queries that were previously guaranteed 134 | never to encounter empty sets now will. Most of the time this will be 135 | harmless but some queries that are analytical in nature will now fail to 136 | return objects with empty pointers, which might lead to bugs in user code. 137 | 138 | Migration challenge 139 | ~~~~~~~~~~~~~~~~~~~ 140 | 141 | All existing EdgeDB instances assume optionality so a migration would be 142 | in order. The trickiness of that migration is a significant flaw of 143 | the proposal to change the current behavior. 144 | 145 | As of Alpha 3, DESCRIBE already includes explicit OPTIONAL qualifiers 146 | everywhere. However, users' ``.esdl`` files and ``.edgeql`` files with 147 | DDL commands don't. 148 | 149 | In consequence, while a DUMP and RESTORE from Alpha 3 will likely work 150 | in the face of a change to implicit REQUIRED, user code does not expect 151 | this. 152 | 153 | For user safety, EdgeDB would need an intermediate release that forces all 154 | pointer definitions to specify qualifiers explicitly. Since there is little 155 | adoption of the product so far, this might not be a big problem. 156 | 157 | Do not provide a default optionality qualifier 158 | ---------------------------------------------- 159 | We rejected this idea because it made even the simplest DDL and SDL 160 | declarations much wordier. This verbosity was detrimental to readability and 161 | required more typing from the user. 162 | 163 | More importantly, despite the usability sacrifice, this approach would solve 164 | the issue only partially. It is still perfectly possible for the user to 165 | choose the wrong qualifier initially, for instance by making the wrong 166 | assumption about a given pointer, or by blindly copying it from somewhere else. 167 | 168 | 169 | Other observations 170 | ================== 171 | 172 | A database is not a peer-to-peer RPC platform 173 | --------------------------------------------- 174 | 175 | Thrift documentation explains that "required is forever". What they mean by 176 | this is that required fields have to be provided by the caller, say a mobile 177 | device. Changing a required field into an optional field requires for the RPC 178 | server to be updated first before any RPC client code can use this 179 | capability. 180 | 181 | While this is a valid concern for RPC systems where new clients won't be able 182 | to connect to old servers, in the case of central databases, the most popular 183 | deployment scheme already is to migrate the database first. The central base 184 | is an easier target to control. It's true that there's a risk that this 185 | ordering will not be kept if the schema definition files are shared between 186 | teams responsible for backend and frontend code. 187 | 188 | The concerns listed by maintainers of protocol buffers and Thrift don't 189 | seem like they apply to a database which is set up as a central API layer 190 | and the source of truth for the given data. 191 | 192 | Protocol buffers removed required fields in version 3, but they also 193 | removed custom default values and rejected the idea of custom validators. 194 | EdgeDB supports both of those features, the latter being constraints. 195 | 196 | The REQUIRED qualifier is a special form of a constraint 197 | -------------------------------------------------------- 198 | 199 | In this sense, specifying a constraint first and removing it later is 200 | easier to deal with than the opposite operation. Not only is the migration 201 | easier but specifying a constraint early usually leads to better data 202 | quality and avoids user-side bugs. 203 | -------------------------------------------------------------------------------- /text/0001-rfc-process.rst: -------------------------------------------------------------------------------- 1 | :: 2 | 3 | Status: Active 4 | Type: Process 5 | Created: 2020-02-04 6 | Authors: Łukasz Langa <lukasz@edgedb.com> 7 | RFC PR: `edgedb/rfcs#0003 <https://github.com/edgedb/rfcs/pull/3>`_ 8 | 9 | ========================= 10 | RFC 0001: The RFC Process 11 | ========================= 12 | 13 | This RFC describes the process of submitting, discussing, and deciding 14 | on large-scale changes to EdgeDB. It is loosely modeled after Python's 15 | `PEP process <https://www.python.org/dev/peps/pep-0001/>`_ which in turn 16 | is loosely modeled after the `Request For Comments 17 | <https://en.wikipedia.org/wiki/Request_for_Comments>`_ process dating 18 | back to the Spring of '69. 19 | 20 | 21 | When to write an RFC 22 | ==================== 23 | 24 | The "Request For Comments" document is intended to be the primary 25 | mechanism for proposing a major change to EdgeDB, both the database 26 | server, as well as related technology like the query languages, binary 27 | protocols, client APIs, and so on. 28 | 29 | The primary audience for RFCs are the core developers of EdgeDB, as well 30 | as the community whose input on the related issue is collected. 31 | 32 | The EdgeDB user community may choose to use the process to design and 33 | document expected conventions, integrations, manage complex design 34 | coordination problems that require collaboration across multiple 35 | projects. 36 | 37 | The following non-exhaustive list demonstrates the kinds of changes 38 | that should be discussed through an RFC: 39 | 40 | * new major feature of EdgeDB (access control, database migrations, 41 | automatic admin panel, and so on); 42 | 43 | * incompatible changes in any API, including the command line; 44 | 45 | * new or changed query syntax. 46 | 47 | For changes of smaller scale use the EdgeDB issue tracker or another 48 | related issue tracker. 49 | 50 | 51 | Structure of an RFC 52 | =================== 53 | 54 | Document Format 55 | --------------- 56 | 57 | An EdgeDB RFC document is a UTF-8 encoded text file formatted with 58 | `ReStructured Text <https://docutils.sourceforge.io/rst.html>`_. 59 | Therefore, it uses the `.rst` file extension. 60 | 61 | The file name should start with a unique four digit number, left-padded 62 | with zeroes if necessary, followed by a lowercase 63 | `slug <https://docs.djangoproject.com/en/3.0/glossary/#term-slug>`_. 64 | 65 | Lines in RFC documents should be wrapped at 72 characters. Lines are 66 | ended with Unix-style newline characters. Trailing whitespace should 67 | be removed from all lines. The last line should end with a newline 68 | character. You can use the ``.editorconfig`` file in this repository 69 | to help you with those requirements. 70 | 71 | Required Sections 72 | ----------------- 73 | 74 | 1. An `IETF RFC 822 <https://tools.ietf.org/html/rfc822>`_ **Preamble** 75 | with metadata about the RFC. See the top of this RFC for a list of 76 | accepted fields and their format. 77 | 78 | 2. A short, ideally tweet-size, **abstract** describing of the issue 79 | being addressed. 80 | 81 | 3. A short but crucial section on **motivation** behind the change. It 82 | should clearly explain why the existing situation is inadequate to 83 | address the problem that the RFC solves. Submissions without 84 | sufficient motivation are unlikely to be discussed and ultimately 85 | accepted. 86 | 87 | 4. A technical **specification** of the semantics and syntax of the 88 | feature. The specification should be detailed enough to allow 89 | competing, interoperable implementations. 90 | 91 | 5. An explicit discussion of **backwards compatibility** and 92 | **security implications** of the discussed change. Both are required 93 | to ensure both the PEP author as well as all readers are aware of the 94 | consequences of the change. 95 | 96 | 6. An extensive discussion of **rejected alternative ideas**. This 97 | covers both large-scale alternative approaches to the problem, as 98 | well as rejected details of implementation of the approach that the 99 | RFC is proposing. The summary of a rejected idea should be presented 100 | along with the reasoning as to why it was rejected. 101 | 102 | Optional Sections 103 | ----------------- 104 | 105 | The RFC document should be split into sections in a natural fashion that 106 | facilitates understanding the subject matter by reading it from top to 107 | bottom. 108 | 109 | Sometimes a larger section on the **rationale** behind the selected 110 | design will be useful. Sometimes it might make sense to discuss 111 | **how the new feature should be taught** to newcomers or discovered by 112 | existing users. A discussion of the **reference implementation** might 113 | be necessary if its technical details are the main subject matter of 114 | the RFC. 115 | 116 | Finally, depending on the state of the RFC, enumerating **open issues** 117 | and **external references** might be helpful to the reader. 118 | 119 | RFC Types and Statuses 120 | ---------------------- 121 | 122 | An RFC can describe a **Feature**, a **Process**, or a **Guideline**. 123 | All RFCs start in the **Draft** status. Depending on the result of 124 | discussion, they get **Accepted**, **Rejected**, or **Deferred**. 125 | 126 | Accepted features become **Final** once they are implemented, with 127 | a note on which EdgeDB version the change is implemented in. 128 | 129 | Accepted processes and guidelines become **Active**, with a note about 130 | scope and effective date. Later, they might become **Inactive** if they 131 | are abandoned or replaced by a different process or guideline. 132 | 133 | 134 | Acceptance Workflow 135 | =================== 136 | 137 | Before you submit an RFC 138 | ------------------------ 139 | 140 | The change you have in mind might have been discussed elsewhere, or 141 | might be part of a larger change, or might be something that is out 142 | of scope for EdgeDB. Before you begin writing a formal RFC, reach out 143 | to the team on any community support channel, like our `Spectrum chat 144 | <https://spectrum.chat/edgedb/>`_, to gather some initial feedback on 145 | your idea. 146 | 147 | Submitting an RFC 148 | ----------------- 149 | 150 | RFCs are submitted to the https://github.com/edgedb/rfcs repository in 151 | the form of pull requests. Choose the next available number above 1000. 152 | Lower numbers are reserved for process-related documentation and guides. 153 | Avoid choosing "cute" arbitrary numbers for RFCs. 154 | 155 | The RFC should have a main champion, typically the author, who is 156 | responsible for moving the discussion forward, as well as gathering and 157 | documenting community and core developer feedback. Having a quick 158 | feedback loop and an up-to-date RFC document is very helpful. 159 | 160 | It's okay if no core developers are co-authors on a given RFC. In this 161 | case prepare for a few more rounds of pull request review about the 162 | logistics of the RFC process and expected content. Try for the initial 163 | pull request to be as close to the document format described in the 164 | section above. 165 | 166 | The discussion period 167 | --------------------- 168 | 169 | The goal of discussing the RFC is to build consensus. 170 | 171 | Ideally the number of communication channels involved in discussing an 172 | RFC is kept at a minimum. The initial version will be discussed on the 173 | original pull request. Subsequent commits on this pull request should 174 | represent granular changes to facilitate easy review. Ideally they 175 | should mark particular comments on the previous version as resolved. 176 | Adding an idea to the "Rejected ideas" section is a form of resolution. 177 | 178 | If there *are* external discussion channels, the RFC champion is 179 | expected to follow them and to gather and integrate feedback from them. 180 | 181 | All community members must be enabled to share feedback. Moderators of 182 | official EdgeDB communication channels enforce the Code of Conduct first 183 | and foremost, to ensure healthy interaction between all interested 184 | parties. If necessary, enforcement can result in a given participant 185 | being excluded from further discussion and thus the decision process. 186 | 187 | Final comment period 188 | -------------------- 189 | 190 | At some point, when the discussion no longer yields new view points, 191 | issues, or solutions, the RFC champion or one of the core developers 192 | can propose a "motion for final comment period", along with 193 | a recommendation to either: 194 | 195 | * accept; 196 | * reject; or 197 | * defer the RFC. 198 | 199 | To enter the final comment period, the motion should be accompanied with 200 | a summary comment of the current state of discussion, ideally already 201 | represented in the RFC text. It's especially important to include any 202 | major points of disagreement and tradeoffs. 203 | 204 | The final comment period lasts for ten business days to allow 205 | stakeholders to file any final objections before a decision is reached. 206 | 207 | Revisiting deferred and rejected RFCs 208 | ------------------------------------- 209 | 210 | Before attempting to restart discussion of a deferred or rejected RFCs, 211 | the relevant interested parties must contact the previous champion and 212 | core developers active in that discussion. If they agree there is 213 | substantial evidence to justify revisiting the idea, a pull request 214 | editing the deferred or rejected RFC can be opened. 215 | 216 | Failure to get proper buy-in beforehand will likely result in immediate 217 | rejection of a pull request on a deferred or rejected RFC. 218 | -------------------------------------------------------------------------------- /text/1021-rewrites.rst: -------------------------------------------------------------------------------- 1 | :: 2 | 3 | Status: Accepted 4 | Type: Feature 5 | Created: 2023-01-12 6 | Authors: Michel J. Sullivan <sully@msully.net> 7 | 8 | =========================== 9 | RFC 1021: Mutation rewrites 10 | =========================== 11 | 12 | This RFC proposes a mechanism for specifying rewrite rules that are 13 | applied to properties and links on INSERT and UPDATE of objects. 14 | 15 | They can be thought of as a generalization of ``default``. 16 | 17 | 18 | Motivation 19 | ========== 20 | 21 | It is sometimes useful to have a pointer in an object automatically updated 22 | whenever the object has an ``UPDATE`` performed on it. 23 | 24 | The most straightforward example of this is having a 25 | ``modification_time`` field that tracks when an object was last 26 | modified. 27 | 28 | It is also valuable when (probably for performance reasons) you wish 29 | to maintain a field that is derived from other object state (for 30 | example, maintaining a count of friends) or if you want to rewrite 31 | incoming data on the database side, perhaps for backward compatability 32 | reasons. 33 | 34 | 35 | Note 36 | ==== 37 | 38 | This is a companion to 39 | `RFC 1020 <https://github.com/edgedb/rfcs/blob/master/text/1020-triggers.rst>`_, 40 | in that they work together to cover most of the desired use cases that motivated 41 | a previous `previous version triggers RFC <https://github.com/edgedb/rfcs/pull/70>`_ 42 | 43 | 44 | Specification 45 | ============= 46 | 47 | Synopsis:: 48 | 49 | type <type-name> "{" 50 | { property | link } <name> -> <type> "{" 51 | rewrite { insert | update } [, ... ] using (<expr>) 52 | "}" 53 | "}" 54 | 55 | We allow pointer definitions to be augmented with a ``rewrite`` rule. 56 | 57 | Expression interpretation 58 | ------------------------- 59 | In ``<expr>``, ``__subject__`` (and the implicit path prefix) will be 60 | bound to the new value object being inserted or updated. 61 | 62 | In purely ``update`` rewrite rules, ``__old__`` will refer to the old 63 | version of the object. 64 | 65 | The special variable ``__specified__`` will be bound to a named tuple 66 | containing a ``bool`` field for each pointer of ``<type-name>`` that 67 | is true if that pointer was explicitly provided in the ``insert`` or 68 | ``update``. 69 | This allows rewrite rules to distinguish between an 70 | explicitly provided empty set and no value being specified. 71 | 72 | The names are sorted alphabetically in the tuple. 73 | Note that which pointers appear in the tuple is determined by the 74 | type where the rewrite rule was *defined*, not the type at which 75 | it is being *evaluated* (which could be a subtype). This ensures 76 | that the type is always consistent. 77 | 78 | (Whether ``<type-name>`` should refer to the object being mutated is 79 | kind of fraught; we leave that undecided for now. Some discussion of 80 | that 81 | `here 82 | <https://github.com/edgedb/edgedb/issues/4142#issuecomment-1386287450>_`.) 83 | 84 | Restrictions 85 | ------------ 86 | Each pointer can have one ``insert`` rewrite rule and one ``update`` 87 | rewrite rule. They can be specified together if desired. 88 | 89 | Behavior 90 | -------- 91 | 92 | For any pointer, the rewrite rule that applies is the one that appears 93 | *first* in the MRO (like we do with defaults). 94 | 95 | When we perform ``insert`` or ``update``, after computing the 96 | values for the object, but before performing the operation or 97 | evaluating the write-side access policies, we evaluate each of the 98 | applicable rewrite rules and take the result as the actual value for 99 | the pointer. 100 | 101 | The rewrites are evaluated "simultaneously"; each rewrite sees the 102 | original object. 103 | 104 | Access policies *are* evaluated inside the expression. 105 | 106 | The rule for a pointer is applied whether or not that pointer is 107 | explicitly specified in the operation. ``__specified__`` can be used 108 | to determine whether it was and take action based on that. 109 | 110 | Note that for ``insert``, rewrite rules are *extremely* similar to 111 | ``default`` (which can now refer to other pointers), but are applied 112 | even when a value was specified. 113 | 114 | If a pointer has both a rewrite and a default, and the value was not 115 | specified during an insert, default is applied first. 116 | Within rewrite's ``<expr>``, ``__subject__`` contains the value provided 117 | by the default, but ``__specified__`` contains ``false``, reflecting the 118 | state of the original query. 119 | 120 | Examples 121 | ======== 122 | 123 | Update a modification time field every time an object is modified:: 124 | 125 | type Entry { 126 | # ... 127 | required property mtime -> datetime { 128 | rewrite insert, update using (datetime_of_statement()) 129 | } 130 | }; 131 | 132 | 133 | Update a modification time field every time an object is modified, but 134 | allow a manual override of it also:: 135 | 136 | type Entry { 137 | # ... 138 | required property mtime -> datetime { 139 | rewrite insert, update using ( 140 | datetime_of_statement() 141 | if not __specified__.mtime 142 | else .mtime 143 | ) 144 | } 145 | }; 146 | 147 | Maintain a cached count of the cardinality of a multi link:: 148 | 149 | type Post { 150 | # ... 151 | multi link likes -> Like; 152 | required property cached_like_count -> int64 { 153 | rewrite insert, update using (count(.likes)) 154 | } 155 | }; 156 | 157 | Rewrite a incoming pointer (maybe for backward compatibility after a 158 | format change):: 159 | 160 | type Item { 161 | # ... 162 | required property product_code -> str { 163 | rewrite insert, update using (str_upper(.product_code)) 164 | } 165 | }; 166 | 167 | 168 | Backwards Compatibility 169 | ======================= 170 | 171 | There should not be any backwards compatibility issues. 172 | 173 | 174 | Implementation considerations 175 | ============================= 176 | 177 | Most of the infrastructure for computing a "contents" row for the 178 | main object table is already there, and it shouldn't be too hard to 179 | wrap that and replace some fields in it. 180 | 181 | Dealing with ``multi`` pointers might be pretty nasty, though. We 182 | don't currently generate "contents" CTEs for them in all the general 183 | cases (such as doing ``-=``), so there might be a lot of subtle 184 | engineering work needed to get everything positioned for this. 185 | 186 | We can probably skip supporting ``multi`` pointers in the first take 187 | of this, if necessary. 188 | 189 | 190 | Security Implications 191 | ===================== 192 | 193 | Access policies *are* evaluated inside the expression. 194 | 195 | 196 | Rejected Alternative Ideas 197 | ========================== 198 | 199 | Different ways of representing which pointers are specified 200 | ----------------------------------------------------------- 201 | 202 | The original proposal had a ``__fields__`` field that contained a 203 | set of strings of names of specified pointers. This worked but was 204 | ugly and would have required some special work to implement 205 | efficiently in the common case. If you actually want such a set, 206 | it can be obtained in the current proposal with:: 207 | 208 | (select json_object_unpack(<json>__specified__) filter <bool>.1).0 209 | 210 | Another proposal was to have a magic "function" (or operator) that 211 | returned whether a field was set, such as ``specified(.friends)`` 212 | would be true if ``friends`` was specified in the DML statement. 213 | This was rejected because it had to either be purely magic syntax 214 | or required introducing a new notion of "unspecified" into the 215 | semantics that could only be distinguished from ``{}`` by the 216 | new ``specified`` function, and because the named tuple proposal 217 | reads just as well but without any worrying implications. 218 | 219 | Calling ``__specified__`` something else 220 | ---------------------------------------- 221 | 222 | Originally I proposed ``__fields__``, which was bad. ``__specified__`` 223 | is kind of long, so something shorter would be nice, but our time 224 | spent looking at a thesaurus did not help us. 225 | 226 | The best option we had was ``__set__``, which Yury hated. That would 227 | look something like:: 228 | 229 | type Entry { 230 | # ... 231 | required property mtime -> datetime { 232 | rewrite insert, update using ( 233 | datetime_of_statement() if not __set__.mtime else .mtime 234 | ) 235 | } 236 | }; 237 | 238 | 239 | Making this explicitly an extension of default 240 | ---------------------------------------------- 241 | 242 | Another proposal was to treat this exactly as default generalized to 243 | ``update`` (to handle the mtime cases) and to add a notion of 244 | ``cached property`` for things like the ``cached_like_count`` case. 245 | 246 | This was rejected because while we do eventually want some kind of 247 | cached/materialized values, there is a lot of complexity in the design 248 | space there and we don't want to ship a super limited version of it 249 | that might mislead users and limit our options in the future. 250 | 251 | It also doesn't support genuine "rewrite" style operations. 252 | 253 | 254 | Making mutation rewrites per-object instead of per-pointer 255 | ---------------------------------------------------------- 256 | 257 | Doing it per-object makes it unclear how it should compose in the 258 | presence of inheritance. We would need to be much more innovative 259 | in terms of syntax and semantics. (Probably: return a free object, 260 | which then gets composed in some way.) 261 | 262 | 263 | Generalized policy based query rewrite 264 | -------------------------------------- 265 | A `previous RFC 266 | <https://github.com/edgedb/rfcs/pull/50>`_ written by Elvis, combined 267 | triggers and access policies into one generic mechanism. We decided 268 | this was likely to be too complex, and that they should be split. 269 | 270 | I also think there would have been severe implementation difficulties. 271 | 272 | 273 | Using triggers and having a BEFORE/AFTER split 274 | ---------------------------------------------- 275 | 276 | Another `previous version of the trigger RFC 277 | <https://github.com/edgedb/rfcs/pull/70>`_, contained 278 | a distinction between ``BEFORE`` triggers and ``AFTER`` triggers. 279 | 280 | ``AFTER`` triggers would be run in a pipelined query, would not have 281 | access to ``__old__`` (and as such could not be used for ``DELETE``), 282 | and *could* modify objects that had already been modified in the 283 | original query. 284 | 285 | That handled this case, and was probably workable, but was generally 286 | complex and the distinctions between ``BEFORE`` and ``AFTER`` triggers 287 | were weird and heavily implementation driven. 288 | 289 | 290 | Implement using postgres triggers 291 | --------------------------------- 292 | 293 | There is a critical semantic problem in using postgres triggers, which 294 | is that postgres triggers only have access to the old state of the 295 | database and to the new rows. But in edgedb, the state of an object 296 | might be spread across multiple tables (for multi pointers), and so 297 | the full state of a new or updated object may be invisible to a 298 | postgres trigger. 299 | -------------------------------------------------------------------------------- /LICENSE-APACHE: -------------------------------------------------------------------------------- 1 | Apache License 2 | Version 2.0, January 2004 3 | http://www.apache.org/licenses/ 4 | 5 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 6 | 7 | 1. Definitions. 8 | 9 | "License" shall mean the terms and conditions for use, reproduction, 10 | and distribution as defined by Sections 1 through 9 of this document. 11 | 12 | "Licensor" shall mean the copyright owner or entity authorized by 13 | the copyright owner that is granting the License. 14 | 15 | "Legal Entity" shall mean the union of the acting entity and all 16 | other entities that control, are controlled by, or are under common 17 | control with that entity. For the purposes of this definition, 18 | "control" means (i) the power, direct or indirect, to cause the 19 | direction or management of such entity, whether by contract or 20 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 21 | outstanding shares, or (iii) beneficial ownership of such entity. 22 | 23 | "You" (or "Your") shall mean an individual or Legal Entity 24 | exercising permissions granted by this License. 25 | 26 | "Source" form shall mean the preferred form for making modifications, 27 | including but not limited to software source code, documentation 28 | source, and configuration files. 29 | 30 | "Object" form shall mean any form resulting from mechanical 31 | transformation or translation of a Source form, including but 32 | not limited to compiled object code, generated documentation, 33 | and conversions to other media types. 34 | 35 | "Work" shall mean the work of authorship, whether in Source or 36 | Object form, made available under the License, as indicated by a 37 | copyright notice that is included in or attached to the work 38 | (an example is provided in the Appendix below). 39 | 40 | "Derivative Works" shall mean any work, whether in Source or Object 41 | form, that is based on (or derived from) the Work and for which the 42 | editorial revisions, annotations, elaborations, or other modifications 43 | represent, as a whole, an original work of authorship. For the purposes 44 | of this License, Derivative Works shall not include works that remain 45 | separable from, or merely link (or bind by name) to the interfaces of, 46 | the Work and Derivative Works thereof. 47 | 48 | "Contribution" shall mean any work of authorship, including 49 | the original version of the Work and any modifications or additions 50 | to that Work or Derivative Works thereof, that is intentionally 51 | submitted to Licensor for inclusion in the Work by the copyright owner 52 | or by an individual or Legal Entity authorized to submit on behalf of 53 | the copyright owner. For the purposes of this definition, "submitted" 54 | means any form of electronic, verbal, or written communication sent 55 | to the Licensor or its representatives, including but not limited to 56 | communication on electronic mailing lists, source code control systems, 57 | and issue tracking systems that are managed by, or on behalf of, the 58 | Licensor for the purpose of discussing and improving the Work, but 59 | excluding communication that is conspicuously marked or otherwise 60 | designated in writing by the copyright owner as "Not a Contribution." 61 | 62 | "Contributor" shall mean Licensor and any individual or Legal Entity 63 | on behalf of whom a Contribution has been received by Licensor and 64 | subsequently incorporated within the Work. 65 | 66 | 2. Grant of Copyright License. Subject to the terms and conditions of 67 | this License, each Contributor hereby grants to You a perpetual, 68 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 69 | copyright license to reproduce, prepare Derivative Works of, 70 | publicly display, publicly perform, sublicense, and distribute the 71 | Work and such Derivative Works in Source or Object form. 72 | 73 | 3. Grant of Patent License. Subject to the terms and conditions of 74 | this License, each Contributor hereby grants to You a perpetual, 75 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 76 | (except as stated in this section) patent license to make, have made, 77 | use, offer to sell, sell, import, and otherwise transfer the Work, 78 | where such license applies only to those patent claims licensable 79 | by such Contributor that are necessarily infringed by their 80 | Contribution(s) alone or by combination of their Contribution(s) 81 | with the Work to which such Contribution(s) was submitted. If You 82 | institute patent litigation against any entity (including a 83 | cross-claim or counterclaim in a lawsuit) alleging that the Work 84 | or a Contribution incorporated within the Work constitutes direct 85 | or contributory patent infringement, then any patent licenses 86 | granted to You under this License for that Work shall terminate 87 | as of the date such litigation is filed. 88 | 89 | 4. Redistribution. You may reproduce and distribute copies of the 90 | Work or Derivative Works thereof in any medium, with or without 91 | modifications, and in Source or Object form, provided that You 92 | meet the following conditions: 93 | 94 | (a) You must give any other recipients of the Work or 95 | Derivative Works a copy of this License; and 96 | 97 | (b) You must cause any modified files to carry prominent notices 98 | stating that You changed the files; and 99 | 100 | (c) You must retain, in the Source form of any Derivative Works 101 | that You distribute, all copyright, patent, trademark, and 102 | attribution notices from the Source form of the Work, 103 | excluding those notices that do not pertain to any part of 104 | the Derivative Works; and 105 | 106 | (d) If the Work includes a "NOTICE" text file as part of its 107 | distribution, then any Derivative Works that You distribute must 108 | include a readable copy of the attribution notices contained 109 | within such NOTICE file, excluding those notices that do not 110 | pertain to any part of the Derivative Works, in at least one 111 | of the following places: within a NOTICE text file distributed 112 | as part of the Derivative Works; within the Source form or 113 | documentation, if provided along with the Derivative Works; or, 114 | within a display generated by the Derivative Works, if and 115 | wherever such third-party notices normally appear. The contents 116 | of the NOTICE file are for informational purposes only and 117 | do not modify the License. You may add Your own attribution 118 | notices within Derivative Works that You distribute, alongside 119 | or as an addendum to the NOTICE text from the Work, provided 120 | that such additional attribution notices cannot be construed 121 | as modifying the License. 122 | 123 | You may add Your own copyright statement to Your modifications and 124 | may provide additional or different license terms and conditions 125 | for use, reproduction, or distribution of Your modifications, or 126 | for any such Derivative Works as a whole, provided Your use, 127 | reproduction, and distribution of the Work otherwise complies with 128 | the conditions stated in this License. 129 | 130 | 5. Submission of Contributions. Unless You explicitly state otherwise, 131 | any Contribution intentionally submitted for inclusion in the Work 132 | by You to the Licensor shall be under the terms and conditions of 133 | this License, without any additional terms or conditions. 134 | Notwithstanding the above, nothing herein shall supersede or modify 135 | the terms of any separate license agreement you may have executed 136 | with Licensor regarding such Contributions. 137 | 138 | 6. Trademarks. This License does not grant permission to use the trade 139 | names, trademarks, service marks, or product names of the Licensor, 140 | except as required for reasonable and customary use in describing the 141 | origin of the Work and reproducing the content of the NOTICE file. 142 | 143 | 7. Disclaimer of Warranty. Unless required by applicable law or 144 | agreed to in writing, Licensor provides the Work (and each 145 | Contributor provides its Contributions) on an "AS IS" BASIS, 146 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 147 | implied, including, without limitation, any warranties or conditions 148 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 149 | PARTICULAR PURPOSE. You are solely responsible for determining the 150 | appropriateness of using or redistributing the Work and assume any 151 | risks associated with Your exercise of permissions under this License. 152 | 153 | 8. Limitation of Liability. In no event and under no legal theory, 154 | whether in tort (including negligence), contract, or otherwise, 155 | unless required by applicable law (such as deliberate and grossly 156 | negligent acts) or agreed to in writing, shall any Contributor be 157 | liable to You for damages, including any direct, indirect, special, 158 | incidental, or consequential damages of any character arising as a 159 | result of this License or out of the use or inability to use the 160 | Work (including but not limited to damages for loss of goodwill, 161 | work stoppage, computer failure or malfunction, or any and all 162 | other commercial damages or losses), even if such Contributor 163 | has been advised of the possibility of such damages. 164 | 165 | 9. Accepting Warranty or Additional Liability. While redistributing 166 | the Work or Derivative Works thereof, You may choose to offer, 167 | and charge a fee for, acceptance of support, warranty, indemnity, 168 | or other liability obligations and/or rights consistent with this 169 | License. However, in accepting such obligations, You may act only 170 | on Your own behalf and on Your sole responsibility, not on behalf 171 | of any other Contributor, and only if You agree to indemnify, 172 | defend, and hold each Contributor harmless for any liability 173 | incurred by, or claims asserted against, such Contributor by reason 174 | of your accepting any such warranty or additional liability. 175 | 176 | END OF TERMS AND CONDITIONS 177 | 178 | APPENDIX: How to apply the Apache License to your work. 179 | 180 | To apply the Apache License to your work, attach the following 181 | boilerplate notice, with the fields enclosed by brackets "[]" 182 | replaced with your own identifying information. (Don't include 183 | the brackets!) The text should be enclosed in the appropriate 184 | comment syntax for the file format. We also recommend that a 185 | file or class name and description of purpose be included on the 186 | same "printed page" as the copyright notice for easier 187 | identification within third-party archives. 188 | 189 | Copyright [yyyy] [name of copyright owner] 190 | 191 | Licensed under the Apache License, Version 2.0 (the "License"); 192 | you may not use this file except in compliance with the License. 193 | You may obtain a copy of the License at 194 | 195 | http://www.apache.org/licenses/LICENSE-2.0 196 | 197 | Unless required by applicable law or agreed to in writing, software 198 | distributed under the License is distributed on an "AS IS" BASIS, 199 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 200 | See the License for the specific language governing permissions and 201 | limitations under the License. 202 | -------------------------------------------------------------------------------- /text/1011-object-level-security.rst: -------------------------------------------------------------------------------- 1 | :: 2 | 3 | Status: Accepted 4 | Type: Feature 5 | Created: 2022-04-18 6 | Authors: Elvis Pranskevichus <elvis@edgedb.com> 7 | RFC PR: `edgedb/rfcs#0054 <https://github.com/edgedb/rfcs/pull/54>`_ 8 | 9 | =============================== 10 | RFC 1011: Object-Level Security 11 | =============================== 12 | 13 | This RFC proposes adding a mechanism to declare per-type access control 14 | rules in schemas to automatically enforce data access and modification on 15 | per-object level. This is analogous to row-level security (RLS) in SQL 16 | systems. 17 | 18 | 19 | Motivation 20 | ========== 21 | 22 | Object-level security is a powerful feature which allows universal leak-proof 23 | enforcement of data access policies across all uses of a database. This is 24 | important for security and compliance, but would also be tremendously useful 25 | for backend-less applications that interact with EdgeDB via HTTP (either via 26 | GraphQL or EdgeQL-over-HTTP). A secondary motivation is that this will allow 27 | implementation of `temporal databases <temporal>`_. 28 | 29 | 30 | Requirements 31 | ============ 32 | 33 | Although there is no direct dependency, this RFC requires 34 | `RFC 1010 <1001-global-vars.rst>`_ (globals) to be implemented to be practical. 35 | See `Interaction with globals`_ below for details. 36 | 37 | 38 | Specification 39 | ============= 40 | 41 | This RFC proposes to add a new schema item type -- *Access Policy* -- that is 42 | defined in the context of an object type and specifies the rules of rewriting 43 | queries that refer to the enclosing object type. 44 | 45 | CREATE ACCESS POLICY 46 | -------------------- 47 | 48 | Define a new access control policy for a given object type. 49 | 50 | Required capabilities: DDL. 51 | 52 | Synopsis:: 53 | 54 | {CREATE|ALTER} OBJECT TYPE <type-name> "{" 55 | CREATE ACCESS POLICY <name> 56 | [ WHEN <condition> ] 57 | { ALLOW | DENY } 58 | { ALL | UPDATE | SELECT | UPDATE READ | UPDATE WRITE | INSERT | DELETE } [ , ... ] 59 | [ USING (<expr>) ] 60 | "}" 61 | 62 | If at least one access rule is defined for a type, then all elements become 63 | invisible/immutable by default and must be made visible/mutable by at least 64 | one ``ALLOW`` rule. 65 | 66 | An ``ALLOW`` rule *adds* all objects, for which the check expression ``<expr>`` 67 | evaluates to true, to the set of visible objects. ``ALLOW`` rules are 68 | combined using the ``OR`` operator, i.e. they are mutually additive. 69 | 70 | A ``DENY`` rule removes objects for which the check expression ``<expr>`` 71 | evaluates to true from the set of visible objects. ``DENY`` rules are combined 72 | using the ``AND`` operator. 73 | 74 | An ``UPDATE`` policy kind is an abbreviation for ``UPDATE READ, UPDATE WRITE`` 75 | while an ``ALL`` policy is an abbreviation for all five policy kinds. 76 | 77 | Per ``SELECT``, ``UPDATE READ``, and ``DELETE`` policies, objects that 78 | are outside of the visible set are silently skipped in any ``SELECT``, 79 | ``UPDATE`` or ``DELETE`` expression that scans the relevant type. 80 | Note that every ``DELETE`` and ``UPDATE`` does an implicit ``SELECT`` 81 | to produce the set to be modified, and so ``SELECT`` policies restrict 82 | the objects that can be modified by DML as well. 83 | 84 | A ``UPDATE WRITE`` and ``INSERT`` policies specify a *validity* check 85 | for new or updated objects and affects ``INSERT`` and ``UPDATE`` 86 | expressions, respectively: if the proposed object is outside of the 87 | visible set, an error is raised immediately and the query is aborted. 88 | 89 | The optional ``<condition>`` expression is evaluated for every object 90 | affected by the statement and the policy is applied only if the expression 91 | evaluates to *true*. It is essentially equivalent to joining ``<condition>`` 92 | with ``<expr>`` with an ``AND`` operator. The reason for a standalone clause 93 | is that it makes it easier to separate *when* a policy is applied from *how* a 94 | policy is applied. 95 | 96 | Access policies on other types apply to both ``when`` and ``using`` 97 | expressions, to prevent information leaks through that channel. 98 | 99 | The check expression ``<expr>`` may be omitted, which implies that the policy 100 | matches all objects, e.g. this is equivalent to specifying ``using (true)``. 101 | 102 | Example read policy:: 103 | 104 | type Movie { 105 | property rating -> str; 106 | # Allow all movie objects to be read by default 107 | access policy default permit read; 108 | # But deny those that are rated 'R' to users aged under 17. 109 | access policy age_appropriate 110 | when ((global current_user).age < 17) 111 | deny read using (.rating = 'R'); 112 | } 113 | 114 | Example read/write policy:: 115 | 116 | type Post { 117 | property author -> User; 118 | 119 | # Only allow reading to the author, but also 120 | # ensure that a user cannot set the `author` link 121 | # to anything but themselves. 122 | access policy author_only 123 | allow all using (.author = global current_user); 124 | } 125 | 126 | Another example of combination of allow/deny policies:: 127 | 128 | abstract type Owned { 129 | link owner -> User; 130 | 131 | # permit read access to owner 132 | access policy owner_only 133 | allow all using (.owner = global current_user); 134 | } 135 | 136 | abstract type Shared extending Owned { 137 | # allow read access to friends 138 | access policy friends_can_read 139 | allow select using (global current_user in .owner.friends); 140 | } 141 | 142 | # Post inherits policies from Shared 143 | # which allow access to either owner 144 | # or friends initially... 145 | type Post extending Shared { 146 | property private -> bool; 147 | 148 | # ... but restrict access to private posts to owner only 149 | # regardless of what permissions were granted in parent types 150 | access policy private_owner_only 151 | when (.private) 152 | deny all using (.owner != global current_user); 153 | } 154 | 155 | 156 | 157 | ALTER ACCESS POLICY 158 | ------------------- 159 | 160 | Alter the definition of an access control policy. 161 | 162 | Required capabilities: DDL. 163 | 164 | Synopsis:: 165 | 166 | ALTER OBJECT TYPE <type-name> "{" 167 | ALTER ACCESS POLICY <name> 168 | [ "{" <subcommand>; [...] "}" ]; 169 | "}" 170 | 171 | # where <subcommand> is one of 172 | 173 | CREATE ANNOTATION <annotation-name> := <value> 174 | ALTER ANNOTATION <annotation-name> := <value> 175 | DROP ANNOTATION <annotation-name> 176 | WHEN (<condition>) 177 | RESET WHEN 178 | USING (<expr>) 179 | { ALLOW | DENY } { ALL | UPDATE | SELECT | UPDATE READ | UPDATE WRITE | INSERT | DELETE } [ , ... ] 180 | 181 | 182 | DROP ACCESS POLICY 183 | ------------------ 184 | 185 | Remove an access control policy. 186 | 187 | Required capabilities: DDL. 188 | 189 | Synopsis:: 190 | 191 | ALTER OBJECT TYPE <type-name> "{" 192 | DROP ACCESS POLICY <name>; 193 | "}" 194 | 195 | 196 | Interaction with globals 197 | ======================== 198 | 199 | Access policies are especially powerful when combined with RFC 1010 200 | globals, because then data visibility can be globally adjusted with a single 201 | ``SET GLOBAL`` statement, which is very useful for authenticated/authorized 202 | data access control. 203 | 204 | Example:: 205 | 206 | global user_id -> uuid; 207 | 208 | abstract object type Owned { 209 | required link owner -> User; 210 | 211 | access policy owner_only 212 | allow all (.owner.id = global user_id) 213 | } 214 | 215 | object type Purchase extending Owned; 216 | 217 | ... 218 | 219 | set global user_id := <uuid-1>; 220 | select count(Purchase); 221 | # 9 222 | set global user_id := <uuid-2> 223 | select count(Purchase); 224 | # 1 225 | 226 | 227 | Bypassing policies 228 | ================== 229 | 230 | A superuser can bypass the execution of query rewrite policies by setting 231 | the ``apply_access_policies`` session configuration setting to ``false``. 232 | 233 | 234 | Mandatory Role-based Access Control (RBAC) 235 | ========================================== 236 | 237 | Coupled with the role-based permission system (discussed in a future RFC), 238 | object-level security provides reliable mandatory RBAC, where an 239 | ``access policy`` is protected by role permissions and cannot be disabled 240 | by unauthorized users. 241 | 242 | 243 | Introspection 244 | ============= 245 | 246 | Policies can be introspected via a new ``schema::AccessPolicy`` in the 247 | introspection schema that is linked from ``schema::ObjectType`` via the new 248 | ``access_policies`` link. The ``schema::AccessPolicy`` is exposed as follows:: 249 | 250 | type schema::AccessPolicy 251 | extending schema::InheritingObject, schema::AnnotationSubject { 252 | multi property access_kinds -> schema::AccessKind; 253 | property condition -> std::str; 254 | required property action -> schema::AccessPolicyAction; 255 | required property expr -> std::str; 256 | }; 257 | 258 | 259 | Implementation considerations 260 | ============================= 261 | 262 | Access policies primarily affect what IR is generated for a given EdgeQL query. 263 | ``READ`` and ``DELETE`` rules wrap set references and transform every ``Foo`` 264 | reference into ``(SELECT Foo FILTER <allow-deny-filter>)``. 265 | 266 | ``WRITE`` actions insert an intermediate shape into ``INSERT`` and ``UPDATE``, 267 | e.g.:: 268 | 269 | INSERT Foo { prop := <value> } 270 | 271 | is roughly transformed into:: 272 | 273 | WITH 274 | input := { prop := <value> }, 275 | checked := input { 276 | prop := prop IF (SELECT _ := <check_expr> FILTER _) ELSE raise() 277 | } 278 | INSERT Foo { prop := checked.prop } 279 | 280 | If specified, the ``WHEN`` conditions must be taken into account, e.g by 281 | combining directly with the ``ALLOW/DENY`` filters and check expressions. 282 | 283 | 284 | Rejected Alternative Ideas 285 | ========================== 286 | 287 | Generalized policy based query rewrite 288 | -------------------------------------- 289 | 290 | A `previous version of this RFC <https://github.com/edgedb/rfcs/pull/50>`_ 291 | proposed a generic "query rewrite" mechanism allowing, besides security, 292 | also trigger-like functionality, but such bundling and generality was 293 | deemed to be too complex, and the decision was made to add explicit mechanisms 294 | for object-level security and (in a future RFC) support for trigger actions. 295 | 296 | Use database views (a.k.a. contexts) to implement security 297 | ------------------------------------------------------------------------- 298 | 299 | A proposal was made to implement security on schema-level instead of 300 | type-level, e.g:: 301 | 302 | context Authenticate (auth_method -> AuthMethod, token_id -> str) { 303 | type view User using ( 304 | SELECT User 305 | FILTER .session.auth_method = global auth_method 306 | AND .session.token_id = global token_id); 307 | type view Sessions using ( 308 | SELECT Sessions 309 | FILTER .auth_method = global auth_method 310 | AND .token_id = global token_id ); 311 | } 312 | 313 | context User (user_id -> uuid) { 314 | type view User using ( 315 | SELECT User Filter .user_id = global user_id); 316 | type view Article using ( 317 | SELECT Article FILTER .owner.id = global user_id); 318 | type view PublicArticle using ( 319 | SELECT Article FILTER .public); 320 | } 321 | 322 | Context would then need to be activated:: 323 | 324 | SET CONTEXT User { user_id: = <uuid>$user_id }; 325 | 326 | This proposal was rejected because this design poses significant challenges to 327 | composition, i.e. composing several levels of security without the need to 328 | duplicate large chunks of schema, as well as lack of support for mandatory 329 | access control, as contexts are application-centric and are opt-in. 330 | 331 | 332 | Backwards compatibility 333 | ======================= 334 | 335 | This RFC does not pose any backwards compatibility issues. 336 | -------------------------------------------------------------------------------- /text/1012-range-types.rst: -------------------------------------------------------------------------------- 1 | :: 2 | 3 | Status: Accepted 4 | Type: Feature 5 | Created: 2022-05-02 6 | Authors: Victor Petrovykh <victor@edgedb.com> 7 | RFC PR: `edgedb/rfcs#0054 <https://github.com/edgedb/rfcs/pull/57>`_ 8 | 9 | ===================== 10 | RFC 1012: Range types 11 | ===================== 12 | 13 | This RFC proposes adding ``range`` and ``multirange`` scalar types to EdgeQL. 14 | The ``range`` type represents an *interval* of values. While ``multirange`` 15 | type represents a set of non-overlapping ranges. 16 | 17 | 18 | Motivation 19 | ========== 20 | 21 | Having a scalar type representing intervals would allow defining a consistent 22 | way of working with them, such as determining whether two intervals overlap, 23 | whether one interval is contained in the other, etc. The intervals could also 24 | be used in constraints. A simple usage would be to define a single constraint 25 | using an interval instead of using ``min_value`` and ``max_value`` 26 | constraints. More advanced usage could be defining a variant of the 27 | ``exclusive`` constraint which ensures non-overlapping interval property for 28 | some objects. 29 | 30 | Another usage of intervals is in generating sequences of values to be used 31 | directly or to serve as an iterator set in a ``for`` loop. 32 | 33 | 34 | Specification 35 | ============= 36 | 37 | Range types would be similar to ``array`` in that they represent a collection 38 | of values of a particular type. Since intervals in Postgres are not completely 39 | arbitrary, we will only be able to define ``range`` and ``multirange`` over 40 | specific scalar types. 41 | 42 | The base scalar types that ``range`` and ``multirange`` can work with: 43 | * ``anypoint`` - any scalar that has "points" which can be used to construct 44 | a range (like ``int64`` and ``float64``, but not, say, ``json`` and ``str``) 45 | * a helper type expression ``increment<type>``, which is syntax sugar for 46 | looking at the return type of the operator ``-(type, type)``. It serves to 47 | identify the appropriate type for "stepping" through values inside an 48 | interval. 49 | * ``anydiscrete extending anypoint`` - any scalar that conceptually 50 | represents discrete values (like ``int64`` or ``date``) 51 | * ``anycontiguous extending anypoint`` - any scalar that conceptually 52 | represents contiguous values (like ``float64`` or ``datetime``) 53 | 54 | The interval types themselves would be defined similar to arrays: 55 | * ``range<anypoint>`` represents intervals over some concrete ``anypoint`` 56 | sub-type 57 | * ``multirange<anypoint>`` represents a set of non-overlapping ``range`` 58 | values over the corresponding concrete ``anypoint`` sub-type 59 | 60 | 61 | Boundaries 62 | ---------- 63 | 64 | A ``range`` value should have an upper and lower bound and an indicator for 65 | each of them whether they are included in the values the ``range`` represents. 66 | 67 | It should be possible to omit one or both of the boundaries to represent an 68 | open-ended interval. The ``{}`` will be used to represent the omitted 69 | boundary. It should be an error to omit the boundary, but indicate that it is 70 | inclusive. 71 | 72 | 73 | Constructors 74 | ------------ 75 | 76 | We will use a function constructor for creating new ``range`` values:: 77 | 78 | function range( 79 | lower: optional anypoint = {}, 80 | upper: optional anypoint = {}, 81 | named only inc_lower: bool = true, 82 | named only inc_upper: bool = false 83 | ) -> range<anypoint> 84 | 85 | The default values are chosen so as to minimize the amount of keystrokes 86 | required to define ranges for common use-cases such as for iterators. 87 | 88 | The ``inc_lower`` and ``inc_upper`` arguments are ``named only`` because when 89 | their default values need to be overridden it is entirely plausible that only 90 | one of them needs to be specified and thus using positional arguments would be 91 | awkward. 92 | 93 | A ``multirange`` could be constructed using a function that takes a set of 94 | basic ``range`` values:: 95 | 96 | function multirange( 97 | ranges: set of range<anypoint> 98 | ) -> multirange<anypoint> 99 | 100 | The constructor must normalize the ranges to be non-overlapping and order them 101 | based on boundaries. This will ensure that when a ``multirange`` is unpacked 102 | into its component ranges, the order of the ``range`` values would be based on 103 | their boundaries rather than arbitrary. 104 | 105 | 106 | Accessing range components 107 | -------------------------- 108 | 109 | Several functions will be used to access ``range`` components:: 110 | 111 | function range_get_upper(range<anypoint>) -> optional anypoint 112 | function range_get_lower(range<anypoint>) -> optional anypoint 113 | function range_is_inclusive_upper(range<anypoint>) -> bool 114 | function range_is_inclusive_lower(range<anypoint>) -> bool 115 | 116 | 117 | Accessing multirange components 118 | ------------------------------- 119 | 120 | There will be a function for extracting sub-ranges from a multirange:: 121 | 122 | function to_ranges(multirange<anypoint>) -> set of range<anypoint> 123 | 124 | By default this function will return the ranges ordered by boundaries from 125 | lowest to highest. This is the order of definition and Postgres by default 126 | returns the sub-intervals in that order. 127 | 128 | 129 | Helper functions 130 | ---------------- 131 | 132 | In addition to the functions mentioned above there must be a way to generate 133 | an ordered set of values from a ``range`` or ``multirange``:: 134 | 135 | function range_unpack( 136 | val: range<anydiscrete> 137 | ) -> set of anydiscrete 138 | 139 | function range_unpack( 140 | val: range<anydiscrete>, 141 | step: increment<anydiscrete> 142 | ) -> set of anydiscrete 143 | 144 | function range_unpack( 145 | val: multirange<anydiscrete> 146 | ) -> set of anydiscrete 147 | 148 | function range_unpack( 149 | val: multirange<anydiscrete>, 150 | step: increment<anydiscrete> 151 | ) -> set of anydiscrete 152 | 153 | 154 | function range_unpack( 155 | val: range<anycontiguous>, 156 | step: increment<anycontiguous> 157 | ) -> set of anycontiguous 158 | 159 | function range_unpack( 160 | val: multirange<anycontiguous>, 161 | step: increment<anycontiguous> 162 | ) -> set of anycontiguous 163 | 164 | The ``range_unpack`` function for ``anydiscrete`` intervals **must** have 165 | versions with and without a ``step``. The version without a ``step`` should 166 | use a minimal value for stepping through the iterations. On the other hand 167 | ``anycontiguous`` intervals **must** specify the ``step`` explicitly in all 168 | cases as no "natural" step value can be asumed. When invoked on an unbounded 169 | interval ``range_unpack`` will raise an error. 170 | 171 | The actual ``std`` funcitons should not use abstract types, but should be 172 | instead overloads with concrete scalar types. The reason is that the ``step`` 173 | is not necessarily the same type as the main interval subtype. Specifically, 174 | ``cal::local_date``, ``cal::local_time``, ``cal::local_datetime``, 175 | ``datetime``, and ``duration`` all have ``duration`` as the step type. This 176 | situation is currently limited to date/time types, but it's conceivable that 177 | in the future there may be some other types that have this interaction. 178 | Overloads accommodate this behavior well and don't require additional complex 179 | mechanisms for figuring out valid range/step type pairs. 180 | 181 | EdgeDB can automatically detect all concrete types for which ``range_unpack`` 182 | can be defined and generate the appropriate overloaded implementations. 183 | 184 | We also need functions for determining overlapping and whether something is 185 | contained in a range:: 186 | 187 | function contains( 188 | haystack: range<anypoint>, 189 | needle: range<anypoint> 190 | ) -> bool 191 | 192 | function contains( 193 | haystack: range<anypoint>, 194 | needle: anypoint 195 | ) -> bool 196 | 197 | function contains( 198 | haystack: multirange<anypoint>, 199 | needle: multirange<anypoint> 200 | ) -> bool 201 | 202 | function contains( 203 | haystack: multirange<anypoint>, 204 | needle: anypoint 205 | ) -> bool 206 | 207 | function overlaps( 208 | haystack: range<anypoint>, 209 | needle: range<anypoint> 210 | ) -> bool 211 | 212 | function overlaps( 213 | haystack: multirange<anypoint>, 214 | needle: multirange<anypoint> 215 | ) -> bool 216 | 217 | It should be noted that if ``contains(A, B) = true`` then ``overlaps(A, B) = 218 | true``. 219 | 220 | 221 | Operators 222 | --------- 223 | 224 | The operators ``+`` and ``-`` should be defined for various combinations of 225 | ``range`` and ``multirange``. The semantics of ``+`` is a "union", whereas the 226 | ``-`` allows excluding some values from an interval. Both of these operators 227 | produce a ``multirange`` in the general case. Sometimes the resulting 228 | ``multirange`` only has one ``range`` component. 229 | 230 | We can use ``*`` to perform interval intersection. 231 | 232 | In EdgeQL every type is *orderable*, so we must define ``<``, ``<=``, ``>``, 233 | and ``>=`` for interval types as well. We will use the same rule as Postgres: 234 | compare the lower bound first and only if it's the same, compare the upper 235 | bounds. 236 | 237 | 238 | Casting 239 | ------- 240 | 241 | ``range`` must have an implicit cast to ``multirange``. 242 | 243 | ``multirange`` must have an assignment cast to ``range``. 244 | 245 | 246 | Backwards Compatibility 247 | ======================= 248 | 249 | Much like ``array`` the ``range`` and ``multirange`` keywords can be 250 | unreserved and should not present any backwards compatibility issues. 251 | 252 | 253 | Security Implications 254 | ===================== 255 | 256 | There are no security implications. 257 | 258 | 259 | Rejected Alternative Ideas 260 | ========================== 261 | 262 | Constructors 263 | ------------ 264 | 265 | One option is to have a special literal to construct a range. A particualrly 266 | neat format can draw inspiration from the way ranges are indicated in math by 267 | using round and square brackets to indicate boundary inclusivity. E.g. 268 | ``[5..10)`` or ``(..-10]``. The upside is that this format is familiar from 269 | math. The downside is that this messes with any parentheses matching 270 | processing (auto-complete, highlighting, etc.). This is also a bit too similar 271 | to tuple and array literals, which can add a layer of confusion. 272 | 273 | Another option is to have matching parentheses and use an extra symbol to 274 | indicate inclusivity. E.g. ``(=5..10)`` or ``(..=-10)``. It seems that 275 | exclusive boundary is the more natural default for this approach, largely 276 | because it works naturally with omitted boundaries. The upside of this format 277 | is that it plays nice with parentheses matching. The downside is that it's 278 | hard to search for and it's not necessarily intuitive when seeing it. 279 | 280 | Generally, having function constructiors is useful in itself and does not 281 | prevent adding literals inthe future if such a need arises. 282 | 283 | 284 | Accessing range components 285 | -------------------------- 286 | 287 | We can have functions to access the 4 different components of a range value. 288 | E.g. ``get_range_component(range_val, 'lower')``. This parametrization does 289 | not seem very useful, though. 290 | 291 | Alternatively, we can have ``.`` accessors directly on the range values. E.g. 292 | ``range_val.upper`` or ``range_val.includes_lower``. This puts a significant 293 | cognitive burden on the developer when trying to determine what ``.`` means in 294 | any given path. 295 | 296 | We can have ``[]`` accessors directly on the range values. E.g. 297 | ``range_val['upper']`` or ``range_val['includes_lower']``. This format would 298 | be similar to accessing elements of a ``json`` object, which can be a source 299 | of confusion. 300 | 301 | 302 | Accessing multirange components 303 | ------------------------------- 304 | 305 | It may be useful to organize individual sub-ranges in a ``multirange`` like an 306 | ``array``. This way we can access them by indexing and even use ``len`` to 307 | determine how many sub-range pieces there are in a ``multirange``. This would 308 | require a fair bit of special processing in the compiler. Additionally it 309 | would place extra cognitive burden on the developer to distinguish a 310 | ``multirange`` from an ``array``. 311 | -------------------------------------------------------------------------------- /text/1043-ext-auth-otc.rst: -------------------------------------------------------------------------------- 1 | :: 2 | 3 | Status: Draft 4 | Type: Feature 5 | Created: 2025-06-17 6 | Authors: Scott Trinh <scott@geldata.com> 7 | 8 | ================================================== 9 | RFC 1043: Email-Based One-Time Code Authentication 10 | ================================================== 11 | 12 | This RFC proposes an extension to the ``ext::auth`` module to support one-time codes (OTCs) as a configurable method for the existing email-based passwordless authentication flow. The primary motivation is to solve common cross-device usability issues inherent in "magic link" style authentication. 13 | 14 | 15 | Motivation 16 | ========== 17 | 18 | PKCE friction 19 | ------------- 20 | 21 | The current ``ext::auth`` passwordless flow uses a "magic link" sent to a user's email. This flow is protected by a Proof Key for Code Exchange (PKCE) session that is initiated on the client device before the link is sent. This creates a significant usability problem in a common scenario: 22 | 23 | 1. A user starts the login process on their **desktop** (Device A). The client on Device A starts a PKCE session. 24 | 2. The user receives the magic link in their email and opens it on their **phone** (Device B). 25 | 3. The authentication attempt on Device B fails because it does not have the PKCE session information from Device A. 26 | 27 | This forces users to ensure they open the link on the same device where they started the process, which is often inconvenient and leads to a frustrating user experience. A one-time code, which can be easily transcribed between devices, elegantly solves this problem. 28 | 29 | Multi-factor authentication 30 | --------------------------- 31 | 32 | You might want to trigger a one-time code authentication flow as part of a multi-factor authentication flow. For example, you might want to send a code to a user's email as part of a two-factor authentication flow, or securing a sensitive action. This would be implemented by the developer as having /both/ a primary factor (like Email+Password or OAuth), and a secondary factor (like Magic Link or WebAuthn with code verification). 33 | 34 | .. code-block:: esdl 35 | 36 | type User { 37 | multi identities: ext::auth::Identity { 38 | constraint exclusive; 39 | }; 40 | }; 41 | 42 | .. code-block:: python 43 | 44 | async def something_sensitive(code: str, email: str): 45 | # Ensure this code is valid for this email address 46 | await user_service.verify_code(email, code) 47 | # Do something sensitive 48 | 49 | 50 | Invite codes 51 | ------------ 52 | 53 | You might have an authentication setup where an admin user will invite other users to join a team or organization. This would be implemented by the developer as having /both/ a primary factor (like Email+Password or OAuth), and a secondary factor (like Magic Link or WebAuthn with code verification), but the admin would only set up the secondary factor and when the user accepts the invite, they link a primary factor to the same user. 54 | 55 | .. code-block:: esdl 56 | 57 | type User { 58 | multi identities: ext::auth::Identity { 59 | constraint exclusive; 60 | }; 61 | }; 62 | 63 | And some pseudo-code: 64 | 65 | .. code-block:: python 66 | 67 | async def invite_user(email: str, team_id: str): 68 | # Create the user and associate it with the team 69 | await user_service.create_team_user(team_id, email) 70 | 71 | # Create a new Magic Link based factor directly for the user 72 | await user_service.register_magic_link(email) 73 | 74 | async def accept_invite(email: str, code: str, primary_identity_id: str): 75 | # Verify the code 76 | user = await user_service.verify_code(email, code) 77 | 78 | # Link the user's primary factor to the secondary factor's user 79 | await user_service.link_primary_factor(user.id, primary_identity_id) 80 | 81 | 82 | High-Level Proposal 83 | =================== 84 | 85 | We will introduce a new method for email+password email verification, magic link, and WebAuthn called ``Code``. This will be a configurable alternative to the existing ``Link`` method. 86 | 87 | The core of this proposal is to **decouple the PKCE session from the authentication initiation step**. When the ``Code`` method is enabled: 88 | 89 | 1. The initiation request (to send a code to an email) will be PKCE-agnostic. It simply triggers the sending of an email. 90 | 2. The verification step (submitting the code) will be responsible for initiating and completing the PKCE flow. 91 | 92 | This allows a user to easily transcribe the one-time code from their email (which might be opened on a different device) to the device where they initiated the login process. The PKCE session remains correctly scoped to the device performing the final verification, ensuring security while improving usability. 93 | 94 | 95 | Detailed Design 96 | =============== 97 | 98 | Configuration 99 | ------------- 100 | 101 | A new enum and configuration option will be added to the ``ext::auth`` module to allow developers to select the desired passwordless flow. 102 | 103 | For the email password provider, we will add a new property to the provider config to allow the developer to select the verification method. 104 | 105 | .. code-block:: esdl 106 | 107 | create scalar type ext::auth::VerificationMethod extending std::enum<Link, Code>; 108 | 109 | create type ext::auth::EmailPasswordProviderConfig 110 | extending ext::auth::ProviderConfig { 111 | # ... existing properties ... 112 | 113 | create required property verification_method: ext::auth::VerificationMethod { 114 | set default := ext::auth::VerificationMethod.Link; 115 | }; 116 | }; 117 | 118 | And for the magic link provider, we will add a new property to the provider config to allow the developer to select the verification method. 119 | 120 | .. code-block:: esdl 121 | 122 | create type ext::auth::MagicLinkProviderConfig 123 | extending ext::auth::ProviderConfig { 124 | # ... existing properties ... 125 | 126 | create required property verification_method: ext::auth::VerificationMethod { 127 | set default := ext::auth::VerificationMethod.Link; 128 | }; 129 | }; 130 | 131 | And for the WebAuthn provider, we will add a new property to the provider config to allow the developer to select the verification method. 132 | 133 | .. code-block:: esdl 134 | 135 | create type ext::auth::WebAuthnProviderConfig 136 | extending ext::auth::ProviderConfig { 137 | # ... existing properties ... 138 | 139 | create required property verification_method: ext::auth::VerificationMethod { 140 | set default := ext::auth::VerificationMethod.Link; 141 | }; 142 | }; 143 | 144 | The default value will be ``Link`` to ensure full backwards compatibility. 145 | 146 | Schema 147 | ------ 148 | 149 | To manage the state of an in-progress OTC authentication and track authentication attempts, two new types will be introduced. 150 | 151 | .. code-block:: esdl 152 | 153 | create type ext::auth::OneTimeCode { 154 | create required property code_hash: std::bytes { 155 | create constraint exclusive; 156 | create annotation std::description := 157 | "The securely hashed one-time code."; 158 | }; 159 | create required property expires_at: std::datetime { 160 | create annotation std::description := 161 | "The date and time when the code expires."; 162 | }; 163 | 164 | create required link factor: ext::auth::Factor { 165 | on target delete delete source; 166 | }; 167 | }; 168 | 169 | create scalar type ext::auth::AuthenticationAttemptType extending std::enum< 170 | SignIn, 171 | EmailVerification, 172 | PasswordReset, 173 | MagicLink, 174 | OneTimeCode 175 | >; 176 | 177 | create type ext::auth::AuthenticationAttempt extending ext::auth::Auditable { 178 | create required link factor: ext::auth::Factor { 179 | on target delete delete source; 180 | }; 181 | create required property attempt_type: ext::auth::AuthenticationAttemptType { 182 | create annotation std::description := 183 | "The type of authentication attempt being made."; 184 | }; 185 | create required property successful: std::bool { 186 | create annotation std::description := 187 | "Whether this authentication attempt was successful."; 188 | }; 189 | }; 190 | 191 | The ``OneTimeCode`` object will be created when the flow is initiated and deleted immediately upon successful verification. We can also delete all expired ``OneTimeCode`` objects when the server makes a verification attempt to avoid needing a separate cleanup job. That means you could have a situation where you've only created a single ``OneTimeCode`` object, it expires and the user never verifies it, and it never gets deleted, but is still invalid, so the only cost is the storage of the object itself. 192 | 193 | PKCE Flow and Authentication Ceremony 194 | ------------------------------------- 195 | 196 | The key innovation of this proposal is the adjustment of the PKCE flow. 197 | 198 | **Phase 1: Code Initiation (PKCE Agnostic)** 199 | 200 | The client sends a request to initiate the flow for an email address. This request does not require a PKCE session. The server generates a user-friendly code, stores its hash in a ``OneTimeCode`` record, and sends the code to the user's email. 201 | 202 | **Phase 2: Code Verification (PKCE Mandatory)** 203 | 204 | The client sends a request to verify the code, which **must** include a PKCE ``code_challenge``. The server validates the code against the stored hash. If successful, it completes the PKCE flow and issues the auth token. 205 | 206 | This decoupled flow enables two essential client-side patterns: 207 | 208 | Same-Device Flow (Session Reuse) 209 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 210 | 211 | 1. A user on Device A initiates the flow. The client immediately starts a PKCE session and stores the ``code_verifier``. 212 | 2. The client sends the initiation request. 213 | 3. The user enters the received code on Device A. 214 | 4. The client sends the verification request, including the ``code_challenge`` from the **existing** PKCE session. 215 | 216 | Cross-Device Flow (New Session) 217 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 218 | 219 | 1. A user on Device A initiates the flow. The client sends the request without needing a PKCE session. 220 | 2. The user receives the code and enters it into the client on Device B. 221 | 3. The client on Device B now starts a **new** PKCE session and immediately sends the verification request with its ``code_challenge``. 222 | 4. The authentication succeeds because the PKCE session is correctly scoped to Device B, which is performing the verification. 223 | 224 | 225 | Design Considerations and Rejected Ideas 226 | ======================================== 227 | 228 | * **A Separate ``CodeFactor``:** We considered introducing a new, distinct factor for OTC. This was rejected because this feature is fundamentally a different *method* for existing factors (e.g. Email+Password, Magic Link), not a new identity type. Reusing the existing factor simplifies the conceptual model and configuration. 229 | 230 | * **Separate HTTP Endpoints:** An initial idea was to create new endpoints like ``/send-code`` and ``/verify-code``. This was rejected in favor of augmenting the existing ``/verify`` and ``/magic-link/authenticate`` endpoints. A unified API is cleaner and makes the feature a true drop-in alternative. 231 | 232 | * **Client-Provided Codes:** We rejected any design that would allow a client to specify the code. The code must be generated and managed entirely server-side to prevent security vulnerabilities when the HTTP server for the extension is exposed to the public internet. 233 | 234 | 235 | Out of Scope 236 | ============ 237 | 238 | * **Time-Based One-Time Passwords (TOTP):** TOTP (e.g., from an authenticator app) relies on a long-lived shared secret and is a distinct authentication factor. It is considered out of scope for this RFC and may be addressed in a future proposal. 239 | 240 | * **Other Delivery Channels:** This RFC focuses exclusively on adding OTC to existing email-based factors. This allows us to leverage the existing SMTP and webhook infrastructure. Support for other channels like SMS or in-app notifications is out of scope, but can be implemented using the existing webhook infrastructure. 241 | 242 | 243 | Backwards Compatibility 244 | ======================= 245 | 246 | This proposal is fully backwards-compatible. The new functionality is opt-in via the ``verification_method`` configuration option, which defaults to ``Link``. Existing projects will continue to function without any changes. 247 | -------------------------------------------------------------------------------- /text/1001-edgedb-server-control.rst: -------------------------------------------------------------------------------- 1 | :: 2 | 3 | Status: Accepted 4 | Type: Feature 5 | Created: 2020-04-29 6 | RFC PR: `edgedb/rfcs#0007 <https://github.com/edgedb/rfcs/pull/7>`_ 7 | 8 | ================================================================= 9 | RFC 1001: CLI for installation and control of local EdgeDB server 10 | ================================================================= 11 | 12 | This RFC describes the design of the ``edgedb`` CLI subcommand for the 13 | purposes of installation, update and control of a local EdgeDB server. 14 | 15 | 16 | Implementation Revision 17 | ======================= 18 | 19 | `Feedback <https://github.com/edgedb/edgedb/issues/2647>`_ on the initial 20 | beta implementation of the RFC prompted a redesign of the ``edgedb server`` 21 | commands group, essentially splitting it into two groups of commands: 22 | ``edgedb server`` and ``edgedb instance``. Read more on that in 23 | `RFC 1006 <https://github.com/edgedb/rfcs/blob/master/text/1006-simplified-cli.rst>`_. 24 | 25 | 26 | Motivation 27 | ========== 28 | 29 | Currently, the tasks of installing, updating and running an EdgeDB server 30 | instances are entirely manual and vary a lot across the supported platforms. 31 | From the standpoint of local development this creates unnecessary friction 32 | and opens lots of possibilities for user error. The current state also 33 | necessitates a non-trivial amount of documentation that new users must read. 34 | 35 | By implementing the installation, update, and control logic into the ``edgedb`` 36 | CLI we can significantly simplify the process of getting started with EdgeDB 37 | without leaving the familiar development process: 38 | 39 | 1. Download the ``edgedb`` binary via the most convenient channel 40 | (`npm`, `pip`, `curl`). 41 | 2. Use the downloaded ``edgedb`` binary to either install and initialize 42 | a local server, or configure a remote server instance for development. 43 | 44 | 45 | Overview 46 | ======== 47 | 48 | The RFC proposes a new group of ``edgedb`` CLI commands under ``edgedb server`` 49 | prefix: 50 | 51 | * ``edgedb server list-versions`` -- list EdgeDB server versions available 52 | for installation; 53 | 54 | * ``edgedb server install`` -- install or update a specific version of the 55 | EdgeDB server on the local machine; 56 | 57 | * ``edgedb server uninstall`` -- uninstall a specific version of the 58 | EdgeDB server or all versions of EdgeDB from the local machine; 59 | 60 | * ``edgedb server init`` -- initialize a new EdgeDB server instance; 61 | 62 | * ``edgedb server start`` -- starts an EdgeDB server instance; 63 | 64 | * ``edgedb server status`` -- show the status of the local EdgeDB server; 65 | 66 | * ``edgedb server logs`` -- show the logs of the specified EdgeDB server 67 | instance; 68 | 69 | * ``edgedb server stop`` -- stop the given EdgeDB server instance; 70 | 71 | * ``edgedb server restart`` -- restart the given EdgeDB server instance; 72 | 73 | * ``edgedb server upgrade`` -- upgrade the specified EdgeDB server instance 74 | to the new major version. 75 | 76 | * ``edgedb server prune`` -- removes upgrade backups and other unused data. 77 | 78 | 79 | Design Considerations 80 | ===================== 81 | 82 | Instance names 83 | -------------- 84 | 85 | Most commands described below refer to EdgeDB server instances by name. 86 | The simplest interpretation is that the instance name is just the name 87 | of a data directory folder in a well-known location. For system instances 88 | (created with ``edgedb server start --system``) this would be 89 | directories under ``/var/lib/edgedb/data/``. For user instances, this 90 | would be directories under ``$XDG_DATA_HOME/edgedb/data/``. In both 91 | situations the base directory location for data should be configurable. 92 | 93 | The set of instance names is unique, and in situations where a system 94 | instance is created with the same name as an existing user instance, 95 | the user instance "masks" the system instance. ``edgedb server status`` 96 | should tell if an instance is system-wide or user-local. 97 | 98 | Interactive mode 99 | ---------------- 100 | 101 | Most commands described below offer an interactive wizard mode that can 102 | be selected by passing the ``-i`` or ``--interactive`` option to the command. 103 | Whenever a command, running in non-interactive mode, encounters a 104 | lack-of-input situation it should hint at the availability of the interactive 105 | mode. 106 | 107 | No ``--docker`` by default 108 | -------------------------- 109 | 110 | Using Docker by default introduces a non-trivial dependency on software that 111 | might not be available to the user, so this is an opt-in feature. 112 | 113 | 114 | edgedb server list-versions 115 | =========================== 116 | 117 | List EdgeDB server versions available for installation. 118 | 119 | Synopsis 120 | -------- 121 | 122 | ``edgedb server list-versions [options]`` 123 | 124 | Options 125 | ------- 126 | 127 | ``--installed-only`` 128 | only list the installed versions of the EdgeDB server. 129 | 130 | 131 | 132 | edgedb server install 133 | ===================== 134 | 135 | Downloads and installs a given EdgeDB server version 136 | (latest stable by default). 137 | 138 | Arguments 139 | --------- 140 | 141 | ``--version=<ver>`` 142 | specifies the major version of the server to install. 143 | 144 | ``--nightly`` 145 | if passed, the latest nightly build from the specified version channel 146 | is installed. 147 | 148 | ``--update`` 149 | if specified, ``edgedb install`` will only attempt to update the existing 150 | installations. 151 | 152 | ``--method={package|docker}`` 153 | Use specified installation method. ``package`` installs a native package on 154 | supported operating systems. While ``docker`` uses Docker instead of 155 | downloading and installing packages directly onto the user's system. The 156 | Docker daemon must be present and accessible by the user. 157 | 158 | 159 | Implementation 160 | -------------- 161 | 162 | By default, ``edgedb install`` will use system's package manager. If platform 163 | is unsupported error message will show other options, like installing it 164 | in Docker container if the latter is available on the system. 165 | 166 | 167 | edgedb server uninstall 168 | ======================= 169 | 170 | Uninstalls the specified version of EdgeDB. 171 | 172 | Synopsis 173 | -------- 174 | 175 | ``edgedb server uninstall [options]`` 176 | 177 | If there are multiple versions installed, either ``--all`` or 178 | ``--version`` or ``--unused`` is required. 179 | 180 | Options 181 | ------- 182 | 183 | ``--version=<ver>`` 184 | Specifies the version to uninstall. The specified server version must 185 | not be currently running. 186 | 187 | ``--all`` 188 | Uninstalls all versions of EdgeDB. 189 | 190 | ``--unused`` 191 | Uninstalls all versions of EdgeDB that are not used in any instance. 192 | 193 | 194 | edgedb server init 195 | ================== 196 | 197 | Initialize a new EdgeDB server instance with the specified name. 198 | 199 | Synopsis 200 | -------- 201 | 202 | ``edgedb server init [options] <name>`` 203 | 204 | Options 205 | ------- 206 | 207 | ``<name>`` 208 | The name of the EdgeDB instance. Must be unique. 209 | 210 | ``--version=<ver>`` 211 | Optionally specifies the server version to use. If not specified, 212 | the latest installed server version is used. 213 | 214 | ``--start-conf=auto|manual`` 215 | If set to ``auto`` (the default), the server will be started automatically 216 | on system boot. 217 | 218 | ``--port=<port-number>`` 219 | Optionally specifies the port number on which the server should listen. 220 | 221 | ``--system`` 222 | By default, ``edgedb server start`` runs the server in the user scope, 223 | if ``--system`` is specified, it is started as a system-wide service 224 | instead. 225 | 226 | ``--server-options -- <options>`` 227 | Specifies the ``edgedb-server`` command line options verbatim. 228 | Must be the last argument. 229 | 230 | 231 | 232 | edgedb server start 233 | =================== 234 | 235 | Starts an EdgeDB server instance with the specified name. 236 | 237 | Synopsis 238 | -------- 239 | 240 | ``edgedb server start [options] <name>`` 241 | 242 | Options 243 | ------- 244 | 245 | ``<name>`` 246 | The name of the EdgeDB instance. Must be unique. 247 | 248 | ``--server-options -- <options>`` 249 | Passes ``edgedb-server`` options verbatim. Must be the last argument. 250 | 251 | ``--foreground`` 252 | Run server in the foreground instead of running as a system service. 253 | 254 | 255 | edgedb server status 256 | ==================== 257 | 258 | Shows the status of the specified server instance or all instances. 259 | 260 | Synopsis 261 | -------- 262 | 263 | ``edgedb server status [options] [<name>]`` 264 | 265 | Options 266 | ------- 267 | 268 | ``<name>`` 269 | The name of the EdgeDB instance. If not specified status of the all 270 | instances is printed. 271 | 272 | 273 | Implementation 274 | -------------- 275 | 276 | The command outputs the state of the server instance 277 | (``running`` or ``stopped``), the port number it is configured to run on, 278 | the scope of the instance (system-wide or user-local), and the runtime under 279 | which the server is running (docker or native). 280 | 281 | 282 | edgedb server logs 283 | ================== 284 | 285 | Show the logs of the specified EdgeDB server instance. 286 | 287 | Synopsis 288 | -------- 289 | 290 | ``edgedb server logs [options] <name>`` 291 | 292 | Options 293 | ------- 294 | 295 | ``<name>`` 296 | The name of the EdgeDB instance. 297 | 298 | ``--tail <number>`` 299 | Show the last ``number`` of log entries. 300 | 301 | ``--follow`` 302 | Show the recent log entries and then continuously output new log entries 303 | as they are added to the log. 304 | 305 | 306 | edgedb server stop 307 | ================== 308 | 309 | Stops the specified EdgeDB server instance. 310 | 311 | Synopsis 312 | -------- 313 | 314 | ``edgedb server stop [options] <name>`` 315 | 316 | Options 317 | ------- 318 | 319 | ``<name>`` 320 | The name of the EdgeDB instance. 321 | 322 | ``--mode=<fast|graceful>`` 323 | The server restart mode. The ``fast`` mode (the default) does not wait 324 | for the clients to disconnect and forcibly terminates connections, all 325 | in-progress transactions are rolled back. The ``graceful`` mode waits 326 | for the clients to disconnect gracefully. 327 | 328 | 329 | edgedb server upgrade 330 | ===================== 331 | 332 | Upgrades the specified EdgeDB server instance to a given EdgeDB version. 333 | 334 | Synopsis 335 | -------- 336 | 337 | There are few modes of operation of this command: 338 | 339 | ``edgedb server upgrade`` 340 | Without arguments this command upgrades all instances which aren't running 341 | nightly EdgeDB to a latest minor version of the server. 342 | 343 | ``edgedb server upgrade <name> [--to-version=<ver>|--to-nightly]`` 344 | Upgrades specified instance to the specified major version of the server or 345 | to the latest nightly, by default upgrades to the latest stable. This only 346 | works for instances that initially aren't running nightly. 347 | 348 | ``edgedb server upgrade --nightly`` 349 | Upgrades all existing nightly instances to the latest EdgeDB nightly. 350 | 351 | Options 352 | ------- 353 | 354 | ``<name>`` 355 | The name of the EdgeDB instance. If omitted all stable instances will 356 | be upgraded to the latest minor version. (Or all nightly instances 357 | will be upgraded with ``--nightly``) 358 | 359 | ``--to-version`` 360 | Specifies the version of EdgeDB to upgrade to. If not specified, 361 | the latest available installed version is used. 362 | 363 | ``--to-nightly`` 364 | Specifies that the instance should be upgraded to the nightly version. 365 | 366 | ``--nightly`` 367 | Upgrade all instances currently running nightly to the latest nightly 368 | version (includes upgrades across major versions). 369 | 370 | ``--allow-downgrade`` 371 | Allow downgrading to an older version. Downgrades are prohibited by 372 | default. 373 | 374 | ``--revert`` 375 | Revert the upgrade if the original data directory has not been removed. 376 | 377 | Implementation 378 | -------------- 379 | 380 | For minor upgrade this command: 381 | 382 | * stops all running instances 383 | * upgrades the package 384 | * starts all instances 385 | 386 | For any other upgrade: 387 | 388 | * dumps everything to a directory ``<instance-name>.dump`` 389 | using ``edgedb dump --all`` 390 | * upgrades needed packages 391 | * renames old data directory to ``<instance-name>.backup`` 392 | * inits new server and restores data via ``edgedb restore --all``` 393 | 394 | This keeps the original data directory in case ``--revert`` is requested. 395 | 396 | 397 | edgedb server restart 398 | ===================== 399 | 400 | Restart the specified EdgeDB server instance. 401 | 402 | Synopsis 403 | -------- 404 | 405 | ``edgedb server restart [options] <name>`` 406 | 407 | Options 408 | ------- 409 | 410 | ``<name>`` 411 | The name of the EdgeDB instance. 412 | 413 | ``--mode=<fast|graceful>`` 414 | The server restart mode. The ``fast`` mode (the default) does not wait 415 | for the clients to disconnect and forcibly terminates connections, all 416 | in-progress transactions are rolled back. The ``graceful`` mode waits 417 | for the clients to disconnect gracefully. 418 | 419 | 420 | edgedb server prune 421 | =================== 422 | 423 | Removes upgrade backups. 424 | 425 | Synopsis 426 | -------- 427 | 428 | ``edgedb server prune [options]`` 429 | 430 | Options 431 | ------- 432 | 433 | ``--upgrade-backups`` 434 | Prune upgrade backups. After this ``edgedb server upgrade --revert`` 435 | will be impossible. 436 | -------------------------------------------------------------------------------- /text/1023-splats.rst: -------------------------------------------------------------------------------- 1 | :: 2 | 3 | Status: Accepted 4 | Type: Feature 5 | Created: 2023-01-25 6 | Authors: Yury Selivanov <yury@edgedb.com> 7 | 8 | 9 | ============================== 10 | RFC 1023: Adding Splats Syntax 11 | ============================== 12 | 13 | Abstract 14 | ======== 15 | 16 | We propose to add a splats syntax to EdgeQL. The closest equivalents to this 17 | proposal are fragments in GraphQL and `SELECT *` in SQL. 18 | 19 | Splats in ``select`` commands and in free object types will allow for better 20 | REPL user experience and will improve readability of complex queries and type 21 | expressions. 22 | 23 | 24 | Splats in select shapes 25 | ======================= 26 | 27 | EdgeQL *shapes* is a syntax construct used to specify what exact properties, 28 | links, and ad-hoc computed data should be fetched by the query. 29 | 30 | This proposal assumes the following schema for all of its example code:: 31 | 32 | abstract type Person { 33 | required property name -> str { constraint exclusive }; 34 | } 35 | 36 | type Hero extending Person { 37 | property secret_identity -> str; 38 | multi link villains := .<nemesis[is Villain]; 39 | } 40 | 41 | type Villain extending Person { 42 | link nemesis -> Hero; 43 | } 44 | 45 | With this schema in mind, here's an example of the EdgeQL's shape syntax:: 46 | 47 | select Person { 48 | name, # propety name 49 | 50 | name_upper := str_upper(.name), # computed property 51 | 52 | villains: { # fetch data traversing the "villains" link 53 | name 54 | }, 55 | 56 | [is Hero].secret_identity, # fetch `secret_identity` property 57 | # for all `Person` objects that are also 58 | # instances of `Hero` 59 | } 60 | 61 | We propose to enhance the shape construct with the following syntax: 62 | 63 | * ``*``: extend the shape with all properties (not links!) defined on the type, 64 | including the inherited ones. 65 | 66 | A simple example:: 67 | 68 | select Person { 69 | * # Will expand into "id, name". 70 | }; 71 | 72 | An example of ``*`` including inherited pointers:: 73 | 74 | select Hero { 75 | * # Will expand into "id, name, secret_identity". 76 | }; 77 | 78 | An example of ``*`` including pointers that were already specified:: 79 | 80 | select Hero { 81 | name := 'try me!', 82 | *, # Will expand into "id, name, secret_identity", 83 | # where `name` will be coming from the `name := 'try me!'` 84 | # computed. 85 | }; 86 | 87 | An complex example illustrating using ``*`` in nested shapes:: 88 | 89 | select Hero { 90 | *, # Will expand into "id, name, secret_identity". 91 | 92 | villains: { 93 | * # Will expand into "id, name" 94 | # (the `Villain` type doesn't have the `secret_identity` property). 95 | 96 | nemesis: { 97 | * # Will expand into "id, name, secret_identity". 98 | } 99 | } 100 | }; 101 | 102 | Another example to demonstrate that splats work at a "symbolic" 103 | level, expanding the actual shape to be selected, yet ignoring how 104 | and where things are actually defined:: 105 | 106 | with 107 | CapHero := Hero { name := str_upper(.name) } 108 | select 109 | ( 110 | CapHero { * }, # will select a shape with upper-cased names 111 | CapHero { Hero.* } # will *also* select a share with upper-cased names 112 | ) 113 | 114 | * ``**``: extend the shape with all properties and *links* defined on the type, 115 | including the inherited ones. Linked shapes will be defined with the 116 | ``*`` splat. 117 | 118 | A simple example:: 119 | 120 | select Person { 121 | ** # Will expand into "id, name". 122 | # The `Person` type doesn't have any links defined on it. 123 | }; 124 | 125 | An example of ``**`` including inherited pointers and links:: 126 | 127 | select Hero { 128 | ** # Will expand into: 129 | # { 130 | # id, 131 | # name, 132 | # secret_identity, 133 | # villains: { * } 134 | # } 135 | # 136 | # which will in turn expand into: 137 | # { 138 | # id, 139 | # name, 140 | # secret_identity, 141 | # villains: { id, name } 142 | # } 143 | }; 144 | 145 | It's possible use ``**`` and redefine the pointers it expands into:: 146 | 147 | select Hero { 148 | **, 149 | villains: { # Use `**` to auto-include all linked types 150 | name, # into the shapes, but define the `villains` 151 | level := 80 # link to include just the `name` property 152 | } # and the `level` computed. 153 | }; 154 | 155 | Note that ``**`` does not expand the ``__type__`` link. 156 | 157 | * ``<type expression>.*`` and ``<type expression>.**``: extend the shape with 158 | all properties/links reachable from the computed type of ``type expression``. 159 | 160 | A trivial example when the type expression is a reference to the base type:: 161 | 162 | select Hero { 163 | Person.* # Will expand into "id, name". 164 | }; 165 | 166 | A more complicated type expression using ``*``:: 167 | 168 | select Hero { 169 | (Hero | Villain).* # Would expand to "id, name". 170 | } 171 | 172 | A more complicated type expression using ``**`` (the query wouldn't 173 | compile but we use it nevertheless to illustrate the proposed behavior 174 | of ``**``):: 175 | 176 | select Hero { 177 | (Hero & Villain).** # Would expand into 178 | # { 179 | # id, 180 | # name, 181 | # secret_identity, 182 | # villains: { * }, 183 | # nemesis: { * } 184 | # } 185 | } 186 | 187 | * ``[is ...].*`` and ``[is ...].**``: polymorphic variants for the above 188 | splat syntaxes. 189 | 190 | An example of ``*``:: 191 | 192 | select Person { 193 | [is Hero].* # Expands into 194 | # { 195 | # [is Hero].id, 196 | # [is Hero].name, 197 | # [is Hero].secret_identity, 198 | # } 199 | } 200 | 201 | An example of ``**``:: 202 | 203 | select Person { 204 | [is Hero].** # Expands into 205 | # { 206 | # [is Hero].id, 207 | # [is Hero].name, 208 | # [is Hero].secret_identity, 209 | # [is Hero].villains: { * }, 210 | # } 211 | } 212 | 213 | Splats in free object types 214 | =========================== 215 | 216 | This section builds on the concepts introduced in 217 | `RFC 1022 - Typing free objects & simplifying SDL syntax <./1022-freetypes.rst>`_. 218 | 219 | Allowing splats to be used in the EdgeQL's type sub-language (particularly, 220 | allowing them to be used in free object type declarations) will 221 | lead to more concise function declarations and type casts. 222 | 223 | We propose to extend the free shape type syntax with the following constructs: 224 | 225 | * `<type expression>.*`: include all properties from the computed type of 226 | ``type expression`` to the final free object's type. Example:: 227 | 228 | function validate(data: { 229 | Person.* 230 | }) -> bool using (...) 231 | 232 | # `data` parameter will accept free objects that have all properties 233 | # declared in the Person type (retaining their cardinality bounds & types) 234 | 235 | An example of a more complicated type expression:: 236 | 237 | function validate(data: { 238 | (Hero | Villain).*, # will expand into: 239 | # { required id: uuid, required name: str } 240 | 241 | foo: str, # add a "foo" property to this free object type 242 | }) -> bool using (...) 243 | 244 | * `<modifier> <type expression>.*`: include all properties from the computed 245 | type expression overriding cardinality. 246 | 247 | An example of including all properties from another type but making 248 | them all optional: 249 | 250 | function validate(data: { 251 | optional Person.* 252 | }) -> bool using (...) 253 | 254 | # `data` parameter will accept free objects that have all properties 255 | # declared in the Person type (making them all optional) 256 | 257 | An example of making all expanded fields required: 258 | 259 | function validate(data: { 260 | required Hero.* # will expand into: 261 | # { 262 | # required id: uuid, 263 | # required name: str, 264 | # required secret_identity: str 265 | # } 266 | }) -> bool using (...) 267 | 268 | 269 | Rejected ideas 270 | ============== 271 | 272 | Use prefix/postfix ``...`` for splats 273 | ------------------------------------- 274 | 275 | The prefix ``...`` operator, available in JavaScript (the spread operator) 276 | and in GraphQL (fragments), seemed like a viable alternative to ``*``. 277 | 278 | We decided against using it in EdgeQL for the following reasons: 279 | 280 | * With the existing EdgeQL grammar in mind, ``...[is Hero]`` splat would 281 | look to the reader as if ``[is Hero]`` is applied to the result of the splat. 282 | E.g.:: 283 | 284 | select Person { 285 | ...[is Hero] 286 | } 287 | 288 | Would be interpreted as:: 289 | 290 | select { 291 | id[is Hero], 292 | name[is Hero] 293 | } 294 | 295 | which is nonsense. 296 | 297 | * ``...`` as a prefix operator would make type expressions syntax look 298 | inconsistent when a splat is used next to a direct field reference. 299 | 300 | Compare: 301 | 302 | +----------------------------------+-----------------------------------+ 303 | |:: | :: | 304 | | | | 305 | | { | { | 306 | | Foo.prop, | Foo.prop, | 307 | | Bar.* | ...Bar | 308 | | } | } | 309 | +----------------------------------+-----------------------------------+ 310 | 311 | * With ``...`` as a postfix operator implementing the proposed ``*`` syntax it 312 | is unclear how we would design its ``**`` variant. Using postfix ``......`` 313 | operator is obviously not a viable option. 314 | 315 | 316 | Make ``*`` expand to both links and properties 317 | ---------------------------------------------- 318 | 319 | * Users will inevitably use splats in their application code (i.e. not just in 320 | REPL) and selecting all links can make queries slower. Besides, selecting all 321 | properties is typically a more common need than selecting all properties 322 | and all linked data. 323 | 324 | * We already have splats in our TypeScript query builder API and the current 325 | implementation only expands ``*`` into list of properties. 326 | 327 | 328 | Field exclusion syntax 329 | ---------------------- 330 | 331 | Field exclusion can be useful to splat every property from a type except a 332 | few specific ones. For example, an earlier revision of this RFC was proposing 333 | to use the ``never`` type for this purpose:: 334 | 335 | function validate(data: { 336 | required Hero.*, 337 | id: never, 338 | 339 | # will expand into: 340 | # { 341 | # required name: str, 342 | # required secret_identity: str 343 | # } 344 | }) -> bool using (...) 345 | 346 | However, it was pointed out that in the context of EdgeQL using ``never`` like 347 | this can be problematic, as it would propagate through the query typing 348 | converting everything to ``never``. Another alternative would be to use an 349 | unary ``-`` operator, as in:: 350 | 351 | function validate(data: { 352 | required Hero.*, 353 | -id, 354 | 355 | # will expand into: 356 | # { 357 | # required name: str, 358 | # required secret_identity: str 359 | # } 360 | }) -> bool using (...) 361 | 362 | the downside of that approach is that the semantics of ``-`` in this context 363 | is not entirely clear. Ultimately it was decided that designing the field 364 | exclusion syntax, while possible, is out of scope of this proposal. 365 | 366 | Allow splats to be used on values 367 | --------------------------------- 368 | 369 | The following query would not compile:: 370 | 371 | with 372 | h := (select Hero { computed := 42 }) 373 | select 374 | Hero { 375 | h.* # compile-time error! 376 | } 377 | 378 | the query would fail with a compile-time error suggesting that ``.*`` can 379 | only be used on a *type*. A simple way to fix the query would be to 380 | use the ``typeof`` operator:: 381 | 382 | with 383 | h := (select Hero { computed := 42 }) 384 | select 385 | Hero { 386 | (typeof h).* # The shape will expand to 387 | # { 388 | # id, 389 | # name, 390 | # secret_identity, 391 | # computed, 392 | # } 393 | } 394 | 395 | 396 | Backwards compatibility 397 | ======================= 398 | 399 | The proposal is fully backwards compatible. 400 | 401 | 402 | Implementation plan 403 | =================== 404 | 405 | The proposal can be implemented in stages. E.g. EdgeDB version 3.0 will have 406 | the basic ``*`` and ``**`` operators supported in shapes, while EdgeDB 4.0 407 | or later can have the proposed type language extensions implemented. 408 | -------------------------------------------------------------------------------- /text/1013-datetime-arithmetic.rst: -------------------------------------------------------------------------------- 1 | :: 2 | 3 | Status: Accepted 4 | Type: Feature 5 | Created: 2022-06-06 6 | Authors: Victor Petrovykh <victor@edgedb.com> 7 | RFC PR: `edgedb/rfcs#0059 <https://github.com/edgedb/rfcs/pull/59>`_ 8 | 9 | ============================== 10 | RFC 1013: Date/time arithmetic 11 | ============================== 12 | 13 | This RFC proposes defining subtraction (``-`` operator) for all of the 14 | "local" variants of date/time scalars. It also proposes adding a 15 | ``date_duration`` to represent intervals with whole number of days and without 16 | any smaller components. This is a conceptual change in how ``local_date`` is 17 | treated w.r.t. arithmetic, being now more analogous to integers. Finally, we 18 | need to add functions for normalizing, truncating and producing various 19 | flavours of duration as well as extracting its various components. 20 | 21 | 22 | Motivation 23 | ========== 24 | 25 | At the time of writing all local date/time scalars lack the operator for 26 | finding the difference (``-``) between values of the same type. It is not 27 | possible to simply subtract a ``local_datetime`` value from another 28 | ``local_datetime``, ``local_date`` from ``local_date``, and ``local_time`` 29 | from ``local_time``. 30 | 31 | We have 2 different scalar type representing a time delta: ``duration`` and 32 | ``relative_duration``. The ``duration`` scalar is meant to represent an 33 | absolute value of some period of time in the real world (e.g. a specific 34 | number of seconds or some other unambiguous equivalent units). The 35 | ``relative_duration`` can represent time intervals in unambiguous units 36 | (seconds, minutes, hours) as well as in ambiguous, but useful calendar units 37 | such as *days*, *weeks*, *months*, *years*, etc. A *month* may not have a 38 | well-defined number of seconds, yet it is a well-defined time period w.r.t. a 39 | specific calendar system and can be used in that context. 40 | 41 | It seems that the result of subtracting one ``local_datetime`` value from 42 | another should result in a ``relative_duration``. It is analogous to the same 43 | operation on ``datetime`` resulting in ``duration``. Since ``datetime`` 44 | represents specific points in actual reality we can define an absolute value 45 | of ``duration`` between them. In contrast, ``local_datetime`` does not contain 46 | enough information to pin it to any specific point in reality and so at best 47 | only a ``relative_duration`` can be given as the result of a subtraction 48 | operation. It's an operation that ignores the potential nuance of DST 49 | affecting the time difference. 50 | 51 | In the same vein as ``local_datetime``, the subtraction of one ``local_time`` 52 | value from another should also produce ``relative_duration``. The 53 | ``local_time`` represents a certain value on a clock. For the purpose of this 54 | RFC it may be easier to imagine the clock as sort of "broken", not able to 55 | tick on its own, but still able to be adjusted by hand. The questions of 56 | arithmetic on such a clock are not about real passage of time, but rather 57 | about the amount of adjustment the clock requires to change the value from one 58 | to another. For example, imagine two snapshots of a clock: one at midnight, 59 | another at 4 AM. The time difference between these clock snapshots is 4 hours. 60 | However we have no way of knowing whether it would be 4, 28 or even 3 or 5 61 | hours (if a DST adjustment happened) between these two states of an accurate 62 | clock in the real world. So despite the fact that ``local_time`` difference 63 | seems to operate entirely in the range of hours/minutes/seconds it is 64 | inherently ambiguous and relative, rather than absolute like ``duration``. 65 | 66 | The ``local_date`` type represents a calendar and by analogy with the other 67 | two local scalar types it may seem that ``relative_duration`` is the natural 68 | unit for measuring the difference between two dates. However, for practical 69 | reasons, it seems that it may be more appropriate to treat ``local_date`` as 70 | being more akin to a special kind of integer (counting the days) whereas 71 | ``local_datetime`` is akin to a float. So the difference between two 72 | ``local_date`` values would be some interger-like duration representing the 73 | number of days. It is unambiguous an often what is practicaly useful. We wll 74 | call this new scalar type ``date_duration`` and it will be like 75 | ``relative_duration``, but without the elements smaller than a day. 76 | 77 | Working with ``relative_duration`` sometimes requires converting between hours 78 | and days or between days and months. We will need to expose the ``justify`` 79 | and ``truncate`` functionality to achieve this. 80 | 81 | Finally, even though relative duration measured in days is arguably more 82 | accurate for many smaller time periods, it can be better to produce a relative 83 | duration measured accurately in larger units for longer time periods and so we 84 | need to expose ``age`` from Postgres for this purpose, possibly under a 85 | different name. 86 | 87 | 88 | Specification 89 | ============= 90 | 91 | The ``local_datetime`` will introduce the following subtraction operator:: 92 | 93 | CREATE INFIX OPERATOR 94 | std::`-` ( 95 | l: cal::local_datetime, 96 | r: cal::local_datetime 97 | ) -> cal::relative_duration { 98 | USING SQL $$ 99 | SELECT ("l" - "r")::edgedb.relative_duration_t 100 | $$; 101 | }; 102 | 103 | The result is expected to be a ``relative_duration`` in days and 104 | hours/minutes/seconds. This is the most precise format as ``local_datetime`` 105 | cannot account for DST and so 1 day and 24 hours are always equivalent 106 | durations in this context. 107 | 108 | The ``local_time`` will introduce the following subtraction operator:: 109 | 110 | CREATE INFIX OPERATOR 111 | std::`-` ( 112 | l: cal::local_time, 113 | r: cal::local_time 114 | ) -> cal::relative_duration { 115 | USING SQL $$ 116 | SELECT ("l" - "r")::edgedb.relative_duration_t 117 | $$; 118 | }; 119 | 120 | The result is expected to be a ``relative_duration`` in the range [-24 hrs, 24 121 | hrs]. 122 | 123 | Introduce ``date_duration`` scalar that is "whole days" version of 124 | ``relative_duration``. Much like integers are not related to floats, 125 | ``date_duration`` is not directly a subtype of ``relative_duration``. Instead 126 | they have the following casts between them:: 127 | 128 | CREATE CAST FROM cal::date_duration TO cal::relative_duration { 129 | USING SQL CAST; 130 | ALLOW IMPLICIT; 131 | }; 132 | 133 | CREATE CAST FROM cal::relative_duration TO cal::date_duration { 134 | USING SQL CAST; 135 | }; 136 | 137 | Similarly, we need an implicit cast from ``local_date`` to 138 | ``local_datetime``:: 139 | 140 | CREATE CAST FROM cal::local_date TO cal::local_datetime { 141 | USING SQL CAST; 142 | ALLOW IMPLICIT; 143 | }; 144 | 145 | The above is analogous to integers implicitly casting into floats. 146 | 147 | The ``local_date`` will introduce the following subtraction operator:: 148 | 149 | CREATE INFIX OPERATOR 150 | std::`-` ( 151 | l: cal::local_date, 152 | r: cal::local_date 153 | ) -> cal::date_duration { 154 | USING SQL $$ 155 | SELECT ("l" - "r")::edgedb.date_duration_t 156 | $$; 157 | }; 158 | 159 | The result is expected to be the number of days. 160 | 161 | The ``+`` operators for ``local_date`` should produce ``local_date`` results 162 | only if the other operand is ``date_duration``, otherwise ``local_datetime`` 163 | should be produced, analogous to ``int64 + int64 = int64``, but ``int64 + 164 | float64 = float64``. So we defined these operators as follows:: 165 | 166 | CREATE INFIX OPERATOR 167 | std::`+` (l: cal::local_date, r: cal::date_duration) -> cal::local_date 168 | { 169 | USING SQL $$ 170 | SELECT ("l" + "r")::edgedb.date_t 171 | $$; 172 | }; 173 | 174 | CREATE INFIX OPERATOR 175 | std::`+` (l: cal::local_date, r: std::duration) -> cal::local_datetime 176 | { 177 | USING SQL $$ 178 | SELECT ("l" + "r")::edgedb.timestamp_t 179 | $$; 180 | }; 181 | 182 | CREATE INFIX OPERATOR 183 | std::`+` ( 184 | l: cal::local_date, r: cal::relative_duration 185 | ) -> cal::local_datetime 186 | { 187 | USING SQL $$ 188 | SELECT ("l" + "r")::edgedb.timestamp_t 189 | $$; 190 | }; 191 | 192 | 193 | Duration functions 194 | ------------------ 195 | 196 | We also will introduce normalization functions for ``relative_duration``:: 197 | 198 | CREATE FUNCTION 199 | cal::duration_normalize_hours(dur: cal::relative_duration) 200 | -> cal::relative_duration 201 | { 202 | USING SQL FUNCTION 'justify_hours'; 203 | }; 204 | 205 | CREATE FUNCTION 206 | cal::duration_normalize_days(dur: cal::relative_duration) 207 | -> cal::relative_duration 208 | { 209 | USING SQL FUNCTION 'justify_days'; 210 | }; 211 | 212 | CREATE FUNCTION 213 | cal::duration_normalize_days(dur: cal::date_duration) 214 | -> cal::date_duration 215 | { 216 | USING SQL FUNCTION 'justify_days'; 217 | }; 218 | 219 | The ``duration_normalize_hours`` converts 24 hrs chunks into days. 220 | The ``duration_normalize_days`` converts days into months assuming that 1 month = 30 days. 221 | 222 | Notice that for ``relative_duration`` the assumtion that 1 day is always 24 223 | hours holds true because it represents the time adjustment necessary on some 224 | clock rather than real time (like ``duration``). Therefore the 225 | ``duration_normalize_hours`` transformation is technically lossless and could 226 | safely be reversed. However, not every month is 30 days and so the 227 | ``duration_normalize_days`` and ``duration_normalize`` are 228 | potentially lossy transformations aiming to produce an approximately 229 | equivalent ``relative_duration`` using months, years, etc. so care must be 230 | taken when using these conversions. 231 | 232 | Only ``duration_normalize_days`` has an overloaded version to accept 233 | ``date_duration`` and return the same type. There are no hours to normalize 234 | for ``date_duration`` and thus the other two normalization functions are 235 | unnecessary for ``date_duration``. 236 | 237 | We also need a function for extracting various ``duration`` and 238 | ``relative_duration`` components:: 239 | 240 | CREATE FUNCTION 241 | std::duration_get(dur: cal::relative_duration, el: std::str) -> std::float64 242 | { 243 | ... 244 | }; 245 | 246 | CREATE FUNCTION 247 | std::duration_get(dur: std::duration, el: std::str) -> std::float64 248 | { 249 | ... 250 | }; 251 | 252 | The components avaialable for extraction from ``relative_duration`` are: 253 | *millenium*, *century*, *decade*, *year*, *quarter*, *month*, *day*, *hour*, 254 | *minutes*, *seconds*, *milliseconds*, *microseconds*, and *totalseconds*. The 255 | *totalseconds* converts a ``relative_duration`` to seconds represented as a 256 | ``float64`` value. Components greater than *hour* are not available for 257 | ``duration`` and will produce an error if there's an attempt to extract them. 258 | 259 | In addition to extraction function we also introduce a truncation function for 260 | ``relative_duration``. We will overload already existing ``duration_truncate`` 261 | to also accept ``relative_duration`` input and extend the list of truncated 262 | precision to include components greate than *hour*:: 263 | 264 | CREATE FUNCTION 265 | std::duration_truncate( 266 | dt: cal::relative_duration, 267 | unit: std::str 268 | ) -> cal::relative_duration 269 | { 270 | ... 271 | }; 272 | 273 | We expose the ``age`` functionality as ``relative_delta``. Basically this is a 274 | counterpart to ``-`` operator, but performed symbolically and producing an 275 | accurate result for the specific inputs:: 276 | 277 | CREATE FUNCTION 278 | cal::relative_delta( 279 | l: cal::local_date, 280 | r: cal::local_date 281 | ) -> cal::date_duration 282 | { 283 | SET force_return_cast := true; 284 | USING SQL FUNCTION 'age'; 285 | }; 286 | 287 | CREATE FUNCTION 288 | cal::relative_delta( 289 | l: cal::local_datetime, 290 | r: cal::local_datetime 291 | ) -> cal::relative_duration 292 | { 293 | SET force_return_cast := true; 294 | USING SQL FUNCTION 'age'; 295 | }; 296 | 297 | 298 | 299 | Backwards Compatibility 300 | ======================= 301 | 302 | There's a change in ``+`` arithmetic. The addition of ``local_date`` and 303 | ``duration`` or ``relative_duration`` now results in ``local_datetime`` 304 | instead of ``local_date``. This is a backwards-incompatible change as 305 | ``local_datetime`` cannot be used instead of ``local_date`` without an 306 | explicit cast. 307 | 308 | It is also desirable to later deprecate arithmetic operators that allow 309 | interactions between the "local" date/time types and absolute ``duration`` as 310 | that is not well-defined. 311 | 312 | 313 | Security Implications 314 | ===================== 315 | 316 | There are no security implications. 317 | 318 | 319 | Rejected Alternative Ideas 320 | ========================== 321 | 322 | We specifically rejected the option of ``-`` operating on two ``local_date`` 323 | values producing a ``relative_duration`` by default. In practice, this 324 | operation would produce ``relative_duration`` expressed using exclusively days 325 | anyways. The main difference then would be that in order to use the result of 326 | this operation one would need to use ``<int64>relative_duration_get(dur, 327 | 'days')``. 328 | 329 | We also rejected the notion that subtracting one ``local_date`` from another 330 | should produce an integer, like it does in Postgres. Basically, this seems to 331 | break the symmetry in ``-`` and ``+`` arithemtic for date/time types, while 332 | not addressing future use of ``date_duration`` as a "step" in 333 | ``generate_series`` type of scenario for ``local_date``, with the step being a 334 | "month" or some other larger unit:: 335 | 336 | CREATE INFIX OPERATOR 337 | std::`-` ( 338 | l: cal::local_date, 339 | r: cal::local_date 340 | ) -> int64 { 341 | USING SQL $$ 342 | SELECT ("l" - "r")::int8 343 | $$; 344 | }; 345 | 346 | By analogy with ``datetime_get`` being able to work with either ``datetime`` 347 | and ``local_datetime`` it makes sense to use ``duration_get`` (instead of 348 | ``relative_duration_get``) to operate on both ``duration`` and 349 | ``relative_duration``. 350 | 351 | We decided against having a convenience function ``duration_normalize``, 352 | because the 2 funcitons it combines have different effect of prcision of the 353 | result:: 354 | 355 | CREATE FUNCTION 356 | cal::duration_normalize(dur: cal::relative_duration) 357 | -> cal::relative_duration 358 | { 359 | USING SQL FUNCTION 'justify_interval'; 360 | }; 361 | -------------------------------------------------------------------------------- /text/1005-edgedb-project.rst: -------------------------------------------------------------------------------- 1 | :: 2 | 3 | Status: Accepted 4 | Type: Feature 5 | Created: 2021-04-02 6 | RFC PR: `edgedb/rfcs#30 <https://github.com/edgedb/rfcs/pull/30>`_ 7 | 8 | ============================================================= 9 | RFC 1005: CLI and conventions for local projects using EdgeDB 10 | ============================================================= 11 | 12 | This RFC describes the design of the ``edgedb project`` CLI commands for 13 | the purposes of initialization of a software project that uses EdgeDB as 14 | well as the associated conventions and changes to the language bindings. 15 | 16 | 17 | Motivation 18 | ========== 19 | 20 | `RFC 1001 <1001-edgedb-server-control.rst>`_ introduced the concept of 21 | EdgedDB *instances* and a straightforward way of managing them with 22 | edgedb-cli. The process of addressing an EdgeDB server has thus been 23 | simplified to passing a ``-I`` argument to the CLI or passing it as the 24 | argument to a ``connect()`` call in bindings. 25 | 26 | However, even with ``edgedb server`` a developer needs to take multiple 27 | steps to start a new project using EdgeDB: 28 | 29 | 1. Install the desired EdgeDB version (``edgedb server install``) 30 | 2. Initialize an EdgeDB instance for the project (``edgedb server init myapp``) 31 | 3. Create the ``dbschema/`` directory and the initial schema file. 32 | 33 | When starting work on a cloned project, the steps are similar: 34 | 35 | 1. Install the desired EdgeDB version (``edgedb server install``) 36 | 2. Initialize an EdgeDB instance for the project (``edgedb server init myapp``) 37 | 3. Apply migrations (``edgedb -I myapp migrate``). 38 | 39 | Since most projects using EdgeDB are expected to follow the conventions 40 | established by the reasonable defaults of the EdgeDB CLI it makes sense to 41 | automate the above operations to enable the following benefits: 42 | 43 | 1. Make it so that the project is ready to go after a single command 44 | (``edgedb project init``). 45 | 2. Allow an EdgeDB instance to be linked with a project directory to obviate 46 | the need to specify an instance name explicitly when running CLI commands 47 | or connecting with language bindings. 48 | 3. Allow specifying certain EdgeDB-specific metadata per project, such as 49 | the minimum required EdgeDB version. 50 | 51 | 52 | Overview 53 | ======== 54 | 55 | In this RFC we propose a new group of ``edgedb`` CLI commands under 56 | the ``edgedb project`` prefix: 57 | 58 | * ``edgedb project init`` -- populate a new project or initialize an existing 59 | cloned object; 60 | 61 | * ``edgedb project unlink`` -- remove association with and optionally destroy 62 | the linked EdgeDB instance; 63 | 64 | * ``edgedb project status`` -- show EdgeDB-related information about a 65 | project, such as the associated instance name; 66 | 67 | * ``edgedb project list`` -- list all known EdgeDB projects for the current 68 | user. 69 | 70 | 71 | Design 72 | ====== 73 | 74 | Project file structure 75 | ---------------------- 76 | 77 | `RFC 1000 <1000-migrations.rst>`_ defined the top-level ``dbschema/`` directory 78 | as the default location of EdgeDB SDL schema, and ``dbschema/migrations`` as 79 | the default expected location of the files containing schema migration scripts. 80 | To mark a directory as an EdgeDB project, an ``edgedb.toml`` file should be 81 | present in the root directory. The file can be empty or contain the following 82 | TOML:: 83 | 84 | [edgedb] 85 | server-version = <semver-range> 86 | schema-directory = <schema-dir> 87 | 88 | Here, the ``server-version`` attribute specifies a SemVer range of acceptable 89 | server versions. Most frequently this would be in the form of a minimum 90 | required version. The ``schema-directory`` attribute specifies the location 91 | of EdgeDB schema directory relative to ``edgedb.toml``. Defaults to 92 | ``dbschema``. 93 | 94 | Project detection 95 | ----------------- 96 | 97 | To detect whether an executable or a script is executed in a project context, 98 | it should look for ``edgedb.toml`` in the current working directory 99 | and then the parent directories in sequence. 100 | 101 | Nested projects 102 | --------------- 103 | 104 | Nesting EdgeDB projects is not allowed to avoid confusion. 105 | 106 | Project association 107 | ------------------- 108 | 109 | In order to remove the need to specify an EdgeDB instance explicitly when 110 | running the CLI commands or connecting via language bindings, we propose 111 | to keep a mapping of absolute project paths to instance names in a well-known 112 | location. The mapping will be updated by the ``edgedb project init`` command 113 | and will be interpreted by language bindings to obtain target instance address 114 | and credentials where no explicit connection configuration is specified. 115 | 116 | The project -> instance mapping is expressed as directories under 117 | ``~/.edgedb/projects``:: 118 | 119 | ~/.edgedb/projects/ 120 | <dir-basename>-<dir-abspath-hash>/ 121 | project-path 122 | instance-name 123 | 124 | Here ``<dir-basename>`` is the trailing component of the path to the project 125 | directory, and ``<dir-abspath-hash>`` is defined as 126 | ```lower(strip(base32(sha1(abspath(project_dir))), '==='))```. Hashing is 127 | necessary to avoid bumping into the maximum directory entry name length. 128 | The ``project-path`` file contains the full absolute path to the project 129 | directory and is necessary for reverse lookup (find project by instance name), 130 | and the ``instance-name`` file contains the name of the associated instance. 131 | 132 | Language bindings and clients may cache the resolved instance name in memory 133 | to avoid performing a lookup on every ``connect()`` call. The cache should 134 | be sensitive to the current working directory, although if the directory 135 | changed to a subdirectory the cache should still be valid as nested projects 136 | are not allowed. 137 | 138 | Running ``project init`` from ``curl | sh`` 139 | ------------------------------------------- 140 | 141 | The CLI bootstrap script can detect if it's being ran from within a project 142 | directory and ask if the user wants to run ``edgedb project init``. 143 | 144 | Effect on ``edgedb server`` commands 145 | ------------------------------------ 146 | 147 | The ``edgedb server destroy`` command should refuse to destroy an instance that 148 | is associated with a project by default, and should recommend to use 149 | ``edgedb project unlink`` (or ``edgedb server destroy --force``). 150 | 151 | The ``edgedb server upgrade`` command should refuse to continue if the target 152 | server version does not match the ``edgedb.server-version`` range in 153 | ``edgedb.toml``. 154 | 155 | 156 | edgedb project init 157 | =================== 158 | 159 | Initialize a new project or re-initialize an existing project. 160 | 161 | Synopsis 162 | -------- 163 | 164 | ``edgedb project init [options]`` 165 | 166 | Options 167 | ------- 168 | 169 | ``--project-dir=<dir>`` 170 | Specifies a project root directory explicitly. If not specified, the project 171 | directory is detected as described in the "Project detection" section above, 172 | and if no project directory is detected, a current working directory is used. 173 | 174 | ``--server-version=<semver-range>`` 175 | Specifies the desired EdgeDB server version as a SemVer range. Only 176 | applicable for new projects. Defaults to ``'*'``, which means latest stable 177 | version for new projects. Accepts ``'nightly'`` as a special value denoting 178 | the latest nightly version. 179 | 180 | ``--server-instance=<instance>`` 181 | Specifies the EdgeDB server instance to be associated with the project. 182 | If the specified instance does not exist, it will be created. If the 183 | specified instance already exists, it must not be associated with another 184 | project. ``edgedb project unlink`` may be used to disassociate an instance 185 | prior to linking it with another project. 186 | 187 | ``--server-instance-type=<instance-type>`` 188 | Specifies the desired instance type. Current allowed value for 189 | ``<instance-type>`` is ``local`` (future additions include ``cloud``) 190 | 191 | ``--server-install-mode=<install-mode>`` 192 | Corresponds to the local installation modes in ``edgedb server``. 193 | 194 | ``--non-interactive`` 195 | Run in non-interactive mode. 196 | 197 | Implementation 198 | -------------- 199 | 200 | The ``edgedb project init`` command initializes a brand new project or 201 | re-initializes an existing project. 202 | 203 | In a new project: 204 | 205 | - an ``edgedb.toml`` file is created in the project directory, 206 | and ``--server-version``, if specified, is recorded in the 207 | ``edgedb.server-version`` attribute. 208 | 209 | - a ``dbschema`` directory and a ``dbschema/default.esdl`` file are created, 210 | the latter containing this declaration:: 211 | 212 | module default { 213 | 214 | } 215 | 216 | - if the specified server version is not installed, ``edgedb server install`` 217 | performs the installation using the first available installation method 218 | in the order of preference (unless specified explicitly with 219 | ``--server-instance-type``). 220 | 221 | - if the specified or implied server instance does not exist, an attempt to 222 | create it is made. 223 | 224 | - a new record in ``~/.edgedb/projects`` is created for the new project. 225 | 226 | In an existing project: 227 | 228 | - the ``edgedb.toml`` file is read and validated; 229 | 230 | - if the specified server version is not installed, ``edgedb server install`` 231 | performs the installation using the first available installation method 232 | in the order of preference (unless specified explicitly with 233 | ``--server-instance-type``). 234 | 235 | - if the specified or implied server instance does not exist, an attempt to 236 | create it is made. 237 | 238 | - the record in ``~/.edgedb/projects`` is updated with the new instance name 239 | if necessary. 240 | 241 | - if ``dbschema/migrations`` exists, ``edgedb migrate`` is executed to ensure 242 | that the configured instance is up-to-date. 243 | 244 | Interactive mode 245 | ---------------- 246 | 247 | Here's a simulation of a proposed interactive mode for a new project:: 248 | 249 | $ edgedb project init 250 | `edgedb.toml` was not found in `/home/user/work/myapp` or above. 251 | Do you want to initialize a new project? [Y/n] Y 252 | What type of EdgeDB instance would you like to use with this project? 253 | 1. Local (native) 254 | 2. Local (Docker) 255 | 3. Cloud 256 | Your choice? 1 257 | Specify the version of EdgeDB to use with this project [latest stable]: 258 | Specify the name of EdgeDB instance to use with this project [myapp]: 259 | 260 | [shows summary of configuration] 261 | 262 | [asks whether to continue or restart configuration] 263 | 264 | Creating instance `myapp`... 265 | 266 | Here's a simulation of a proposed interactive mode for a cloned project:: 267 | 268 | $ edgedb project init 269 | Found `edgedb.toml` in `/home/user/work/myapp`. 270 | Found no associated EdgeDB instance. 271 | What type of EdgeDB instance would you like to use with this project? 272 | 1. Local (native) 273 | 2. Local (Docker) 274 | 3. Cloud 275 | Your choice? 1 276 | Specify the name of EdgeDB instance to use with this project [myapp]: 277 | 278 | [shows summary of configuration] 279 | 280 | [asks whether to continue or restart configuration] 281 | 282 | Creating instance `myapp` ... 283 | Running migrations ... 284 | 285 | 286 | edgedb project unlink 287 | ===================== 288 | 289 | Remove association with and optionally destroy the linked EdgeDB intstance. 290 | 291 | Synopsis 292 | -------- 293 | 294 | ``edgedb project unlink [options]`` 295 | 296 | Options 297 | ------- 298 | 299 | ``--project-dir=<dir>`` 300 | Specifies a project root directory explicitly. If not specified, the project 301 | directory is detected as described in the "Project detection" section above. 302 | 303 | ``--destroy-server-instance, -D`` 304 | If specified, the associated EdgeDB instance is destroyed by running 305 | ``edgedb server destroy``. 306 | 307 | ``--non-interactive`` 308 | Run in non-interactive mode. Assume affirmative answer for all questions. 309 | 310 | Implementation 311 | -------------- 312 | 313 | The ``edgedb project unlink`` command removes the association with its EdgeDB 314 | instance by removing the corresponding entry from the ``~/.edgedb/projects`` 315 | directory. If ``--destroy-server-instance`` is specified, the associated 316 | instance is destroyed. 317 | 318 | 319 | edgedb project status 320 | ===================== 321 | 322 | Shows the information about a project. Includes information about the 323 | associated instance name, its status, as well as the migration status. 324 | 325 | Synopsis 326 | -------- 327 | 328 | ``edgedb project status [options]`` 329 | 330 | Options 331 | ------- 332 | 333 | ``--project-dir=<dir>`` 334 | Specifies a project root directory explicitly. If not specified, the project 335 | directory is detected as described in the "Project detection" section above. 336 | 337 | ``--json`` 338 | Use JSON as output format. 339 | 340 | 341 | Implementation 342 | -------------- 343 | 344 | The ``edgedb project status`` shows the following information about the 345 | project: 346 | 347 | - associated EdgeDB instance name, or `<none>` if not associated; 348 | - migration status (which deprecates ``edgedb show-status``) 349 | 350 | 351 | edgedb project list 352 | =================== 353 | 354 | Lists all known EdgeDB projects for the current user. 355 | 356 | Synopsis 357 | -------- 358 | 359 | ``edgedb project list [options]`` 360 | 361 | Options 362 | ------- 363 | 364 | ``--json`` 365 | Use JSON as output format. 366 | 367 | 368 | Implementation 369 | -------------- 370 | 371 | The ``edgedb project list`` outputs a list of projects where each entry 372 | contains a full path to the project directory and the name of an associated 373 | EdgeDB instance. 374 | 375 | 376 | Rejected Ideas 377 | ============== 378 | 379 | Store all EdgeDB instance data alongside the project 380 | ---------------------------------------------------- 381 | 382 | We considered placing the credentials for the instance and optionally also 383 | instance data in ``<project-dir>/.edgedb``. 384 | 385 | The advantage of this approach is that it does not require explicit linking 386 | of projects with EdgeDB instances and avoids possible instance name conflicts 387 | by generating instance names. 388 | 389 | This approach has the following downsides: 390 | 391 | - ``<project-dir>/.edgedb`` MUST NOT be committed into VCS, most importantly 392 | due to possible exposure of secrets and other sensitive data. Automatic 393 | modification of ``.gitignore`` (and other VCS ignorefiles) may mitigate this, 394 | but risk still exists if a user runs ``git add -f``. 395 | 396 | - when properly ignored, ``<project-dir>/.edgedb`` is susceptible to accidental 397 | removal by ``git clean`` which may lead to data loss. 398 | 399 | - many users run their project directories on network filesystems or use 400 | Dropbox for synchronization across their machines. Placing an EdgeDB 401 | data directory on any sort of "magical" filesystem may lead to random 402 | corruption or significant performance issues. 403 | 404 | Maintain a copy of instance credentials in the project directory 405 | ---------------------------------------------------------------- 406 | 407 | This is a variant of the approach described above, except data is stored in 408 | the normal location and only the credentials file is copied to the project 409 | directory. 410 | 411 | This approach shares the VCS-related concerns of the approach above in that 412 | the credentials file must not be committed, and that ``git clean`` would 413 | disassociate the project with its instance leading to developer puzzlement 414 | and inconvenience. 415 | 416 | Use a globally-unique identifier for projects 417 | --------------------------------------------- 418 | 419 | We considered giving each project a globally-unique identifier recorded in 420 | ``edgedb.toml``, and using it as an alias to the name of the associated 421 | EdgeDB instance. The idea was rejected as it complicates project forks, 422 | because one has to remember to change the project id, and, most importantly, 423 | once the project has been shared, the project id must not be changed to 424 | avoid breaking project clones. 425 | 426 | Finally, *global* uniqueness is actually no necessary, we only care about 427 | local uniqueness, which is perfectly solved by using filesystem paths as 428 | keys. 429 | -------------------------------------------------------------------------------- /text/1003-cli-naming.rst: -------------------------------------------------------------------------------- 1 | :: 2 | 3 | Status: Withdrawn 4 | Type: Guideline 5 | Created: 2020-07-02 6 | RFC PR: `edgedb/rfcs#0014 <https://github.com/edgedb/rfcs/pull/14>`_ 7 | 8 | 9 | =============================== 10 | RFC 1003: Consistent CLI Design 11 | =============================== 12 | 13 | This RFC discusses the naming scheme for EdgeDB command line tools 14 | and their shortcuts in REPL. The RFC also documents which command line tools 15 | can be exposed in REPL and how, as well as how various REPL options should be 16 | specified. 17 | 18 | 19 | RFC Status 20 | ========== 21 | 22 | This RFC has been withdrawn in favor of RFC 1006 - Simplified CLI Design. 23 | 24 | 25 | Motivation 26 | ========== 27 | 28 | Since the CLI commands are likely to land in third-party documentation 29 | and automation, we want to ensure the design behind them is consistent 30 | and natural to the user. 31 | 32 | 33 | Guidelines for future CLI commands 34 | ================================== 35 | 36 | * For commands operating on a single server, especially commands around 37 | a single database within a single server, prefer ``edgedb <action>`` 38 | and ``edgedb <action>-<object>`` over ``edgedb <group> <command>``; 39 | 40 | * For commands allowing maintenance of many servers, use 41 | ``edgedb server <action>`` and ``edgedb server <action>-<object>``; 42 | 43 | * For interactive output, ``--help`` that exceeds the height of the 44 | current terminal should be shown in a pager like ``less``. 45 | 46 | * When using the ``<action>-<object>`` scheme, if the action in question 47 | would be the literal word "show", it can be omitted, for example: 48 | ``edgedb migration-log``. 49 | 50 | * Sub-command descriptions need to be as informative as possible. 51 | Repeating the name of the sub-command is not very useful without 52 | highlighting some details of the action. 53 | 54 | 55 | Specific Concerns 56 | ================= 57 | 58 | During discussions about the CLI, a number of specific issues were 59 | raised. They are addressed below for reference. 60 | 61 | Ballooning ``--help`` output 62 | ---------------------------- 63 | 64 | **Concern**: With more and more functionality exposed to the CLI, the 65 | ``<action>-<category>`` approach has the potential to balloon the output 66 | of ``edgedb -h`` to more than 100 lines, making it harder to read. 67 | 68 | **Decision**: a long but flat ``--help`` provides better discoverability 69 | of new functionality compared to ``<group> <command>`` as more possible 70 | actions are listed right away. If the output exceeds the current height 71 | of the terminal, an effective solution is to use a pager, as done by 72 | ``git`` and many other tools. More initial help output allows for 73 | quicker search of not only command names but also command descriptions. 74 | An alternative short single-page output is provided when ``-h`` is 75 | specified (this is what Git does). 76 | 77 | Command-line design doesn't really follow EdgeQL 78 | ------------------------------------------------ 79 | 80 | **Concern**: The initial design goal of using the ``<action>-<category>`` 81 | scheme was to have the CLI naming follow EdgeQL commands, e.g. 82 | ``$ edgedb create-superuser-role`` and ``CREATE SUPERUSER ROLE`` look 83 | similar. However this similarity is only superficial as command-line tools 84 | have additional CLI-specific options, such as ``--password-from-stdin``. 85 | 86 | **Decision**: This is true. One-to-one mapping between command-line 87 | functionality and EdgeQL should not be assumed by the user. There isn't 88 | much we can realistically do to bridge this gap because modes of use 89 | are different between EdgeQL and a command-line tool: 90 | 91 | * different escaping rules allowing for more flexibility within EdgeQL 92 | queries; 93 | 94 | * a multi-line editor for EdgeQL queries is provided but user shells 95 | might not provide multi-line editing of terminal commands; and 96 | 97 | * different information passing paradigms, with command-line tools using 98 | standard I/O pipelines whereas EdgeQL using sub-queries. 99 | 100 | Vague verbs or nouns can be misleading 101 | -------------------------------------- 102 | 103 | **Concern**: the ``edgedb show-status`` command does not indicate what kind 104 | of status it will show. It could be showing status of the current migration, 105 | or of the currently running server, or of how many active connections the 106 | server has at the moment. Renaming it to ``edgedb show-migration-status`` 107 | would make it longer to type and overall very verbose. With 108 | ``<group> <command>`` scheme the solution is obvious: 109 | ``edgedb migration status``. 110 | 111 | **Decision**: ``edgedb show-status`` should be renamed to 112 | ``edgedb migration-status`` which is consistent with 113 | ``edgedb migration-log``. 114 | 115 | In general, vague nouns and verbs should be avoided unless 116 | the feature in question is dead obvious (like ``edgedb dump``, 117 | ``edgedb query``, or ``edgedb server init``). 118 | 119 | This will also future-proof the CLI for new kinds of functionality that 120 | would use the same vague terms (e.g. ``edgedb time-machine-log``). 121 | 122 | Category-specific help sections 123 | ------------------------------- 124 | 125 | **Concern**: The ``<action>-<category>`` scheme does not allow 126 | category-specific help sections. With ``<group> <command>`` scheme, top-level 127 | help sections are natural: ``edgedb migration -h`` command can have a 128 | description of the recommended migrations workflow, while 129 | ``edgedb server -h`` would explain to the user how to work with EdgeDB 130 | servers. 131 | 132 | **Decision**: the CLI is an unlikely place for a user to learn about new 133 | complex functionality like the migration subsystem. Better suited avenues 134 | for this are the online documentation and/or ``man`` pages, or better yet, 135 | ``tldr``. 136 | 137 | To address discoverability of related sub-commands, top-level listing of 138 | sub-commands should not be alphabetical but rather grouped by functionality. 139 | 140 | ``configure`` uses ``<group> <command>`` whereas ``list-`` is flat 141 | ------------------------------------------------------------------ 142 | 143 | **Concern**: ``configure`` is inconsistent with ``list-``. 144 | 145 | **Decision**: It should be replaced by ``edgedb insert-config``, 146 | ``edgedb set-config``, and``edgedb reset-config``. 147 | 148 | Why is ``server`` special? 149 | -------------------------- 150 | 151 | **Concern**: the ``edgedb server`` CLI design outlined in RFC 1001 152 | directly conflicts with the ``<action>-<category>`` scheme. 153 | 154 | **Decision**: this sub-system is different because it is in fact 155 | a tool within a tool. Whereas other commands deal with a single 156 | database server, often a single database within that server, the 157 | ``edgedb server`` sub-system deals with management of potentially 158 | many databases and server instances. With that in mind, keeping it 159 | separate makes sense. 160 | 161 | In particular, trying to shoehorn the sub-commands of ``edgedb server`` 162 | into the ``<action>-<object>`` scheme would essentially move all of 163 | them into flat ``*-instance`` (or ``*-server``) sub-commands. 164 | 165 | Are ``list-*`` commands useful? 166 | ------------------------------- 167 | 168 | **Concern**: there are nine ``list-*`` sub-commands which is over 36% 169 | of the available sub-commands. They mirror REPL functionality for listing 170 | items in the current database, for example ``\la`` or ``\list-aliases`` 171 | will list available aliases. Are all of them necessary? 172 | 173 | **Decision**: the CLI should keep listings of entities that it can 174 | directly manipulate with other sub-commands. This includes: 175 | 176 | * ``list-databases`` (through ``create-database``); 177 | 178 | * ``list-ports`` (through ``configure``); and 179 | 180 | * ``list-roles`` (through ``alter-role``). 181 | 182 | 183 | Connection options are passed in an unnatural spot 184 | -------------------------------------------------- 185 | 186 | **Concern**: Current ``edgedb`` command usage is defined as:: 187 | 188 | edgedb [FLAGS] [OPTIONS] [SUBCOMMAND] 189 | 190 | where ``[OPTIONS]`` is defined as:: 191 | 192 | --dsn <dsn> DSN for EdgeDB to connect to 193 | -d, --database <database> Database name to connect to 194 | -H, --host <host> Host of the EdgeDB instance 195 | -P, --port <port> Port to connect to EdgeDB 196 | -I, --instance <instance> Local instance name created with 197 | `edgedb server init` to connect to 198 | (overrides host and port) 199 | -u, --user <user> User name of the EdgeDB user 200 | 201 | This sometimes creates unreadable commands through inconvenient syntax, 202 | for instance:: 203 | 204 | $ edgedb -d tutorial dump tutorial.edgedb 205 | 206 | **Under consideration**: Passing *connection* options could be allowed 207 | anywhere between ``edgedb`` and sub-commands, enabling the following:: 208 | 209 | $ edgedb dump -d tutorial tutorial.edgedb 210 | $ edgedb dump tutorial.edgedb -d tutorial 211 | 212 | This simplifies the overall UX, as for some commands it's logical to 213 | receive the DB name as part of their ``[COMMAND-FLAGS]``. 214 | 215 | Allowing connection options to come last also simplifies copy-pasting 216 | them, which is especially useful for full DSNs. 217 | 218 | There's open discussion currently around specifics of those global 219 | options, their propagation, and conflicts with sub-command options 220 | using the same names. 221 | 222 | 223 | Rejected Ideas 224 | ============== 225 | 226 | Adopt the ``<group> <command>`` naming scheme for all of CLI 227 | ------------------------------------------------------------ 228 | 229 | An outline of ``edgedb`` subcommands would look like this: 230 | 231 | * ``server`` (see also RFC 1001.) 232 | - ``init`` 233 | - ``install`` 234 | - ``restart`` 235 | - ``start`` 236 | - ``status`` 237 | - ``stop`` 238 | 239 | * ``dump`` 240 | - ``db`` -- backup a database. 241 | - ``all`` -- backup all databases, as well as roles, configs, etc into 242 | a directory. 243 | - ``restore-db`` 244 | - ``restore-all`` 245 | - ``config`` -- backup system configuration. 246 | 247 | * ``migration`` 248 | - ``status`` 249 | - ``create`` 250 | - ``apply`` 251 | 252 | * ``role`` 253 | - ``create [--superuser]`` 254 | - ``alter`` 255 | - ``drop`` 256 | - ``list`` 257 | 258 | * ``db`` 259 | - ``create`` 260 | - ``rename`` 261 | - ``drop`` 262 | - ``list`` 263 | 264 | * ``config [--system]`` (used to be ``edgedb configure``) 265 | - ``set`` 266 | - ``reset`` 267 | - ``add`` (used to be ``insert``) 268 | - ``show`` 269 | 270 | * ``run [--stdin | -c]`` -- run an EdgeQL script or command from stdin 271 | or passed via the command line with ``-c``. 272 | 273 | This is rejected due to: 274 | 275 | * inability to adjust every command naturally in this way; 276 | 277 | * disruptive nature of the change; 278 | 279 | * less verbose ``help`` output; and 280 | 281 | * less natural-sounding commands. 282 | 283 | Read "Appendix 2" for research conducted in this area. 284 | 285 | 286 | Expose CLI-specific commands in the REPL via the ``!`` prefix 287 | ------------------------------------------------------------- 288 | 289 | The suggested syntax separated internal escaped commands:: 290 | 291 | >>> \list-databases 292 | tutorial 293 | 294 | and CLI-derived commands:: 295 | 296 | >>> !dump db tutorial 297 | done 298 | 299 | The idea was that through this separation the internal REPL help system is 300 | not overloaded with rarely needed help on CLI commands and would only show 301 | the list of convenient ``\`` commands with a hint that ``!help`` or ``!h`` 302 | can be used to list all CLI options. Name-spacing ``\help`` and ``!help`` 303 | could be good for usability, because the latter set of commands is not going 304 | to be used as frequently as the former. 305 | 306 | This was rejected because the usability claims were not in fact universally 307 | accepted. An important concern was that two schemes of escaped commands would 308 | increase user confusion, making them search for some commands twice. In the 309 | end EdgeDB CLI would have to implement a scheme like:: 310 | 311 | some-db> \dump my.dump 312 | No such command `dump`. 313 | Did you mean `!dump`? 314 | > !list-databases 315 | Do such command `list-databases` 316 | Did you mean `\list-databases`? 317 | 318 | 319 | Appendix 1: Current CLI state as of 1.0a7 320 | ========================================= 321 | 322 | The current naming scheme of EdgeDB command line tools is inconsistent. Both 323 | ``<group> <command>`` as well as ``<action>-<category>`` schemes are used, 324 | and the `<group>` in the first kind can either be a noun or verb. 325 | 326 | List of sub-commands:: 327 | 328 | $ edgedb -h 329 | <...> 330 | SUBCOMMANDS: 331 | alter-role 332 | configure 333 | create-database 334 | create-migration 335 | create-superuser-role 336 | describe 337 | drop-role 338 | dump 339 | help 340 | list-aliases 341 | list-casts 342 | list-databases 343 | list-indexes 344 | list-modules 345 | list-object-types 346 | list-ports 347 | list-roles 348 | list-scalar-types 349 | migrate 350 | migration-log 351 | query 352 | restore 353 | self-upgrade 354 | server 355 | show-status 356 | 357 | Server sub-commands:: 358 | 359 | $ edgedb server -h 360 | <...> 361 | SUBCOMMANDS: 362 | destroy 363 | help 364 | info 365 | init 366 | install 367 | list-versions 368 | logs 369 | reset-password 370 | restart 371 | start 372 | status 373 | stop 374 | uninstall 375 | upgrade 376 | 377 | Configuration sub-commands:: 378 | 379 | $ edgedb configure -h 380 | <...> 381 | SUBCOMMANDS: 382 | help 383 | insert 384 | reset 385 | set 386 | 387 | 388 | Appendix 2: CLI design of the 70 most downloaded Homebrew packages 389 | ================================================================== 390 | 391 | To determine what the other popular tools in the industry are doing, 392 | 70 Homebrew packages were investigated in terms of their CLI UX. 393 | 394 | The packages were chosen from a list of most downloaded Homebrew 395 | packages. Packages that were clearly libraries or otherwise lacked 396 | a non-trivial CLI were skipped. 397 | 398 | It turns out only 20 of those packages support sub-commands: git, yarn, 399 | imagemagick, awscli, go, maven, heroku, rbenv, gradle, tmux, carthage, 400 | docker, nvm, pyenv, ansible, sbt, terraform, hg, kubectl, hugo, 401 | docker-compose. Almost all do it inconsistently. 402 | 403 | Interesting findings: 404 | 405 | * Go started with ``go [verb]`` and now with modules had to do 406 | ``go mod init``. 407 | 408 | * Docker deprecated the ``<action>-<object>`` scheme but it looks like 409 | the community is unaware of it. New documentation and third-party 410 | materials are still written using this syntax. In effect, the tool 411 | supports both schemes. 412 | 413 | * Git and Mercurial mix nouns and verbs as the command ("tag", "branch", 414 | vs. "pull", "push", "commit"). 415 | 416 | * Ansible is interesting: "name-of-group" is the sole argument followed 417 | by command-line options mimicking sub-commands. 418 | 419 | * GPG also uses CLI options as sub-commands. 420 | 421 | * kubectl is one example that consistently applies "kubectl <verb> <object>". 422 | To keep this consistency, unnatural sub-commands like "get" are present 423 | which require another subcommand like "pods". The verbs are called 424 | "operations" and the operands are called "resources" when they can be 425 | many things. For most operations, they can only be one thing (like in 426 | "drain", "convert", "apply", "run"). When they can't, help for the 427 | polymorphic ones is a challenge to follow. 428 | 429 | * The AWS CLI is exactly backwards: it (almost) consistently applies 430 | ``<service> <verb>`` but that's because it's a multi-tool for dealing 431 | with logically separate services. As soon as you get to a particular 432 | service, ``<action>-<object>`` is used. 433 | 434 | * A lot of tools use cute but vague commands like "init", "new", and "add". 435 | It's mostly pretty obvious from context what they are. 436 | 437 | Conclusions 438 | ----------- 439 | 440 | Successful command-line tools mostly use `<verb>` as the sub-command 441 | scheme which is similar to what we are doing now for EdgeDB with the 442 | exception of ``edgedb server`` and the implicit "show". 443 | 444 | And when the tools do go for the ``<group> <command>`` scheme, they 445 | don't do it consistently. This shows it's very tricky to accomplish in 446 | a non-trivial application and likely isn't a deciding factor in the 447 | overall user experience. 448 | --------------------------------------------------------------------------------