├── .gitignore ├── FAQ.md ├── design-discussions ├── README.md ├── definition.md ├── upcast-safety.md ├── vtable-layout.md ├── upcast-safety-3-options.md ├── upcast-safety-3.md └── upcast-safety-2.md ├── updates.md ├── CODE_OF_CONDUCT.md ├── .github ├── ISSUE_TEMPLATE │ ├── question.md │ ├── proposal.md │ ├── bug-report.md │ └── experience-report.md └── workflows │ └── deploy_mdbook.yml ├── explainer.md ├── book.toml ├── SUMMARY.md ├── updates └── 2021-oct.md ├── CHARTER.md ├── LICENSE-MIT ├── README.md └── LICENSE-APACHE /.gitignore: -------------------------------------------------------------------------------- 1 | /book 2 | -------------------------------------------------------------------------------- /FAQ.md: -------------------------------------------------------------------------------- 1 | # 😕 FAQ 2 | -------------------------------------------------------------------------------- /design-discussions/README.md: -------------------------------------------------------------------------------- 1 | # Design questions 2 | 3 | Catalogs various interesting design questions that have arisen. -------------------------------------------------------------------------------- /updates.md: -------------------------------------------------------------------------------- 1 | # ✏️ Updates 2 | 3 | Lang-team initiatives give monthly updates. This section collects the updates from this initiative for posterity. -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | # The Rust Code of Conduct 2 | 3 | The Code of Conduct for this repository [can be found online](https://www.rust-lang.org/conduct.html). 4 | -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/question.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Question 3 | about: Ask for clarification about some aspect of the design 4 | title: '' 5 | labels: question 6 | assignees: '' 7 | 8 | --- 9 | 10 | Please describe what you want to know! -------------------------------------------------------------------------------- /.github/workflows/deploy_mdbook.yml: -------------------------------------------------------------------------------- 1 | on: 2 | push: 3 | branches: 4 | - master 5 | 6 | jobs: 7 | ci: 8 | name: CI 9 | runs-on: ubuntu-latest 10 | steps: 11 | - uses: actions/checkout@v2 12 | - uses: XAMPPRocky/deploy-mdbook@v1 13 | with: 14 | token: ${{ secrets.GITHUB_TOKEN }} 15 | -------------------------------------------------------------------------------- /explainer.md: -------------------------------------------------------------------------------- 1 | # 📚 Explainer 2 | 3 | > The "explainer" is "end-user readable" documentation that explains how to use the feature being deveoped by this initiative. 4 | > If you want to experiment with the feature, you've come to the right place. 5 | > Until the feature enters "feature complete" form, the explainer should be considered a work-in-progress. -------------------------------------------------------------------------------- /book.toml: -------------------------------------------------------------------------------- 1 | [book] 2 | authors = ["Dyn upcast Members"] 3 | language = "en" 4 | multilingual = false 5 | src = "." 6 | title = "Dyn upcast initiative" 7 | 8 | [output.html] 9 | no-section-label=true 10 | git-repository-url="https://github.com/rust-lang/dyn-upcasting-coercion-initiative" 11 | edit-url-template="https://github.com/rust-lang/dyn-upcasting-coercion-initiative/edit/master/{path}" 12 | site-url="/dyn-upcasting-coercion-initiative/" 13 | 14 | [output.html.fold] 15 | enable = true 16 | level = 0 -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/proposal.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Proposal 3 | about: Propose a change to the way this feature works 4 | title: '' 5 | labels: proposal 6 | assignees: '' 7 | 8 | --- 9 | 10 | **When proposing changes, please check the FAQ and design questions to see if this idea has already been considered. It's also a good idea to check with the owner first.** 11 | 12 | ### Motivation 13 | 14 | Describe what you're trying to accomplish -- what problem are you trying to solve? 15 | 16 | ### Details 17 | 18 | Describe your idea. 19 | 20 | -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/bug-report.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Bug report 3 | about: Report a bug using this feature 4 | title: '' 5 | labels: bug 6 | assignees: '' 7 | --- 8 | 9 | ### When using 10 | 11 | Please indicate the precise nightly build you are using; try `rustc --version` 12 | 13 | ### What I tried to do 14 | 15 | Please include sample code or links to your project! 16 | 17 | ### What happened 18 | 19 | Describe what happened in as much detail as possible. 20 | For compiler ICEs, a good idea would be to include the output with `RUST_BACKTRACE=1`. 21 | 22 | ### What I expected 23 | 24 | Describe what you thought should happen. 25 | -------------------------------------------------------------------------------- /design-discussions/definition.md: -------------------------------------------------------------------------------- 1 | # What qualitifies as a dyn upcasting coercion 2 | 3 | The condition comes from the existing infrastructure within the compiler. 4 | 5 | Currently the unsizing coercion on a trait object type allows it to: 6 | 7 | 1. Removing one or more auto traits. (i.e. `dyn Foo + Send: Unsize`) 8 | 9 | 2. Changing lifetime bounds according to subtyping rules. (`dyn Bar<'static>: Unsize>`) 10 | 11 | 3. Changing the principal trait to one of its supertraits. (`dyn Goo: Unsize` where `Goo` is `trait Goo: Foo {}`) 12 | 13 | When the third rule is involved, this unsizing coercion is a dyn upcasting coercion. 14 | 15 | 16 | -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/experience-report.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Experience report 3 | about: Describe your experiences using this feature 4 | title: '' 5 | labels: experience-report 6 | assignees: '' 7 | 8 | --- 9 | 10 | ### Code description 11 | 12 | Describe what you were trying to do with upcasting dyn Trait values to supertraits. Links to source code are always appreciated! 13 | 14 | ### What worked well 15 | 16 | Describe what aspects of dyn upcasting coercions worked well for you. 17 | 18 | ### What worked less well 19 | 20 | Describe what aspects of dyn upcasting coercions did not work well for you. Perhaps they were confusing, or you weren't able to get your code to compile the way you wanted? Or perhaps it was just not ergonomic. 21 | -------------------------------------------------------------------------------- /SUMMARY.md: -------------------------------------------------------------------------------- 1 | 2 | # Summary 3 | 4 | - [👋 Welcome](./README.md) 5 | - [✏️ Updates](./updates.md) 6 | - [2021-Oct](./updates/2021-oct.md) 7 | - [📜 Charter](./CHARTER.md) 8 | - [📚 Explainer](./explainer.md) 9 | - [💬 Design discussions](./design-discussions/README.md) 10 | - [Definition](./design-discussions/definition.md) 11 | - [Upcast safety](./design-discussions/upcast-safety-3.md) 12 | - [Detailed options](./design-discussions/upcast-safety-3-options.md) 13 | - [Older draft 2](./design-discussions/upcast-safety-2.md) 14 | - [Older draft 1](./design-discussions/upcast-safety.md) 15 | - [Vtable layout and runtime behavior](./design-discussions/vtable-layout.md) 16 | - [😕 FAQ](./FAQ.md) 17 | 18 | -------------------------------------------------------------------------------- /updates/2021-oct.md: -------------------------------------------------------------------------------- 1 | # 2021-Oct: Lang team update 2 | 3 | ## Summary 4 | 5 | * Impl is largely ready 6 | * Blocked on deciding the question of [upcast safety](../design-discussions) 7 | 8 | ## Goals for this month 9 | 10 | * Hold design meeting about upcast safety and reach a conclusion 11 | 12 | ## Deref coercion 13 | 14 | It was discovered in [#89190] that extending "unsizing coercions" to include upcasts can cause deref coercions not to trigger, if those deref coercions "deref" to a `dyn` type. Current plan is to issue future compatibility warnings in [#89461](https://github.com/rust-lang/rust/pull/89461), since the only known use of this is emulating the upcasting coercion (and hence the code will continue to work afterwards, but the Deref impl is not invoked). 15 | 16 | [#89190]: https://github.com/rust-lang/rust/issues/89190 -------------------------------------------------------------------------------- /CHARTER.md: -------------------------------------------------------------------------------- 1 | # Dyn upcast Charter 2 | 3 | ## Goals 4 | 5 | ### Summary and problem statement 6 | 7 | * Add the `trait_upcasting` feature to the language. 8 | 9 | ### Motivation, use-cases, and solution sketches 10 | 11 | * The `trait_upcasting` feature adds support for trait upcasting coercion. This allows a 12 | trait object of type `dyn Bar` to be cast to a trait object of type `dyn Foo` 13 | so long as `Bar: Foo`. 14 | 15 | ```rust,edition2018 16 | #![feature(trait_upcasting)] 17 | 18 | trait Foo {} 19 | 20 | trait Bar: Foo {} 21 | 22 | impl Foo for i32 {} 23 | 24 | impl Bar for T {} 25 | 26 | let bar: &dyn Bar = &123; 27 | let foo: &dyn Foo = bar; 28 | ``` 29 | 30 | ### Non-goals 31 | 32 | * To support `dyn Trait1 + Trait2` for general traits (this syntax naturally works, and should continue to work, with auto traits). 33 | 34 | ## Membership 35 | 36 | | Role | Github | 37 | | --- | --- | 38 | | [Owner] | [crlf0710](https://github.com/crlf0710) | 39 | | [Liaison] | [nikomatsakis](https://github.com/nikomatsakis) | 40 | -------------------------------------------------------------------------------- /LICENSE-MIT: -------------------------------------------------------------------------------- 1 | Permission is hereby granted, free of charge, to any 2 | person obtaining a copy of this software and associated 3 | documentation files (the "Software"), to deal in the 4 | Software without restriction, including without 5 | limitation the rights to use, copy, modify, merge, 6 | publish, distribute, sublicense, and/or sell copies of 7 | the Software, and to permit persons to whom the Software 8 | is furnished to do so, subject to the following 9 | conditions: 10 | 11 | The above copyright notice and this permission notice 12 | shall be included in all copies or substantial portions 13 | of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF 16 | ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED 17 | TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A 18 | PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT 19 | SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY 20 | CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION 21 | OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR 22 | IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER 23 | DEALINGS IN THE SOFTWARE. 24 | -------------------------------------------------------------------------------- /design-discussions/upcast-safety.md: -------------------------------------------------------------------------------- 1 | # Upcast safety 2 | 3 | **NB: This document is outdated.** [Click here to see the latest version.](./upcast-safety-3.md) 4 | 5 | ## Scenario 6 | 7 | * Casting `*dyn Foo` to `*dyn Bar` requires adjusting the vtable from a vtable for `Foo` to one for `Bar` 8 | * For raw pointers, the metadata can be supplied by the user with `from_raw_parts` 9 | * If that metadata is incorrect, this could cause UB: 10 | * our current vtable format requires loads, and the pointers may not be valid 11 | * a flat vtable layout could give rise to out-of-bounds loads 12 | * Unsafety is needed, but where? 13 | 14 | ## Options 15 | 16 | ### Unsafe to create a `*dyn Foo` 17 | 18 | > Announce that every fat pointer needs to have valid metadata part. ~~Needs to switch the std::ptr::from_raw_parts{,_mut} APIs to be unsafe.~~ And updates other documentations. 19 | 20 | **Implication:** Not able to create a "null pointer" version of `*dyn Foo` unless: 21 | 22 | * You use `Option<*dyn Foo>`, of course 23 | * We create some kind of "dummy" vtable that is structurally correct but has no actual functions within it; we would need a function for creating "valid-but-default" metadata as part of custom DST 24 | 25 | Update: 26 | * It was pointed out by `steffahn` that `std::ptr::from_raw_parts` won't create fat pointer with invalid metadata. 27 | * One of the remaining ways to create a pointer with invalid metadata is [by using `transmute`](https://github.com/rust-lang/rust/issues/81513#issuecomment-798158332). 28 | 29 | Question: 30 | * Does the language UB happen at `transmute` site or coercion site? 31 | 32 | ### Alter vtable layout to be flat 33 | 34 | > Make vtables "flat", by removing all pointer indirections in vtables and appending all the data to the tail. This makes upcasting coercion codegen become adding an offset to the metadata scalar, so won't cause real UB. Will waste some more static bytes in multiple inheritance cases than before, might make embedded-dev people unhappy. 35 | 36 | Question from Niko: 37 | * Is this sufficient? Does it imply we can't use `InBoundsGep`? Is that already true? 38 | 39 | ### Make raw pointer unsizing unsafe 40 | 41 | > Announce that raw pointer unsizing coercion must happen in unsafe blocks, while other unsizing coercions can happen outside an unsafe block. This is actually a small breaking change. So need a future compat lint to migrate existing users dealing with raw pointers and some more changes to std(POC PR at #88239 explains the details but it's a bit long). A few other MIR-level details become observable by user: whether the compiler thinks it's a unsizing coercion or not. 42 | > 43 | > nikomatsakis: (cc @RalfJ, if you happen to be around) 44 | 45 | ## Conversation log 46 | 47 | * [2021-08-25](https://zulip-archive.rust-lang.org/stream/144729-wg-traits/topic/object.20upcasting.html#250616592) -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Dyn upcasting coercion initiative 2 | 3 | ![initiative status: active](https://img.shields.io/badge/status-active-brightgreen.svg) 4 | 5 | ## What is this? 6 | 7 | This page tracks the work of the dyn upcasting coercion [initiative]! To learn more about what we are trying to do, and to find out the people who are doing it, take a look at the [charter]. 8 | 9 | [charter]: ./CHARTER.md 10 | [initiative]: https://lang-team.rust-lang.org/initiatives.html 11 | 12 | ## Current status 13 | 14 | The following table lists of the stages of an initiative, along with links to the artifacts that will be produced during that stage. 15 | 16 | | Stage | State | Artifact(s) | 17 | | ------------------------------------- | ----- | ----------- | 18 | | [Proposal] | ✅ | [Proposal issue](https://github.com/rust-lang/lang-team/issues/98) | 19 | | | | [Charter](./CHARTER.md) | 20 | | | | [Tracking issue](https://github.com/rust-lang/rust/issues/65991) | 21 | | [Development] | 🦀 | [Explainer](./explainer.md) | 22 | | [Feature complete] | 💤 | Stabilization report | 23 | | [Stabilized] | 💤 | | 24 | 25 | [Proposal]: https://lang-team.rust-lang.org/initiatives/process/stages/proposal.html 26 | [Experimental]: https://lang-team.rust-lang.org/initiatives/process/stages/proposal.html 27 | [Development]: https://lang-team.rust-lang.org/initiatives/process/stages/development.html 28 | [Feature complete]: https://lang-team.rust-lang.org/initiatives/process/stages/feature-complete.html 29 | [Stabilized]: https://lang-team.rust-lang.org/initiatives/process/stages/stabilized.html 30 | 31 | Key: 32 | 33 | * ✅ -- phase complete 34 | * 🦀 -- phase in progress 35 | * 💤 -- phase not started yet 36 | 37 | ## How Can I Get Involved? 38 | 39 | * Check for 'help wanted' issues on this repository! 40 | * If you would like to help with development, please contact the [owner](./charter.md#membership) to find out if there are things that need doing. 41 | * If you would like to help with the design, check the list of active [design discussions](./design-discussions/README.md) first. 42 | * If you have questions about the design, you can file an issue, but be sure to check the [FAQ](./FAQ.md) or the [design-discussions](./design-discussions/README.md) first to see if there is already something that covers your topic. 43 | * If you are using the feature and would like to provide feedback about your experiences, please [open a "experience report" issue]. 44 | * If you are using the feature and would like to report a bug, please open a regular issue. 45 | 46 | We also participate on [Zulip][chat-link], feel free to introduce yourself over there and ask us any questions you have. 47 | 48 | [open issues]: /issues 49 | [chat-link]: https://rust-lang.zulipchat.com/#narrow/stream/144729-wg-traits 50 | [team-toml]: https://github.com/rust-lang/team/blob/master/teams/initiative-dyn-upcasting-coercion.toml 51 | 52 | ## Building Documentation 53 | This repository is also an mdbook project. You can view and build it using the 54 | following command. 55 | 56 | ``` 57 | mdbook serve 58 | ``` 59 | -------------------------------------------------------------------------------- /design-discussions/vtable-layout.md: -------------------------------------------------------------------------------- 1 | # Vtable format to support dyn upcasting coercion 2 | 3 | This current design was proposed by `Mario Carneiro` based on previous proposals on [Zulip discussion](https://zulip-archive.rust-lang.org/stream/243200-t-lang/major-changes/topic/Trait.20Upcasting.20lang-team.2398.html#242876426). It's a hybrid approach taking the benefits of both a "flat" design, and a "pointer"-based design. 4 | 5 | This is implemented in [#86461](https://github.com/rust-lang/rust/pull/86461). 6 | 7 | The vtable is generated by this algorithm in principle for a type `T` and a trait `Tr`: 8 | 1. First emit the header part, including `MetadataDropInPlace`, `MetadataSize`, `MetadataAlign` items. 9 | 2. Create a tree of all the supertraits of this `TraitRef`, by filtering out all of duplicates. 10 | 3. Collect a set of `TraitRef`s consisting the trait and its first supertrait and its first supertrait's super trait,... and so on. Call this set `PrefixSet` 11 | 4. Traverse the tree in post-order, for each `TraitRef` emit all its associated functions as either `Method` or `Vacant` entries. If this `TraitRef` is not in `PrefixSet`, emit a `TraitVPtr` containing a constant pointer to the vtable generated for the type `T` and this `TraitRef`. 12 | 13 | ## Example 14 | 15 | ```rust 16 | trait A { 17 | fn foo_a(&self) {} 18 | } 19 | 20 | trait B: A { 21 | fn foo_b(&self) {} 22 | } 23 | 24 | trait C: A { 25 | fn foo_c(&self) {} 26 | } 27 | 28 | trait D: B + C { 29 | fn foo_d(&self) {} 30 | } 31 | ``` 32 | 33 | ```text 34 | Vtable entries for ``: [ 35 | MetadataDropInPlace, 36 | MetadataSize, 37 | MetadataAlign, 38 | Method(::foo_a), 39 | Method(::foo_b), 40 | Method(::foo_c), 41 | TraitVPtr(), 42 | Method(::foo_d), 43 | ] 44 | 45 | Vtable entries for ``: [ 46 | MetadataDropInPlace, 47 | MetadataSize, 48 | MetadataAlign, 49 | Method(::foo_a), 50 | Method(::foo_c), 51 | ] 52 | ``` 53 | 54 | ## Runtime behavior for dyn upcasting coercion 55 | At the codegen time, the same algorithm above is performed for the source principal trait and the source trait object type. If the target principal trait is in the `PrefixSet` set, this coercion is a no-op. 56 | 57 | If the target principal trait is not in the `PrefixSet`, generate code that read the data pointer from the 58 | corresponding `TraitVPtr` slot. 59 | 60 | # Alternatives 61 | ## Flat design 62 | 63 | The vtable is generated by this algorithm in principle for a type `T` and a trait `Tr`: 64 | 1. Create a tree of all the supertraits of this `TraitRef`, duplicate for the cyclic cases. 65 | 2. Traverse the tree in post-order, for each `TraitRef`, 66 | 1. if it has no supertrait, emit a header part, including `MetadataDropInPlace`, `MetadataSize`, `MetadataAlign` items. 67 | 2. emit all its associated functions as either `Method` or `Vacant` entries. 68 | 69 | ### Cons 70 | If there is a lot of diamond inheritance that could result in exponential blowup of the vtable 71 | for example, `trait A(n+1): Bn + Cn {}, trait Bn: An { fn bn(&self); }, trait Cn: An { fn cn(&self); }` 72 | 73 | `the vtable for An will contain 2^n DSAs` 74 | 75 | ## Pointer-based design 76 | 77 | All traits have their own vtables, and embedded all links to supertraits in the vtables 78 | 79 | ### Cons 80 | Codegen regression for single-inheritance cases, which is very widely used. 81 | -------------------------------------------------------------------------------- /design-discussions/upcast-safety-3-options.md: -------------------------------------------------------------------------------- 1 | # Upcast safety options considered 2 | 3 | This document details a number of options that were considered while preparing the [upcast-safety-3](./upcast-safety-3.md) proposal. 4 | 5 | ## Summary table 6 | 7 | | proposal | unconditional-upcast | can-create-from-zeroed | can-create-from-arbitrary-bits | niche-for-dyn | can-release-something-zeroed-to-safe | 8 | | --- | --- | --- | --- | --- | --- | 9 | | fully-valid-vtable | :ballot_box_with_check: | :octagonal_sign: | :octagonal_sign: | :ballot_box_with_check: | :octagonal_sign: | 10 | | null-placeholder | :octagonal_sign: | :ballot_box_with_check: | :octagonal_sign: | :ballot_box_with_check: | :octagonal_sign: 11 | | operations-are-ub | :ballot_box_with_check: | :ballot_box_with_check: | :ballot_box_with_check: | :octagonal_sign: | :octagonal_sign: | 12 | | operations-are-ub-nza | :ballot_box_with_check: | :octagonal_sign: | :octagonal_sign: | :ballot_box_with_check: | :octagonal_sign: | 13 | 14 | ## True no matter what 15 | 16 | * `MaybeUnit<*const dyn Foo>` can be used to create an uninitialized `*const dyn Foo` easily, like with any other type. 17 | * If we add `*const self` methods (which we should), and permit those to be invoked from safe code (which would be consistent, given that one can invoke a `self` method defined on a `*const T` type in safe code), then it will be *unsound* (i.e., potentially allow safe code to create UB) to release a `*const dyn Foo` to safe code unless the vtable is *fully valid* (created by the compiler). 18 | * vtable layout is not defined. 19 | * No matter what proposal we adopt, if you create a `*const dyn Foo` with anything less than a fully valid vtable (including e.g. a null placeholder), **you must not allow that to escape to arbitrary code** 20 | * This is actually a consequence of wanting to support `*const self` methods and method dispatch -- it could conceivably be true only for a trait `Foo` is it includes a `*const self` method, since those are not yet legal. 21 | 22 | ## Proposals 23 | 24 | ### fully-valid-vtable 25 | 26 | It is UB to create a `*const dyn Foo` unless the vtable is fully valid (given that vtable layouts are undefined, this means produced by the compiler for the time being). 27 | 28 | If you need to create a `*const dyn Foo` and don't have a valid vtable available, you can use `MaybeUninit<*const dyn Foo>`, or a union, or `Option`, which is probably good practice. 29 | 30 | Forwards compatibility: 31 | 32 | * Given that vtable layouts are not defined, there isn't much you can do at present in unsafe code with a vtable besides invoking methods, upcasting, computing the size, or other such operations. 33 | * If we were to adopt other proposals that weaken the validity requirement, compatibility is a bit complex. At the time we make the change, all `*const dyn` values in extant code would have valid vtables. But people would be able to write code that created values with less-than-valid vtables (e.g., maybe null, etc, depending on what change we made). They would not however be able to give that code to other functions, even unsafe functions, unless those functions stated explicitly that they don't require the `*const dyn` value to have a fully valid vtable (i.e., they promised not to upcast it, etc). 34 | * This is effectively the same as the operations-are-ub proposal, even if we adopt it today. You can create invalid vtables, but you cannot allow them to escape. 35 | 36 | ### null-placeholder 37 | 38 | It is UB to create a `*const dyn Foo` unless the vtable is "sufficiently valid", except that NULL is permitted as a placeholder value. 39 | 40 | This has the consequence that when we upcast, we have to check for NULL, which is less efficient. 41 | 42 | Upshot: 43 | 44 | * You may create a `*const dyn Foo` from `std::mem::zeroed`. 45 | * The only thing you can do on a `*const dyn Foo` that is zeroed is copy it and upcast it. 46 | * *Currently*, this means you can release a `*const dyn Foo` to safe code, but if we were to add `*const self` methods, that might no longer be true, unless those methods are also prepared to deal with the null pointer in some way (e.g., abort?), or unless it's unsafe to invoke such a method (seems odd). But that's really an issue for `*const self` methods to deal with, since it's already true today. 47 | * If receiving one from an unknown source, you can assume the vtable is either value for some type or NULL (of course, you can't assume anything about the *data* pointer, and vtable layouts are unknown, so it's not clear how useful that is). 48 | 49 | Forwards compatibility: 50 | 51 | * We can adopt any other proposal later, but we might lose the niche. 52 | 53 | Notes: 54 | 55 | * We can in theory support a niche with a value like 0x1 for the vtable? 56 | 57 | ### operations-are-ub 58 | 59 | There is no validity invariant for `*const dyn Foo` -- the vtable can be arbitrary. However, the only "non-UB" operation that you can do on a `*const dyn Foo` is to copy it, unless the vtable is known to be valid (produced by the compiler). Without a known valid vtable, all other options, including but not limited to the following, are UB: 60 | 61 | * Upcasting it do `*const dyn Bar` (where `trait Foo: Bar`) 62 | * Invoking methods from `Foo` (not currently possible without arbitrary-self-types) 63 | * Creating a safe pointer type (e.g., `&*foo`) 64 | 65 | Upshot: 66 | 67 | * You can create a `*const dyn Bar` with `mem::zeroed`, but it must not escape to safe code (that would permit the safe code to create UB, and hence be unsound). 68 | * If you are going to do anything besides copy that value around, it must be known to have a valid vtable at that time, or UB will result. 69 | 70 | Forwards compatibility: 71 | 72 | * If we adopt this proposal, we can't move to the others without introducing UB into otherwise valid code (consequence of the can-create-from-arbitrary-bits) 73 | 74 | ### operations-are-ub-nza 75 | 76 | The validity invariant for `*const dyn Foo` is that the vtable must be non-zero and aligned, ensuring a niche. However, -- the vtable can be arbitrary. However, the only "non-UB" operation that you can do on a `*const dyn Foo` is to copy it, unless the vtable is known to be valid (produced by the compiler). Without a known valid vtable, all other options, including but not limited to the following, are UB: 77 | 78 | 79 | ## Properties 80 | 81 | ### unconditional-upcast 82 | 83 | If this property holds, then... 84 | 85 | ```rust 86 | trait Foo: Bar { } 87 | 88 | let x: *const dyn Foo = ...; 89 | let y: *const dyn Bar = x; 90 | ``` 91 | 92 | can be done without any conditional operations. 93 | 94 | ### can-create-from-zeroed 95 | 96 | If this property holds, then... 97 | 98 | ```rust 99 | let x: *const dyn Foo = std::mem::zeroed(); 100 | ``` 101 | 102 | ...is not insta-UB. 103 | 104 | ### can-create-from-arbitrary-bits 105 | 106 | If this property holds, then... 107 | 108 | ```rust 109 | let x: *const dyn Foo = /* arbitrary bits */; 110 | ``` 111 | 112 | ...is not insta-UB. 113 | 114 | 115 | ### niche-for-dyn 116 | 117 | Raw pointers for sized types do not presently have niches -- but under some of these proposals, vtables could potentially act as a niche, such that `Option<*const dyn Foo>` has the same size as `*const dyn Foo`. This seems like a pretty niche (no pun intended) advantage, and is not clearly desirable, but it helps to illustrate certain things. 118 | 119 | Note that in some cases this property would be lost if we made future 'forwards compatible' changes (e.g., by moving from a more restricted variant to operations-are-ub). 120 | 121 | ## Removed proposals 122 | 123 | There were some ideas that turned out to not add value and were removed from the vtable. 124 | 125 | ### sufficiently-valid-vtable 126 | 127 | An earlier doc proposed a "sufficiently valid" vtable as the condition, where it was only legal to create a `*const dyn Foo` with a valid vtable, but you could not generally assume the vtable was valid. However, it was pointed out that because vtable layout is unstable, it's not clear what this "sufficiently valid" language adds. 128 | 129 | The original intent was to prevent unsafe code authors who receive a `*const dyn Foo` from assuming that the vtable was valid. The fear was that if we made changes that permitted a "less than fully valid" vtable, that unsafe code would now be getting fewer guaranteese than it had before. But given that vtable layouts are unstable, the code can't do much in practice, and in any case, if we *did* change the requirements, we could safe that "unsafe code which doesn't say otherwise requires a fully valid vtable". 130 | 131 | Spelling out the scenarios: 132 | 133 | 1. when I release to safe code, full validity is required, because there are safe operations that require it 134 | 2. so if I write a safe function, I can always assume full validity on input 135 | 3. if I write an unsafe fn, in Rust *now* I can assume that I get full validity (because we defined that as the invariant), but the only thing I can do with that are defined operations at present (e.g., invoke methods, etc). If in the future we define the layout, I could read data from that vtable, but since it is required to be fully valid, that seems ok. In the future, we could loosen the validity requirement, but we would say that unsafe functions must explicitly state that they accept a `*const dyn` with a less than fully valid vtable (in other words, we'd have "edition-like" treatment of validity invariants. 136 | 137 | Regardless, it's not clear what kind of "less than fully valid" vtables we would accept. Some possibilities: 138 | 139 | * Null placeholder. 140 | * A "skeleton" vtable that permits upcast but has invalid methods. Feels awfully niche, not worth it. 141 | 142 | ### -------------------------------------------------------------------------------- /LICENSE-APACHE: -------------------------------------------------------------------------------- 1 | Apache License 2 | Version 2.0, January 2004 3 | http://www.apache.org/licenses/ 4 | 5 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 6 | 7 | 1. Definitions. 8 | 9 | "License" shall mean the terms and conditions for use, reproduction, 10 | and distribution as defined by Sections 1 through 9 of this document. 11 | 12 | "Licensor" shall mean the copyright owner or entity authorized by 13 | the copyright owner that is granting the License. 14 | 15 | "Legal Entity" shall mean the union of the acting entity and all 16 | other entities that control, are controlled by, or are under common 17 | control with that entity. For the purposes of this definition, 18 | "control" means (i) the power, direct or indirect, to cause the 19 | direction or management of such entity, whether by contract or 20 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 21 | outstanding shares, or (iii) beneficial ownership of such entity. 22 | 23 | "You" (or "Your") shall mean an individual or Legal Entity 24 | exercising permissions granted by this License. 25 | 26 | "Source" form shall mean the preferred form for making modifications, 27 | including but not limited to software source code, documentation 28 | source, and configuration files. 29 | 30 | "Object" form shall mean any form resulting from mechanical 31 | transformation or translation of a Source form, including but 32 | not limited to compiled object code, generated documentation, 33 | and conversions to other media types. 34 | 35 | "Work" shall mean the work of authorship, whether in Source or 36 | Object form, made available under the License, as indicated by a 37 | copyright notice that is included in or attached to the work 38 | (an example is provided in the Appendix below). 39 | 40 | "Derivative Works" shall mean any work, whether in Source or Object 41 | form, that is based on (or derived from) the Work and for which the 42 | editorial revisions, annotations, elaborations, or other modifications 43 | represent, as a whole, an original work of authorship. For the purposes 44 | of this License, Derivative Works shall not include works that remain 45 | separable from, or merely link (or bind by name) to the interfaces of, 46 | the Work and Derivative Works thereof. 47 | 48 | "Contribution" shall mean any work of authorship, including 49 | the original version of the Work and any modifications or additions 50 | to that Work or Derivative Works thereof, that is intentionally 51 | submitted to Licensor for inclusion in the Work by the copyright owner 52 | or by an individual or Legal Entity authorized to submit on behalf of 53 | the copyright owner. For the purposes of this definition, "submitted" 54 | means any form of electronic, verbal, or written communication sent 55 | to the Licensor or its representatives, including but not limited to 56 | communication on electronic mailing lists, source code control systems, 57 | and issue tracking systems that are managed by, or on behalf of, the 58 | Licensor for the purpose of discussing and improving the Work, but 59 | excluding communication that is conspicuously marked or otherwise 60 | designated in writing by the copyright owner as "Not a Contribution." 61 | 62 | "Contributor" shall mean Licensor and any individual or Legal Entity 63 | on behalf of whom a Contribution has been received by Licensor and 64 | subsequently incorporated within the Work. 65 | 66 | 2. Grant of Copyright License. Subject to the terms and conditions of 67 | this License, each Contributor hereby grants to You a perpetual, 68 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 69 | copyright license to reproduce, prepare Derivative Works of, 70 | publicly display, publicly perform, sublicense, and distribute the 71 | Work and such Derivative Works in Source or Object form. 72 | 73 | 3. Grant of Patent License. Subject to the terms and conditions of 74 | this License, each Contributor hereby grants to You a perpetual, 75 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 76 | (except as stated in this section) patent license to make, have made, 77 | use, offer to sell, sell, import, and otherwise transfer the Work, 78 | where such license applies only to those patent claims licensable 79 | by such Contributor that are necessarily infringed by their 80 | Contribution(s) alone or by combination of their Contribution(s) 81 | with the Work to which such Contribution(s) was submitted. If You 82 | institute patent litigation against any entity (including a 83 | cross-claim or counterclaim in a lawsuit) alleging that the Work 84 | or a Contribution incorporated within the Work constitutes direct 85 | or contributory patent infringement, then any patent licenses 86 | granted to You under this License for that Work shall terminate 87 | as of the date such litigation is filed. 88 | 89 | 4. Redistribution. You may reproduce and distribute copies of the 90 | Work or Derivative Works thereof in any medium, with or without 91 | modifications, and in Source or Object form, provided that You 92 | meet the following conditions: 93 | 94 | (a) You must give any other recipients of the Work or 95 | Derivative Works a copy of this License; and 96 | 97 | (b) You must cause any modified files to carry prominent notices 98 | stating that You changed the files; and 99 | 100 | (c) You must retain, in the Source form of any Derivative Works 101 | that You distribute, all copyright, patent, trademark, and 102 | attribution notices from the Source form of the Work, 103 | excluding those notices that do not pertain to any part of 104 | the Derivative Works; and 105 | 106 | (d) If the Work includes a "NOTICE" text file as part of its 107 | distribution, then any Derivative Works that You distribute must 108 | include a readable copy of the attribution notices contained 109 | within such NOTICE file, excluding those notices that do not 110 | pertain to any part of the Derivative Works, in at least one 111 | of the following places: within a NOTICE text file distributed 112 | as part of the Derivative Works; within the Source form or 113 | documentation, if provided along with the Derivative Works; or, 114 | within a display generated by the Derivative Works, if and 115 | wherever such third-party notices normally appear. The contents 116 | of the NOTICE file are for informational purposes only and 117 | do not modify the License. You may add Your own attribution 118 | notices within Derivative Works that You distribute, alongside 119 | or as an addendum to the NOTICE text from the Work, provided 120 | that such additional attribution notices cannot be construed 121 | as modifying the License. 122 | 123 | You may add Your own copyright statement to Your modifications and 124 | may provide additional or different license terms and conditions 125 | for use, reproduction, or distribution of Your modifications, or 126 | for any such Derivative Works as a whole, provided Your use, 127 | reproduction, and distribution of the Work otherwise complies with 128 | the conditions stated in this License. 129 | 130 | 5. Submission of Contributions. Unless You explicitly state otherwise, 131 | any Contribution intentionally submitted for inclusion in the Work 132 | by You to the Licensor shall be under the terms and conditions of 133 | this License, without any additional terms or conditions. 134 | Notwithstanding the above, nothing herein shall supersede or modify 135 | the terms of any separate license agreement you may have executed 136 | with Licensor regarding such Contributions. 137 | 138 | 6. Trademarks. This License does not grant permission to use the trade 139 | names, trademarks, service marks, or product names of the Licensor, 140 | except as required for reasonable and customary use in describing the 141 | origin of the Work and reproducing the content of the NOTICE file. 142 | 143 | 7. Disclaimer of Warranty. Unless required by applicable law or 144 | agreed to in writing, Licensor provides the Work (and each 145 | Contributor provides its Contributions) on an "AS IS" BASIS, 146 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 147 | implied, including, without limitation, any warranties or conditions 148 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 149 | PARTICULAR PURPOSE. You are solely responsible for determining the 150 | appropriateness of using or redistributing the Work and assume any 151 | risks associated with Your exercise of permissions under this License. 152 | 153 | 8. Limitation of Liability. In no event and under no legal theory, 154 | whether in tort (including negligence), contract, or otherwise, 155 | unless required by applicable law (such as deliberate and grossly 156 | negligent acts) or agreed to in writing, shall any Contributor be 157 | liable to You for damages, including any direct, indirect, special, 158 | incidental, or consequential damages of any character arising as a 159 | result of this License or out of the use or inability to use the 160 | Work (including but not limited to damages for loss of goodwill, 161 | work stoppage, computer failure or malfunction, or any and all 162 | other commercial damages or losses), even if such Contributor 163 | has been advised of the possibility of such damages. 164 | 165 | 9. Accepting Warranty or Additional Liability. While redistributing 166 | the Work or Derivative Works thereof, You may choose to offer, 167 | and charge a fee for, acceptance of support, warranty, indemnity, 168 | or other liability obligations and/or rights consistent with this 169 | License. However, in accepting such obligations, You may act only 170 | on Your own behalf and on Your sole responsibility, not on behalf 171 | of any other Contributor, and only if You agree to indemnify, 172 | defend, and hold each Contributor harmless for any liability 173 | incurred by, or claims asserted against, such Contributor by reason 174 | of your accepting any such warranty or additional liability. 175 | 176 | END OF TERMS AND CONDITIONS 177 | 178 | APPENDIX: How to apply the Apache License to your work. 179 | 180 | To apply the Apache License to your work, attach the following 181 | boilerplate notice, with the fields enclosed by brackets "[]" 182 | replaced with your own identifying information. (Don't include 183 | the brackets!) The text should be enclosed in the appropriate 184 | comment syntax for the file format. We also recommend that a 185 | file or class name and description of purpose be included on the 186 | same "printed page" as the copyright notice for easier 187 | identification within third-party archives. 188 | 189 | Copyright [yyyy] [name of copyright owner] 190 | 191 | Licensed under the Apache License, Version 2.0 (the "License"); 192 | you may not use this file except in compliance with the License. 193 | You may obtain a copy of the License at 194 | 195 | http://www.apache.org/licenses/LICENSE-2.0 196 | 197 | Unless required by applicable law or agreed to in writing, software 198 | distributed under the License is distributed on an "AS IS" BASIS, 199 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 200 | See the License for the specific language governing permissions and 201 | limitations under the License. 202 | -------------------------------------------------------------------------------- /design-discussions/upcast-safety-3.md: -------------------------------------------------------------------------------- 1 | # Dyn upcast safety 2 | 3 | This document proposes a resolution to the last outstanding question blocking `dyn` upcast. 4 | 5 | Tracking issue: https://github.com/rust-lang/rust/issues/101336 6 | 7 | ## Background 8 | 9 | We are trying to enable "upcasts" from a `dyn Trait` to its supertraits: 10 | 11 | ```rust 12 | trait Foo { 13 | fn foo_method(&self); 14 | } 15 | trait Bar: Foo { 16 | fn bar_method(&self); 17 | } 18 | 19 | let x: &dyn Bar = /* ... */; 20 | let y: &dyn Foo = x; // compiles 21 | ``` 22 | 23 | The key factor for these upcasts is that they require adjusting the vtable. The current implementation strategy is that the vtables for the `Bar` trait embed pointers to a `Foo` vtable within them: 24 | 25 | ``` 26 | +----------------------------+ 27 | | Bar vtable for some type T | 28 | |----------------------------| 29 | | Foo vtable | ----------> +------------------+ 30 | | foo_method | -----+ | Foo vtable for T | 31 | | bar_method | --+ | |------------------| 32 | +----------------------------+ | | | foo_method | ---+ 33 | | | +------------------+ | 34 | | | | 35 | | +---> ::foo_method <-+ 36 | v 37 | ::bar_method 38 | 39 | (this diagram is only meant to convey the general idea of the vtable 40 | layout, and doesn't represent the exact offsets we would use etc; 41 | in fact, with the current implementation, 42 | the first supertrait is stored 'inline' and hence no load is required) 43 | ``` 44 | 45 | This way, given a `&dyn Bar` object, we convert its `Bar` vtable to the appropriate `Foo` vtable by loading the appropriate field. 46 | 47 | Although we don't want to commit to a particular implementation strategy, we do want to leave room for this strategy. One implication is that performing an upcast may require loading from the vtable, which implies that the vtable must be a valid pointer to an actual Rust vtable. Although `&dyn Bar` references would always contain a valid vtable, the same is not necessarily true for a raw pointer like `*const dyn Bar` or `*mut dyn Bar`. 48 | 49 | In the language today, we only support "noop upcasts" that don't affect the vtable, and these are safe (e.g., converting from `*const dyn Foo + Send` to `*const dyn Foo`). If we extend the set of upcasts to permit vtable-adjusting upcasts, like `*const dyn Bar` to `*const dyn Foo`, this implies that, for safe code at least, all `*const dyn Trait` values must have a valid vtable, so that we know we can safely load the required field and perform the upcast. 50 | 51 | On the other hand, we do not generally require raw `*mut T` pointers to point to valid data. In fact, we explicitly permit them to have any value, including null, and only require that they point to valid data when they are dereferenced. Because dereferencing a raw pointer is an unsafe operation, it has always been considered safe to expose an arbitrary raw pointer to unsafe code -- the unsafety arises when you take a raw pointer from an unknown source and dereference it, since unless you can trace the origin of that pointer you can't possible guarantee that it is valid to dereference. 52 | 53 | This brings us to the conflict: 54 | 55 | * It has historically been safe to "release" a raw pointer to safe code, but not safe to receive one (since you cannot know if it is valid). 56 | * It has historically been safe to upcast `*const dyn` values (e.g., `*const dyn Foo + Send` to `*const dyn Foo`). 57 | * Unlike the upcasts we are considering now, this upcast does not require changing the vtable at runtime, but the distinction is subtle for end-users. 58 | * Moreover, there are future extensions (e.g., upcasting `*const dyn Foo + Bar` to `*const dyn Foo`) that would require adjusting the vtable much like the upcasts currently being stabilized. 59 | 60 | ## Related future consideration: virtual method calls on raw pointers 61 | 62 | There have been requests to extend traits with the option to include raw pointer methods: 63 | 64 | ```rust 65 | trait PtrLike { 66 | fn is_null(v: *const Self) -> bool; 67 | } 68 | ``` 69 | 70 | These methods would be useful when writing unsafe code because having an `&self` method requires satisfying the validity conditions of an `&`-reference, which may not be possible. If we did have such methods, however, it raises the question of whether it would be safe to invoke `is_null` on a `*const dyn PtrLike` reference. Just as with upcasting, invoking a method from the vtable requires loading from the vtable, and hence requires a valid vtable generated by the compiler. 71 | 72 | The solution we propose in this document also resolves this future dilemma. 73 | 74 | ## Definitions: validity vs safety invariant 75 | 76 | We adopt the terms *validity* and *safety* invariant from the [unsafe code guidelines](https://rust-lang.github.io/unsafe-code-guidelines/glossary.html): 77 | 78 | > The *validity invariant* is an invariant that all data must uphold any time it is accessed or copied in a typed manner. This invariant is known to the compiler and exploited by optimizations such as improved enum layout or eliding in-bounds checks. 79 | > 80 | > The *safety invariant* is an invariant that safe code may assume all data to uphold. This invariant is used to justify which operations safe code can perform. The safety invariant can be temporarily violated by unsafe code, but must always be upheld when interfacing with unknown safe code. It is not relevant when arguing whether some program has UB, but it is relevant when arguing whether some code safely encapsulates its unsafety -- in other words, it is relevant when arguing whether some library is sound. 81 | 82 | In short, the *validity invariant* defines a condition that must always be true, even in unsafe code, and the *safety invariant* defines an invariant that unsafe code must guarantee before a value can be released to be used by arbitrary code. 83 | 84 | ## Contours of the solution space 85 | 86 | We can fix this conflict in one of two basic ways: 87 | 88 | First, **we could make vtable-adjusting upcasts casts (and `*Self` method calls) unsafe**. This is difficult to implement and would require changes to the `Coerce` trait, which is already excessively complicated. In exchange, it offers at best marginal benefit: raw `*dyn` pointers can be released to safe code, but safe code can't do anything interesting with them. For this reason, we do not recommend this option. 89 | 90 | If vtable-adjusting casts (and `*Self` method calls) are safe, then the **safety invariant** for `*dyn` types must be that their metadata points to a **fully valid vtable** (i.e., a vtable created by the compiler). This ensures safe code can perform upcasts or dynamic dispatch. This also implies that `std::ptr::null` (which is safe) cannot be extended to `T` where `T: ?Sized` unless further changes are made, since we would need to provide a valid vtable (it would be possible to permit a sentinel value, like null, to be used, but that would imply that upcasting must have a branch, making it less efficient). 91 | 92 | There are, however, various options for the **validity invariant**, ranging from no invariant to requiring a fully valid vtable at all times. The strict invariants offer some benefits, such as the ability to have a niche for `*dyn` pointers. We survey the options here: 93 | 94 | | Validity invariant for `*dyn` metadata | Supports niche | Can be initialized with `std::mem::zeroed` | Constant initializer | 95 | | -------------------------------------- | -------------- | ------------------------------------------ | -------------------- | 96 | | None | | ✅ | ✅ | 97 | | Word-aligned | ✅ | ✅ | ✅ | 98 | | Word-aligned, non-null | ✅ | | ✅ | 99 | | Valid vtable | ✅ | | | 100 | 101 | Explanations for the column titles: 102 | 103 | * Validity invariant for `*dyn` metadata -- describes the invariant that applies to the metadata for a `*dyn` value. 104 | * Supports niche -- true if there is a niche value so that `sizeof(Option<*const dyn Foo>) == sizeof(*const dyn Foo)`. 105 | * Can be initialized with `std::mem::zeroed` -- true if `std::mem::zeroed` can be used to create a valid value (very convenient). This makes it trivial to innitialize a `*const dyn` with a dummy value, though the value cannot be released to safe code. 106 | * Constant initializer -- true if there is some constant value for `*const dyn Foo` that satisfies the validity invariant, no matter the trait `Foo`. This makes it easy to initialize a `*const dyn` with a dummy value, though the value cannot be released to safe code. 107 | 108 | Other points: 109 | 110 | * `*dyn` values [currently have a niche](https://play.rust-lang.org/?version=nightly&mode=debug&edition=2021&gist=bcd18cd3870ba98e2c59b527418f8f68). 111 | * Other pointer-like values (such as `fn` and `&`-references) are expected to have a validity invariant of word-aligned, non-null. 112 | * Whichever option we use, we can backwards-compatibly move "up the table" and adopt a less-strict validity invariant without introducing UB into any extant programs. 113 | 114 | ## Proposal 115 | 116 | The proposal is as follows: 117 | 118 | * Vtable-adjusting upcasts are **safe operations**. The upcast is UB if performed on a value without a valid vtable 119 | * As such, the "safety invariant" requires a **fully valid vtable**. 120 | * The "validity invariant" requires `*dyn` metadata to be **word-aligned and non-null**. 121 | 122 | Vtable-adjusting upcasts are defined as: 123 | 124 | * Trait upcasts that alter the set of methods that can be invoked on the resulting value at runtime (e.g., `dyn Bar` to `dyn Foo` from the introduction). In particular, upcasts that simply add or remove auto-traits are not vtable-adjusting (e.g., `dyn Debug + Send` to `dyn Debug`). 125 | 126 | This approach... 127 | 128 | * permits safe upcasting (and method calls, in the future); 129 | * preserves the existing niche for `*const dyn`; 130 | * is consistent with other validity requirements for "pointer-like" things such as `fn`; 131 | 132 | The rules also imply that... 133 | 134 | * valid (i.e., compiler-generated) vtables are only required for a `*dyn` pointer when 135 | * the `*dyn` pointer is upcast (or invoke methods); 136 | * or, when the `*dyn` pointer is released to arbitrary code, because that code may upcast (or invoke methods). 137 | * By implication, extending [`std::ptr::null`](https://doc.rust-lang.org/std/ptr/fn.null.html) to permit `T: ?Sized` would not be safe. 138 | 139 | Possible future changes... 140 | 141 | * without introducing UB, we could loosen the validity invariant if desired, for example to permit NULL as a valid value in unsafe code. 142 | * without introducing UB, we could also loosen the safety invariant to permit a sentinel value (such as NULL), but that would require a branch or other check in the upcast code, which would be less efficient. 143 | 144 | ## Prior links 145 | 146 | * [Dyn safety write-up](https://rust-lang.github.io/dyn-upcasting-coercion-initiative/design-discussions/upcast-safety-2.html), which includes links to prior write-ups 147 | * [Exhaustive set of validity invariants considered](./upcast-safety-3-options.md) -------------------------------------------------------------------------------- /design-discussions/upcast-safety-2.md: -------------------------------------------------------------------------------- 1 | # Dyn upcast safety 2 | 3 | **NB: This document is outdated.** [Click here to see the latest version.](./upcast-safety-3.md) 4 | 5 | ## Summary 6 | 7 | This doc concerns possible resolutions for the following potential problem: 8 | 9 | * Given: 10 | * unsize coercions of raw pointers are legal and stable in safe code 11 | * upcasting may require loading data from vtable 12 | * If: 13 | * dyn upcast coercions are a kind of unsize coercion 14 | * the [safety invariant][vvsi] for `*const dyn Foo` permits an invalid vtable 15 | * Then: 16 | * safe code could cause UB by upcasting a `*const dyn SubTrait` to a `*const dyn SuperTrait` 17 | 18 | The doc recommends: 19 | 20 | * Defining the safety invariant for a `*const dyn Foo` to be a "sufficiently valid" vtable for `Foo` 21 | * the definition of "sufficiently valid" is intentionally left undefined except to say: 22 | * a fully valid vtable is also "sufficiently valid" 23 | * it permits upcasting from `*const dyn SubTrait` to `*const dyn SuperTrait` without UB 24 | * this implies that the only way for users to produce a "sufficiently valid" vtable is with a fully valid one, but conversely safe code cannot rely on a `*const dyn Foo` having a *fully* valid vtable 25 | 26 | ## Background 27 | 28 | ### Unsize coercions are possible in safe code 29 | 30 | A brief review of the [DST coercion design](https://github.com/rust-lang/rfcs/blob/master/text/0982-dst-coercion.md): 31 | 32 | * The `PT: CoerceUnsized` trait indicates that a smart pointer `PT` can be coerced to a smart pointer `PU` 33 | * This trait is manually implemented for each smart pointer type, e.g. `impl CoerceUnsized> for Rc where T: Unsize`. 34 | * The compiler requires that `Rc -> Rc` be a legal DST coercion per various built-in rules (e.g., `T` must be stored in the final field of the struct and cannot be behind another pointer indirection). 35 | * Example: `Rc: CoerceUnsized>` 36 | * The `T: Unsize` trait indicates that the `T: ?Sized` referent can be "unsized" to the reference `U` 37 | * This trait is automatically implemented by the compiler. 38 | * Example: `String: Unsize` 39 | * Example: `[String; 32]: Unsize<[String]>` 40 | 41 | This design permits one to write generic code that performances unsized coercions, but the trait is unstable so this only works on nightly: 42 | 43 | ```rust 44 | // Unstable: 45 | fn foo(pt: PT) -> PU 46 | where 47 | PT: CoerceUnsized, 48 | { 49 | pt 50 | } 51 | ``` 52 | 53 | [Example](https://play.rust-lang.org/?version=nightly&mode=debug&edition=2021&gist=0b3aef50e32fb224bc053f7ef4640e94) 54 | 55 | You can however coerce from a `*const T` to a `*const dyn Trait` in safe code on stable ([playground](https://play.rust-lang.org/?version=nightly&mode=debug&edition=2021&gist=695eeb19aa21e4762b9f5025fea7176c 56 | )): 57 | 58 | ```rust 59 | fn main() { 60 | let x: *const u32 = &32; 61 | let y: *const dyn std::fmt::Debug = x; 62 | } 63 | ``` 64 | 65 | ### Upcasting `dyn SubTrait` to `dyn SuperTrait` is considered an unsize coercion 66 | 67 | We implemented upcasting from one dyn trait to a supertrait as an upsizing coercion. This means that `dyn SubTrait: Unsize`. This is a coercion because it requires adjusting the vtable. 68 | 69 | ### How upcasting coercions adjust the vtable 70 | 71 | The vtable for a `dyn SubTrait` [now embeds pointers](https://rust-lang.github.io/dyn-upcasting-coercion-initiative/design-discussions/vtable-layout.html) to the vtables for super traits. Upcasting therefore requires loading the new vtable for the supertrait from the specific slot within the subtrait's table. 72 | 73 | There are alternatives we could use such that upcasting would be a pure integer adjustment with no load, but that would be less efficient in terms of space usage. 74 | 75 | Although not directly relevant here, there is another operation that requires accessing the vtable, which is finding the offset of fields -- see [the discussion in the appendices](#mfo) for details. This operation is however only permitted in unsafe code because it requires a dereference of the raw pointer. 76 | 77 | ### Ergo, upcasting raw pointers is possible in safe code 78 | 79 | Without any further changes, the following upcasting is legal in safe Rust ([playground](https://play.rust-lang.org/?version=nightly&mode=debug&edition=2021&gist=37e49f69ebf992856b07a430c4d5510f)): 80 | 81 | ```rust 82 | #![feature(trait_upcasting])] 83 | 84 | trait Sub: Sup { } 85 | trait Sup { } 86 | 87 | fn convert(x: *const dyn Sub) -> *const dyn Sup { 88 | x 89 | } 90 | ``` 91 | 92 | 93 | ### Raw pointers are traditionally permitted to be "garbage" 94 | 95 | A sized raw pointer like `*const u32` has no validity/safety invariants to speak of. It is not required to be aligned, it may be null, and it may point at arbitrary memory. This is why `Option<*const u32>` requires an extra word fo the discriminant and it is why dereferencing a raw pointer is unsafe. 96 | 97 | However, as noted in the previous section: 98 | 99 | * if upcasting raw pointers is possible in safe code and 100 | * if upcasting requires loading data from the vtable 101 | 102 | then the safety condition of `*const dyn Foo` must include a "sufficiently valid" vtable. Sufficiently valid means that it is "structurally complete" in that it contains valid pointers that can be loaded to do upcasting. 103 | 104 | ### What do we do for null or garbage data? 105 | 106 | The challenge with `*const dyn Trait` is "what do we do to make a null pointer"? For better or worse, `*const T` pointers can traditionally be null: but when one creates a vtable, one is supposed to have some underlying data type to make the vtable for, and with a null pointer that is not possible. 107 | 108 | ### Creating a `*const dyn` 109 | 110 | If we wish to ensure a safety invariant for `*const dyn` values, we have to ask ourselves how one could go about producing such a value. There are currently two ways to create a `*const dyn Trait`, one safe and one unsafe: 111 | 112 | * [`std::ptr::from_raw_parts`](https://doc.rust-lang.org/std/ptr/fn.from_raw_parts.html) -- safe, unstable 113 | * [`std::mem::transmute`](https://doc.rust-lang.org/std/mem/fn.transmute.html) -- unsafe, unwise 114 | 115 | There is also a third way to make a `*const T` that doesn't work for `dyn` now but which could be made to work: 116 | 117 | * [`std::ptr::null`](https://doc.rust-lang.org/std/ptr/fn.null.html) -- safe, stable, but currently limited only to `*const T` where `T: Sized` 118 | 119 | Of these three: 120 | 121 | * `from_raw_parts` requires a valid vtable for `T` as argument, so it would meet the safety requirement 122 | * `transmute` is unsafe, but it would indeed be a requirement that users of transmute must uphold 123 | * `null`, if extended to unsized types, would be tricky -- we would need to have some way to get a "dummy" vtable that is structurally sound enough to permit upcasting, but which has (for example) null pointers for the actual methods. This is, however, eminently doable. 124 | 125 | ### Interaction: Raw pointer method calls 126 | 127 | It would be useful if unsafe code could declare `*const self` and `*mut self` methods in traits 128 | 129 | ```rust 130 | pub trait FreeMe { 131 | pub unsafe fn free_me(*const self); 132 | } 133 | ``` 134 | 135 | Given that using `&self` implies (for ex 136 | 137 | ## Core decision to be made 138 | 139 | The core decision to be made is to choose between two paths: 140 | 141 | * Splitting out "unsafe" unsizing operations from safe ones; 142 | * Unsizing operations on safe pointers like `&dyn SubTrait` would continue to work as they do today. 143 | * Implicit unsizing operations on raw pointers like `*const dyn SubTrait` would work sometimes, but those unsizing operations that require the metadata meet some sort of condition would require an explicit function call (e.g. the proposal below adds an unsafe function `unsize_raw_pointer`). 144 | * Raw pointers have a validity or safety invariant that puts conditions on metadata 145 | * There are options in terms of when this invariant must hold and how strict the invariant must be, but the upshot is that whenever you synthesize the metadata for a raw, wide pointer (e.g., from a transmute), you need to ensure that this metadata comes from some valid source, and is not just garbage. This is contrast to the data pointer, which can generally be arbitrary bytes (though not unininitialized). 146 | 147 | ### Why prefer unsafe raw pointer upcasting? 148 | 149 | Raw pointers have traditionally been viewed as "untrusted data until they are used". This is why dereferencing a raw pointer is unsafe: that's the point where it must be valid. It makes sense to extend this model to the metadata as well. When working with raw pointers and unsafe code, implicit operations like upcasting are a bug, not a feature, so it's useful to segregate them out and make them explicit via some kind of function call. It does require adding a new trait (see below) to the unsizing mechanism, but it only requires an internal "implementation" trait, and doesn't affect the "main trait" (`CoerceUnsized`). 150 | 151 | Besides, so long as the validity/safety invariants remain in flux (which they will be for a while yet), this choice is forwards compatible with the others. We have the option to remove the "unsafe upcast" and merge it with safe upcast and strengthen the relevant invariant(s). 152 | 153 | ### Why prefer unsafe some form of invariant? 154 | 155 | It's not clear why one would ever have invalid metadata in a wide pointer to start with; it's not an easy thing to do, you have to basically transmute from random bytes (e.g., zeroed memory). If you want to have a garbage pointer that is not yet initialized, use `MaybeUninit`. If you want a null pointer, use `ptr::null` or `Option>`. In exchange for following these best practices, you get two things: 156 | 157 | * Safe unsafe code overall, since you are being clearer about your intentions. 158 | * A simpler coercion model, with fewer traits and moving parts, and things that work the same for all kinds of pointers 159 | * A language that is 160 | 161 | ## Options 162 | 163 | The following options have been identified. The preferred solution is not yet clear. 164 | 165 | ### RawUnsafe: Make raw pointer upcasting unsafe (not possible once `dyn SubTrait: Unsize` is stable) 166 | 167 | We could permit "safe pointer" upcasting but make raw pointer upcasting unsafe. This would require changing the design of the coercion traits somewhat. We would introduce a new "magic trait" `RawUnsize`, analogous to `Unsize` except that it doesn't permit upcasting or any other operations that could require valid metadata. We would then modify the impl of `CoerceUnsized` for raw pointers to be: 168 | 169 | ```rust 170 | impl CoerceUnsized<*const U> for *const T where 171 | T: Unsize + ?Sized, 172 | U: ?Sized, 173 | ``` 174 | 175 | and it would become 176 | 177 | 178 | ```rust 179 | impl CoerceUnsized<*const U> for *const T where 180 | T: RawUnsize + ?Sized, 181 | U: ?Sized, 182 | ``` 183 | 184 | To support upcasting on raw pointers, we can then introduce some other intrinsic for doing raw pointer upcast, such as something like this (modulo stacked borrows, which this *particular* definition may invalidate): 185 | 186 | ```rust 187 | /// Unsafefy condition: If the metadata for `T` must be valid. 188 | pub unsafe fn unsize_raw_pointer(t: *const T) -> *const U 189 | where 190 | T: Unsize, 191 | { 192 | let (_, t_metadata) = t.into_raw_parts(); 193 | unsafe { &*t } 194 | } 195 | ``` 196 | 197 | So long as `Unsize` remains a strict superset of `RawUnsize`, we could change things in the future to make `RawUnsize` an alias for `Unsize` (or, if it is unstable, remove it altogether) and thus deprecate the `unsize_raw_pointer` function. This is therefore forwards compatible with the preferred proposal here as well as other things that say "still possible in the future". 198 | ### VISufficientlyValid: Extend the validity invariant of `dyn Trait` to require a "sufficiently valid" vtable 199 | 200 | One solution is to extend the [validity invariant][vvsi] for raw pointers to require a "sufficiently valid" vtable. We don't specify the precise condition that makes a vtable "sufficiently valid" except to say that a fully valid vtable is "sufficiently valid", and that a "sufficiently valid" vtable permits dyn upcasting without UB. 201 | 202 | Implications: 203 | 204 | * Whenever one creates a wide pointer, one must ensure that the metadata is "sufficiently valid": 205 | * For `*const dyn Trait`, this would mean that one must use a valid vtable for `Trait`. 206 | * In particular, we don't define what is "sufficiently valid" so you have to use something that is fully valid; at the same time, you cannot rely on the vtable for `*const dyn Trait` being fully valid yourself, only "sufficiently valid" (which is "valid enough for upcast" and that's it). 207 | * If the pointer is not initialized, and hence you don't know which vtable to use, you have the following options: 208 | * Use a dummy vtable for any type, it doesn't matter which. 209 | * Use `MaybeUninit<*const dyn Foo>`, in which case no safety invariant is assumed. 210 | * Use `Option<*const dyn Foo>` and `None` instead of null: safer, wastes space. 211 | * Use `Option>` and `None` instead of null: safer, generally better, perhaps less ergonomic. 212 | 213 | One downside of this proposal is that the validity invariant is stricter than is needed: that is, the purpose of the validity invariant is primarily to enable the compiler's ability to perform layout optimizations. This rule would enable the compiler to silently insert upcasting operations if it needed to do so, but it's not clear why it would need to do that spontaneously: those operations are always tied to something else (e.g., a coercion or a method call). Therefore, the safety invariant might seem like a better fit. 214 | 215 | ### SISufficientlyValid: Extend the safety invariant of `dyn Trait` to require a "sufficiently valid" vtable 216 | 217 | As an alternative to modifying the validity invariant, we could modify the [safety invariant][vvsi] for wide pointers to include "sufficiently valid" metadata (see VISufficientlyValid for details). 218 | 219 | 220 | Implications: 221 | 222 | * Whenever one performs an upcast or other operation with a wide pointer, one must ensure that the metadata is "sufficiently valid": 223 | * If the pointer is not initialized, and hence you don't know which vtable to use, you have the same options as described under VISufficientlyValid. 224 | 225 | The primary downsice of this proposal versus VISufficientlyValid is that the causes of UB are rather more subtle. Instead of UB occurring when the pointer is created, it occurs when an upcast occurs, and the locations for upcasts can be implicit (eg., any assignment or function call). Consider the following example: 226 | 227 | ```rust 228 | fn noop(x: *mut dyn SuperTrait) { 229 | /* look ma, empty body */ 230 | } 231 | 232 | fn creator() { 233 | // No UB yet: the metadata for `x` is not sufficiently valid, 234 | // but we haven't done anything with it yet. 235 | let x: *mut dyn SubTrait = unsafe { std::mem::zeroed() }; 236 | 237 | // `y = x` does not trigger UB, just a copy. 238 | let y = x; 239 | 240 | // UB! Here there is a coercion. 241 | noop(y); 242 | } 243 | ``` 244 | 245 | For this reason, it would probably be "best practice" to treat this condition "as if" it were part of the validity invariant. 246 | 247 | ### SIFullyValid: Extend safety condition to require a "fully valid" vtable (still possible in the future) 248 | 249 | This would permit safe code to read values from the vtable of a `*const dyn Trait` without any unsafety (just as it does for upcasting). It's not clear why we would want to permit this, and it may foreclose useful options in the future. 250 | 251 | Adopting this option remains a possibility in the future, as it would be a backwards compatible extension to the above rule. 252 | 253 | ### SIStructValid: Extend safety condition to require a "structurally valid" vtable (still possible in the future) 254 | 255 | Instead of requiring a valid vtable, we could require a "structurally valid" vtable. This vtable would have null pointers for all methods as well as a dummy type-id but would have the same "structure" as an ordinary vtable. There would be an intrinsic that gives access to the vtable: 256 | 257 | ```rust 258 | fn dummy_vtable::() -> T::Metadata 259 | ``` 260 | 261 | This would permit one to represent an uninitialized dyn pointer as `*const dyn Foo` and use the dummy-vtable for its metadata. This could be convenient, but is less typesafe than `MaybeUnit` or `Option` and not obviously better. 262 | 263 | Adopting this option remains a possibility in the future, as it would be a backwards compatible extension to the above rule. 264 | 265 | ### NullVtable: Special case the vtable for null (still possible in the future) 266 | 267 | Instead of saying that a "null dyn pointer" must have a structurally sound vtable, we could permit null as the value for the vtable. This would require an "if branch" or some kind of more complex logic in the upcasting path, since we couldn't unconditionally do a load. That might be acceptable, but it seems silly to slow down the upcasting path for a relatively unusual case of having a null 268 | 269 | ### FlatVtable: Adjust vtable layout to not require a load (still possible in the future) 270 | 271 | We could adjust the vtable layout for a subtrait to include embedded copies of all supertraits. This way the upcast is a pure offset operation and does not require a load. This would be less efficient in terms of space usage. We generally prefer not to limit the possible vtable designs that an implementation can use unless we have to, so as to leave room for future developments. 272 | 273 | ## Appendices 274 | 275 | [vvsi]: #validity-versus-safety-invariants 276 | 277 | ### Validity versus safety invariants 278 | 279 | Let us take a digression to cover Ralf's distinction of [validity vs safety invariants](https://www.ralfj.de/blog/2018/08/22/two-kinds-of-invariants.html): 280 | 281 | * The **validity invariant** for a type `T` defines the invariant that **all values of type T must meet, all of the time**. These invariants are respected in both safe and unsafe code and are primarily used to do layout optimizations. 282 | * For example, the validity invariant of `bool` requires that the value is 0 or 1 but not 2. Thanks to this validity invariant, `Option` can be represented by the compiler in the same amount of space as `bool`. 283 | * The **safety invariant** for a type `T` defines the invariant that **all values of type `T` must meet when given to safe code**. These invariants are used to justify unsafe code, but aren't understood by the compiler. 284 | * For example, a vector has a field for its length and capacity, and the safety invariant requires that they accurately describe the allocate space available in the vector's buffer (and aren't just random values). Thanks to this safety invariant, we can create `push` as a safe function: it can read those fields and rely on them being accurate to decide whether the memory still has any free capacity. 285 | 286 | [mfo]: #metadata-and-field-offsets 287 | 288 | ### Metadata and field offsets 289 | 290 | Consider these struct definitions: 291 | 292 | ```rust 293 | struct RefCounted { 294 | counter: usize, 295 | data: T, 296 | } 297 | 298 | struct Foo { 299 | field: u32, 300 | data: U 301 | } 302 | ``` 303 | 304 | and now assume that I have a `*const RefCounted>` which is coerced: 305 | 306 | ```rust 307 | let pointer1: *const RefCounted> = ...; 308 | let pointer2: *const RefCounted> = pointer1; 309 | ``` 310 | 311 | If I now try to get the address of the field `field`... 312 | 313 | ```rust 314 | let pointer3: *const u32 = unsafe { raw_addr!((*pointer2).data.field) }; 315 | ``` 316 | 317 | ...this operation requires valid metadata. This is because the offset of `field` is a function of the alignment of `Foo`, which is a function of the alignment of `U`. In these cases, the compiler will load the alignment from the vtable and do the required address computations itself. --------------------------------------------------------------------------------