├── .github ├── ISSUE_TEMPLATE │ ├── -shiny-vision--story-issue.md │ ├── -status-quo--story-issue.md │ ├── meeting-proposal.md │ ├── new-roadmap-goal.md │ └── new-roadmap-initiative.md └── workflows │ └── ci.yml ├── .gitignore ├── Cargo.lock ├── Cargo.toml ├── LICENSE-APACHE ├── LICENSE-MIT ├── README.md ├── book.toml ├── get_contributors.sh ├── mermaid-init.js ├── mermaid.min.js ├── rfc-drafts ├── must_not_await_lint.md └── stream.md ├── src ├── CHARTER.md ├── LICENSE-APACHE ├── LICENSE-MIT ├── SUMMARY.md ├── acknowledgments.md ├── conversations.md ├── conversations │ └── 2021-02-12-Twitter-Thread.md ├── design_docs.md ├── design_docs │ ├── async_closures.md │ ├── async_drop.md │ ├── async_fn_in_traits.md │ ├── async_lifecycle.md │ ├── async_main.md │ ├── async_read_write.md │ ├── async_stack_traces.md │ ├── channels.md │ ├── completion_based_futures.md │ ├── generator_syntax.md │ ├── join.md │ ├── mutex.md │ ├── select.md │ ├── sink_trait.md │ ├── stream.md │ └── yield_safe.md ├── glossary.md ├── meetings.md ├── triage.md ├── vision.md ├── vision │ ├── characters.md │ ├── characters │ │ ├── alan.md │ │ ├── barbara.md │ │ ├── grace.md │ │ └── niklaus.md │ ├── how_it_feels.md │ ├── how_to_vision.md │ ├── how_to_vision │ │ ├── awards.md │ │ ├── comment.md │ │ ├── evaluations.md │ │ ├── owners.md │ │ ├── shiny_future.md │ │ ├── stakeholders.md │ │ └── status_quo.md │ ├── projects.md │ ├── projects │ │ ├── DistriData.md │ │ ├── MonsterMesh.md │ │ ├── SLOW.md │ │ ├── TrafficMonitor.md │ │ ├── YouBuy.md │ │ └── template.md │ ├── roadmap.md │ ├── roadmap │ │ ├── async_fn.md │ │ ├── async_fn │ │ │ ├── async_main_and_tests.md │ │ │ └── boxable.md │ │ ├── async_iter.md │ │ ├── async_iter │ │ │ ├── generators.md │ │ │ └── traits.md │ │ ├── async_overloading.md │ │ ├── documentation.md │ │ ├── documentation │ │ │ └── async_book.md │ │ ├── polish.md │ │ ├── polish │ │ │ ├── error_messages.md │ │ │ ├── lint_blocking_fns.md │ │ │ ├── lint_large_copies.md │ │ │ ├── lint_must_not_suspend.md │ │ │ ├── precise_generator_captures.md │ │ │ ├── stacktraces.md │ │ │ └── sync_and_async.md │ │ ├── portable.md │ │ ├── portable │ │ │ ├── read_write.md │ │ │ ├── runtime.md │ │ │ ├── spawn.md │ │ │ └── timers.md │ │ ├── scopes.md │ │ ├── scopes │ │ │ ├── capability.md │ │ │ ├── capability │ │ │ │ ├── variant_async_trait.md │ │ │ │ └── variant_leak.md │ │ │ └── scope_api.md │ │ ├── testing.md │ │ ├── threadsafe_portability.md │ │ ├── tooling.md │ │ └── tooling │ │ │ ├── console.md │ │ │ └── crashdump.md │ ├── shiny_future.md │ ├── shiny_future │ │ └── users_manual.md │ ├── status_quo.md │ ├── submitted_stories.md │ ├── submitted_stories │ │ ├── shiny_future.md │ │ ├── shiny_future │ │ │ ├── alan_learns_async_on_his_own.md │ │ │ ├── alan_switches_runtimes.md │ │ │ ├── alans_trust_in_the_compiler_is_rewarded.md │ │ │ ├── barbara_appreciates_performance_tools.md │ │ │ ├── barbara_enjoys_an_async_sandwich.md │ │ │ ├── barbara_makes_a_wish.md │ │ │ ├── barbara_wants_async_rw.md │ │ │ ├── barbara_wants_async_tracing.md │ │ │ ├── grace_debugs_a_crash_dump_again.md │ │ │ └── template.md │ │ ├── status_quo.md │ │ └── status_quo │ │ │ ├── alan_builds_a_cache.md │ │ │ ├── alan_builds_a_task_scheduler.md │ │ │ ├── alan_creates_a_hanging_alarm.md │ │ │ ├── alan_finds_database_drops_hard.md │ │ │ ├── alan_has_an_event_loop.md │ │ │ ├── alan_hates_writing_a_stream.md │ │ │ ├── alan_iteratively_regresses.md │ │ │ ├── alan_lost_the_world.md │ │ │ ├── alan_misses_c_sharp_async.md │ │ │ ├── alan_needs_async_in_traits.md │ │ │ ├── alan_picks_web_server.md │ │ │ ├── alan_runs_into_stack_trouble.md │ │ │ ├── alan_started_trusting_the_rust_compiler_but_then_async.md │ │ │ ├── alan_thinks_he_needs_async_locks.md │ │ │ ├── alan_tries_a_socket_sink.md │ │ │ ├── alan_tries_processing_some_files.md │ │ │ ├── alan_tries_to_debug_a_hang.md │ │ │ ├── alan_wants_prefetch_iterator.md │ │ │ ├── alan_writes_a_web_framework.md │ │ │ ├── aws_engineer.md │ │ │ ├── aws_engineer │ │ │ ├── borrow_check_errors.md │ │ │ ├── coming_from_java.md │ │ │ ├── debugging_performance_problems.md │ │ │ ├── ecosystem.md │ │ │ ├── encountering_pin.md │ │ │ ├── failure_to_parallelize.md │ │ │ ├── figuring_out_the_best_option.md │ │ │ ├── getting_ready_to_deploy.md │ │ │ ├── getting_started_with_rust.md │ │ │ ├── juggling_error_handling.md │ │ │ ├── missed_waker_leads_to_lost_performance.md │ │ │ ├── solving_a_deadlock.md │ │ │ ├── testing_the_service.md │ │ │ ├── using_jni.md │ │ │ ├── using_the_debugger.md │ │ │ └── writing_a_java_based_service.md │ │ │ ├── barbara_anguishes_over_http.md │ │ │ ├── barbara_battles_buffered_streams.md │ │ │ ├── barbara_benchmarks_async_trait.md │ │ │ ├── barbara_bridges_sync_and_async.md │ │ │ ├── barbara_builds_an_async_executor.md │ │ │ ├── barbara_carefully_dismisses_embedded_future.md │ │ │ ├── barbara_compares_some_cpp_code.md │ │ │ ├── barbara_gets_burned_by_select.md │ │ │ ├── barbara_makes_their_first_steps_into_async.md │ │ │ ├── barbara_needs_async_helpers.md │ │ │ ├── barbara_plays_with_async.md │ │ │ ├── barbara_polls_a_mutex.md │ │ │ ├── barbara_tries_async_streams.md │ │ │ ├── barbara_tries_unix_socket.md │ │ │ ├── barbara_trims_a_stacktrace.md │ │ │ ├── barbara_wants_async_insights.md │ │ │ ├── barbara_wants_single_threaded_optimizations.md │ │ │ ├── barbara_wants_to_use_ghostcell.md │ │ │ ├── barbara_wishes_for_easy_runtime_switch.md │ │ │ ├── barbara_writes_a_runtime_agnostic_lib.md │ │ │ ├── grace_debugs_a_crash_dump.md │ │ │ ├── grace_deploys_her_service.md │ │ │ ├── grace_tries_new_libraries.md │ │ │ ├── grace_waits_for_gdb_next.md │ │ │ ├── grace_wants_a_zero_copy_api.md │ │ │ ├── grace_wants_to_integrate_c_api.md │ │ │ ├── grace_writes_a_new_fb_service.md │ │ │ ├── niklaus_simulates_hydrodynamics.md │ │ │ ├── niklaus_wants_to_share_knowledge.md │ │ │ └── template.md │ ├── unresolved_questions.md │ └── unresolved_questions │ │ ├── async_fn.md │ │ ├── await_or_not.md │ │ ├── cancellation.md │ │ ├── default_runtime.md │ │ └── portable_without_generics.md └── welcome.md ├── tools └── fixlinks │ ├── .gitignore │ ├── Cargo.toml │ └── src │ └── main.rs └── triagebot.toml /.github/ISSUE_TEMPLATE/-shiny-vision--story-issue.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: '"Shiny future" story issue' 3 | about: Propose a "shiny future" story idea or look for someone to write it 4 | title: '' 5 | labels: good first issue, help wanted, shiny-vision-story-ideas 6 | assignees: '' 7 | 8 | --- 9 | 10 | ### Brief summary 11 | 12 | *Give a brief summary of the story or vision here* 13 | 14 | ### Optional details 15 | 16 | * (Optional) Which [character(s)] would be the best fit and why? 17 | * [ ] [Alan]: the experienced "GC'd language" developer, new to Rust 18 | * [ ] [Grace]: the systems programming expert, new to Rust 19 | * [ ] [Niklaus]: new programmer from an unconventional background 20 | * [ ] [Barbara]: the experienced Rust developer 21 | * (Optional) Which [project(s)] would be the best fit and why? 22 | * *List some projects here.* 23 | * (Optional) What are the key points or morals to emphasize? 24 | * *Write some morals here.* 25 | 26 | [character(s)]: https://rust-lang.github.io/wg-async/vision/characters.html 27 | [project(s)]: https://rust-lang.github.io/wg-async/vision/projects.html 28 | [Alan]: https://rust-lang.github.io/wg-async/vision/characters/alan.html 29 | [Grace]: https://rust-lang.github.io/wg-async/vision/characters/grace.html 30 | [Niklaus]: https://rust-lang.github.io/wg-async/vision/characters/niklaus.html 31 | [Barbara]: https://rust-lang.github.io/wg-async/vision/characters/barbara.html 32 | -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/-status-quo--story-issue.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: '"Status quo" story issue' 3 | about: Propose a "status quo" story idea or look for someone to write it 4 | title: '' 5 | labels: good first issue, help wanted, status-quo-story-ideas 6 | assignees: '' 7 | 8 | --- 9 | 10 | ### Brief summary 11 | 12 | *Give a brief summary of the story or experience here* 13 | 14 | ### Optional details 15 | 16 | * (Optional) Which [character(s)] would be the best fit and why? 17 | * [ ] [Alan]: the experienced "GC'd language" developer, new to Rust 18 | * [ ] [Grace]: the systems programming expert, new to Rust 19 | * [ ] [Niklaus]: new programmer from an unconventional background 20 | * [ ] [Barbara]: the experienced Rust developer 21 | * (Optional) Which [project(s)] would be the best fit and why? 22 | * *List some projects here.* 23 | * (Optional) What are the key points or morals to emphasize? 24 | * *Write some morals here.* 25 | 26 | [character(s)]: https://rust-lang.github.io/wg-async/vision/characters.html 27 | [project(s)]: https://rust-lang.github.io/wg-async/vision/projects.html 28 | [Alan]: https://rust-lang.github.io/wg-async/vision/characters/alan.html 29 | [Grace]: https://rust-lang.github.io/wg-async/vision/characters/grace.html 30 | [Niklaus]: https://rust-lang.github.io/wg-async/vision/characters/niklaus.html 31 | [Barbara]: https://rust-lang.github.io/wg-async/vision/characters/barbara.html 32 | -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/meeting-proposal.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Meeting proposal 3 | about: Propose a topic for a meeting. 4 | title: "(My meeting proposal)" 5 | labels: meeting-proposal, WG-async 6 | assignees: "" 7 | --- 8 | 9 | # Summary 10 | 11 | *Describe your meeting topic here. It doesn't have to be long, but it's always good to try and identify a concrete question to be answered or narrow topic for discussion.* 12 | 13 | # Background reading 14 | 15 | *Include any links to material that people might read before-hand.* 16 | 17 | # About this issue 18 | 19 | This issue is a proposal for a WG-async [meeting][]. 20 | 21 | [meeting]: https://rust-lang.github.io/wg-async/meetings.html 22 | -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/new-roadmap-goal.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: New roadmap goal 3 | about: Checklist for owners setting up a new top-level goal on the roadmap 4 | title: "Roadmap goal: [...]" 5 | labels: roadmap 6 | assignees: '' 7 | --- 8 | 9 | This issue tracks the steps for setting up a new top-level goal in the async vision [roadmap]. 10 | Please read the [instructions for owning a goal][owning]. 11 | 12 | 13 | 14 | Goal owner: 15 | 16 | - [ ] Goal has a landing page listing the impact 17 | - [ ] Initiatives are defined 18 | - [ ] All initiatives have a landing page 19 | 26 | - [ ] 27 | - [ ] Goal and all initiatives listed on the [roadmap] with links to landing pages 28 | 29 | Remember, you don't have to do these all at once! 30 | 31 | This issue can be closed when all of the above items are complete. 32 | 33 | [roadmap]: https://rust-lang.github.io/wg-async/vision/roadmap.html 34 | [owning]: https://rust-lang.github.io/wg-async/vision/how_to_vision/owners.html 35 | -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/new-roadmap-initiative.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: New roadmap initiative 3 | about: Checklist for owners setting up a new initiative on the roadmap 4 | title: "Roadmap initiative: [...]" 5 | labels: roadmap 6 | assignees: '' 7 | --- 8 | 9 | 13 | 14 | This issue tracks the steps for setting up a new initiative in the async vision [roadmap]. 15 | Please read the [instructions for owning an initiative][owning]. 16 | 17 | 18 | 19 | Top-level goal: 20 | Initiative owner: 21 | 22 | Each initiative should have a landing page, linked to from the [roadmap]. This can be a page on this website or a dedicated repo. 23 | 24 | For in-progress initiatives the landing page should include, or have pointers to: 25 | 26 | - [ ] Goals and impact of the initiative 27 | - [ ] Milestones 28 | - [ ] Design notes and documentation (this may be sparse in the beginning) 29 | - [ ] Links to any organizing tools, such as a project board 30 | - [ ] The initiative owner 31 | - [ ] The current set of [stakeholders] and the area(s) they represent 32 | - [ ] Notes on how to get involved 33 | - [ ] For landing pages not on this website, a link back to the overall [roadmap] 34 | 35 | For making a dedicated repo, it's recommended to use this [initiative template][template] as a starting point. 36 | 37 | Remember, you don't have to do these all at once! 38 | 39 | This issue can be closed when all of the above items are complete. 40 | 41 | [roadmap]: https://rust-lang.github.io/wg-async/vision/roadmap.html 42 | [owning]: https://rust-lang.github.io/wg-async/vision/how_to_vision/owners.html 43 | [stakeholders]: https://rust-lang.github.io/wg-async/vision/how_to_vision/stakeholders.html 44 | [template]: https://github.com/rust-lang/initiative-template 45 | -------------------------------------------------------------------------------- /.github/workflows/ci.yml: -------------------------------------------------------------------------------- 1 | name: CI 2 | 3 | on: 4 | pull_request: 5 | push: 6 | branches: 7 | - master 8 | 9 | jobs: 10 | test: 11 | name: build and test 12 | runs-on: ubuntu-latest 13 | steps: 14 | - uses: actions/checkout@v2 15 | - name: Install mdbook 16 | run: | 17 | tag=$(curl -LsSf https://api.github.com/repos/rust-lang/mdBook/releases/latest | jq -r '.tag_name') 18 | curl -LsSf https://github.com/rust-lang/mdBook/releases/download/$tag/mdbook-$tag-x86_64-unknown-linux-gnu.tar.gz | tar xzf - 19 | tag=$(curl -LsSf https://api.github.com/repos/badboy/mdbook-mermaid/releases/latest | jq -r '.tag_name') 20 | curl -LsSf https://github.com/badboy/mdbook-mermaid/releases/download/$tag/mdbook-mermaid-$tag-x86_64-unknown-linux-gnu.tar.gz | tar xzf - 21 | echo $(pwd) >> $GITHUB_PATH 22 | - run: mdbook build 23 | - name: Configure git 24 | run: | 25 | git config --global http.postBuffer 50000000 26 | git config --global https.postBuffer 50000000 27 | - uses: rust-lang/simpleinfra/github-actions/static-websites@master 28 | with: 29 | deploy_dir: book 30 | github_token: ${{ secrets.GITHUB_TOKEN }} 31 | if: github.event_name == 'push' && github.ref == 'refs/heads/master' && github.repository_owner == 'rust-lang' 32 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | *.un~ 2 | book 3 | target/ 4 | -------------------------------------------------------------------------------- /Cargo.lock: -------------------------------------------------------------------------------- 1 | # This file is automatically @generated by Cargo. 2 | # It is not intended for manual editing. 3 | version = 3 4 | 5 | [[package]] 6 | name = "aho-corasick" 7 | version = "0.7.18" 8 | source = "registry+https://github.com/rust-lang/crates.io-index" 9 | checksum = "1e37cfd5e7657ada45f742d6e99ca5788580b5c529dc78faf11ece6dc702656f" 10 | dependencies = [ 11 | "memchr", 12 | ] 13 | 14 | [[package]] 15 | name = "camino" 16 | version = "1.0.5" 17 | source = "registry+https://github.com/rust-lang/crates.io-index" 18 | checksum = "52d74260d9bf6944e2208aa46841b4b8f0d7ffc0849a06837b2f510337f86b2b" 19 | 20 | [[package]] 21 | name = "fixlinks" 22 | version = "0.1.0" 23 | dependencies = [ 24 | "camino", 25 | "pathdiff", 26 | "regex", 27 | ] 28 | 29 | [[package]] 30 | name = "memchr" 31 | version = "2.4.1" 32 | source = "registry+https://github.com/rust-lang/crates.io-index" 33 | checksum = "308cc39be01b73d0d18f82a0e7b2a3df85245f84af96fdddc5d202d27e47b86a" 34 | 35 | [[package]] 36 | name = "pathdiff" 37 | version = "0.2.0" 38 | source = "registry+https://github.com/rust-lang/crates.io-index" 39 | checksum = "877630b3de15c0b64cc52f659345724fbf6bdad9bd9566699fc53688f3c34a34" 40 | 41 | [[package]] 42 | name = "regex" 43 | version = "1.5.5" 44 | source = "registry+https://github.com/rust-lang/crates.io-index" 45 | checksum = "1a11647b6b25ff05a515cb92c365cec08801e83423a235b51e231e1808747286" 46 | dependencies = [ 47 | "aho-corasick", 48 | "memchr", 49 | "regex-syntax", 50 | ] 51 | 52 | [[package]] 53 | name = "regex-syntax" 54 | version = "0.6.25" 55 | source = "registry+https://github.com/rust-lang/crates.io-index" 56 | checksum = "f497285884f3fcff424ffc933e56d7cbca511def0c9831a7f9b5f6153e3cc89b" 57 | -------------------------------------------------------------------------------- /Cargo.toml: -------------------------------------------------------------------------------- 1 | [workspace] 2 | members = [ 3 | "tools/fixlinks", 4 | ] 5 | -------------------------------------------------------------------------------- /LICENSE-MIT: -------------------------------------------------------------------------------- 1 | Permission is hereby granted, free of charge, to any 2 | person obtaining a copy of this software and associated 3 | documentation files (the "Software"), to deal in the 4 | Software without restriction, including without 5 | limitation the rights to use, copy, modify, merge, 6 | publish, distribute, sublicense, and/or sell copies of 7 | the Software, and to permit persons to whom the Software 8 | is furnished to do so, subject to the following 9 | conditions: 10 | 11 | The above copyright notice and this permission notice 12 | shall be included in all copies or substantial portions 13 | of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF 16 | ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED 17 | TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A 18 | PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT 19 | SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY 20 | CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION 21 | OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR 22 | IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER 23 | DEALINGS IN THE SOFTWARE. 24 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # wg-async 2 | 3 | Working group dedicated to improving the foundations of async I/O in Rust 4 | 5 | Please visit our [rendered page] for more information! 6 | 7 | [rendered page]: https://rust-lang.github.io/wg-async/ 8 | -------------------------------------------------------------------------------- /get_contributors.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | # Get a list of all contributors that contributed in some way 4 | # either by directly opening a PR, or by participating in issues. 5 | # 6 | # This script is a helper to fill out the "Contributors" chapter. 7 | # It'll only spit out the names and an user must add them to the book. 8 | # 9 | # This script takes two arguments: 10 | # - `username`: Your github username used to authenticate the GitHub API 11 | # - `token`: A github Personal Access Token used to authenticate the GitHub API 12 | 13 | set -euo pipefail 14 | 15 | # Check if there are `username` and `token` arguments 16 | if [ $# -eq 0 ] 17 | then 18 | user="$(git config github.user)" 19 | token="$(git config github.oauth-token)" 20 | elif [ $# -eq 2 ] 21 | then 22 | user="$1" 23 | token="$2" 24 | else 25 | user="" 26 | token="" 27 | fi 28 | 29 | if [ "$user" == "" -o "$token" == "" ] 30 | then 31 | echo "github token required. The token is normally loaded from" 32 | echo "git config (github.user, github.oauth-token), but you can" 33 | echo "also use as follows:" 34 | echo "" 35 | echo "Usage: $0 " 36 | exit 1 37 | fi 38 | 39 | # Check if a command is available, otherwise exit. 40 | function check_bin() { 41 | if ! command -v $1 &> /dev/null 42 | then 43 | echo "'$1' is not installed, but required to run this script." 44 | exit 1 45 | fi 46 | } 47 | 48 | check_bin curl 49 | check_bin jq 50 | 51 | function get_issue_numbers() { 52 | local result="$(curl -s -u $user:$token -H "Accept: application/vnd.github.v3+json" \ 53 | "https://api.github.com/repos/rust-lang/wg-async/issues?state=all&labels=$1&per_page=100" \ 54 | | jq '.[].number')" 55 | 56 | if [ "$(echo $result | wc -w)" -ge 100 ]; 57 | then 58 | echo "Found 100 results for $1. Due to GitHub's paging limits, there might be some issues that are missing" 59 | fi 60 | echo $result 61 | } 62 | 63 | # Get a list of users that participated in issues. 64 | function issue_contributors() { 65 | local numbers="$(get_issue_numbers status-quo-story-ideas) $(get_issue_numbers shiny-future)" 66 | local numbers="$(echo $numbers | sort | uniq)" 67 | 68 | for num in $numbers; do 69 | curl -s -u $user:$token -H "Accept: application/vnd.github.v3+json" \ 70 | https://api.github.com/repos/rust-lang/wg-async/issues/$num/comments | jq -r \ 71 | '.[] | "[" + .user.login + "](" + .user.html_url + ")"' 72 | done | sort | uniq 73 | } 74 | 75 | # Get a list of direct code contributors 76 | function code_contributors() { 77 | curl -s -u $user:$token -H "Accept: application/vnd.github.v3+json" \ 78 | "https://api.github.com/repos/rust-lang/wg-async/contributors?per_page=100" | jq -r \ 79 | '.[] | "[" + .login + "](" + .html_url + ")"' | sort | uniq 80 | } 81 | 82 | echo "Issue contributors" 83 | issue_contributors 84 | 85 | echo 86 | 87 | echo "Code contributors" 88 | code_contributors 89 | -------------------------------------------------------------------------------- /mermaid-init.js: -------------------------------------------------------------------------------- 1 | mermaid.initialize({startOnLoad:true}); 2 | -------------------------------------------------------------------------------- /src/CHARTER.md: -------------------------------------------------------------------------------- 1 | # wg-async charter 2 | 6 | 7 | ## Goals 8 | 9 | 14 | 15 | This working group is focused around implementation/design of the “foundations” for Async I/O. This means that we are focused on designing and implementing extensions to the language, standard library, and other "core" bits of support offered by the Rust organization. 16 | 17 | ## Constraints And Considerations 18 | 19 | We do not directly work on external projects like [tokio], [async-std], [smol], [embassy] and so forth, although we definitely discuss ideas and coordinate with them where appropriate. 20 | 21 | [tokio]: https://tokio.rs/ 22 | [async-std]: https://async.rs/ 23 | [smol]: https://github.com/smol-rs/smol/ 24 | [embassy]: https://github.com/akiles/embassy 25 | 26 | 27 | 31 | 32 | ## Membership 33 | 34 | Leads: [@tmandry] and [@nikomatsakis] 35 | 36 | [Members] 37 | 38 | [@tmandry]: https://github.com/tmandry 39 | [@nikomatsakis]: https://github.com/nikomatsakis 40 | [Members]: https://www.rust-lang.org/governance/teams/lang#team-wg-async 41 | 42 | ### Membership requirements 43 | 44 | Members are invited at the discretion of the working group leads, usually after a period of sustained contribution to the working group. 45 | 46 | In order to remain an active member, WG members must have active participation in last 6 months, where participation is one of the following: 47 | 48 | * Attended triage/sprint meetings 49 | * Opened rust-lang PRs related to async 50 | * Reviewed rust-lang PRs related to async 51 | * Mentored people to fix polish issues 52 | * Led a focus area or initiative 53 | -------------------------------------------------------------------------------- /src/LICENSE-APACHE: -------------------------------------------------------------------------------- 1 | ../LICENSE-APACHE -------------------------------------------------------------------------------- /src/LICENSE-MIT: -------------------------------------------------------------------------------- 1 | ../LICENSE-MIT -------------------------------------------------------------------------------- /src/conversations.md: -------------------------------------------------------------------------------- 1 | # 💬 Conversations 2 | 3 | This section contains notes and summaries from conversations that we have had with people are using Rust and async and describing their experiences. These conversations and links are used as "evidence" when building [the "status quo" section][sq]. 4 | 5 | [sq]: vision/status_quo.md 6 | 7 | ## Not exhaustive nor mandatory 8 | 9 | This section is not meant to be an "exhaustive list" of all sources. That would be impossible. Many conversations are short, not recorded, and hard to summaize. Others are subject to NDA. We certainly don't require that all claims in the [status quo][sq] section are backed by evidence found here. Still, it's useful to have a place to dump notes and things for future reference. 10 | -------------------------------------------------------------------------------- /src/conversations/2021-02-12-Twitter-Thread.md: -------------------------------------------------------------------------------- 1 | # 🐦 2021-02-12 Twitter thread 2 | 3 | Notes taken from the thread in response to [Niko's tweet](https://twitter.com/nikomatsakis/status/1359454255971770372). 4 | 5 | * [Enzo](https://twitter.com/enzo_mdd/status/1359544617121820676) 6 | * A default event loop. "choosing your own event loop" takes time, then you have to understand the differences between each event loop etc. 7 | * Standard way of doing `for await (variable of iterable)` would be nice. 8 | * Standard promise combinators. 9 | * [creepy_owlet](https://twitter.com/creepy_owlet/status/1359649695103131649) 10 | * https://github.com/dtantsur/rust-osauth/blob/master/src/sync.rs 11 | * async trait -- 12 | * https://twitter.com/jcsp_tweets/status/1359820431151267843 13 | * "I thought async was built-in"? 14 | * nasty compiler errors 15 | * [ownership puzzle](https://www.fpcomplete.com/blog/ownership-puzzle-rust-async-hyper/) blog post 16 | * [rubdos](https://twitter.com/rubdos/status/1359462402702606336) 17 | * [blog post](https://www.rubdos.be/corona/qt/rust/tokio/actix/2020/05/23/actix-qt.html) describes integrating two event loops 18 | * mentions desire for runtime independent libraries 19 | * qt provides a mechanism to integrate one's own event loop 20 | * [llvm bug](https://github.com/rust-lang/rust/issues/60605) generates invalid arm7 assembly 21 | * [alexmiberry](https://twitter.com/alexmiberry/status/1359559299161325581) 22 | * kotlin/scala code, blocked by absence of async trait 23 | * helpful blog post 24 | * [jamesmcm](http://jamesmcm.github.io/blog/2020/05/06/a-practical-introduction-to-async-programming-in-rust/) 25 | * note that `join` and `Result` play poorly together 26 | * [async-std version](https://github.com/jamesmcm/async-rust-example/blob/async-std/client_async/src/main.rs#L50-L59) 27 | * [tokio version](https://github.com/jamesmcm/async-rust-example/blob/master/client_async/src/main.rs#L40-L61) has some wild "double question marks" -- I guess that spawn must be adding a layer of `Result`? 28 | * the post mentions rayon but this isn't really a case where one ought to use rayon -- still, Rayon's APIs here are SO much nicer :) 29 | * [rust aws and lambda](http://jamesmcm.github.io/blog/2020/04/19/data-engineering-with-rust-and-aws-lambda/#en) 30 | * [issue requiring async drop](https://github.com/jamesmcm/s3rename/issues/16) 31 | * [fasterthanlime](https://fasterthanli.me/articles/getting-in-and-out-of-trouble-with-rust-futures) -- 32 | * this post is amazing 33 | * the discussion on Send bounds and the ways to debug it is great 34 | * [bridging different runtimes using GATs](https://github.com/thanethomson/async-channel-abs/blob/master/src/runtime.rs) 35 | * first server app, [great thread with problems](https://twitter.com/richardsabow/status/1345815109201842178) 36 | * "I wasn't expecting that it will be easy but after Go and Node.js development it felt extremely hard to start off anything with Rust." 37 | * "felt like I have to re-learn everything from scratch: structuring project and modules, dependency injection, managing the DB and of course dealing with concurrency" 38 | * common thread: poor docs, though only somewhat in async libraries 39 | * I had enums in the DB and it was a bit more complex to map them to my custom Rust enums but I succeeded with the help of a couple of blog posts – and not with Diesel documentation 40 | * I used Rusoto for dealing with AWS services. It's also pretty straightforward and high quality package – but again the documentation was sooo poor. 41 | * [implaustin](https://t.co/4rlyfUlFES?amp=1) wrote a [very nice post](https://t.co/4rlyfUlFES?amp=1) but it felt more like a "look how well this worked" post than one with actionable feedback 42 | * "Async has worked well so far. My top wishlist items are Sink and Stream traits in std. It's quite difficult to abstract over types that asynchronously produce or consume values." 43 | * "AsyncRead/AsyncWrite work fine for files, tcp streams, etc. But once you are past I/O and want to pass around structs, Sink and Stream are needed. One example of fragmentation is that Tokio channels used to implement the futures Sink/Stream traits, but no longer do in 1.0." 44 | * "I usually use Sink/Stream to abstract over different async channel types. Sometimes to hide the details of external dependencies from a task (e.g. where is this data going?). And sometimes to write common utility methods." 45 | * "One thing I can think of: there are still a lot of popular libraries that don't have async support (or are just getting there). Rocket, Criterion, Crossterm's execute, etc." 46 | * [EchoRior](https://twitter.com/EchoRior/status/1359964691305496579): 47 | * "I've written a bit of rust before, but rust is my first introduction to Async. My main gripes are that it's hard to figure our what the "blessed" way of doing async is. I'd love to see async included in the book, but I understand that async is still evolving too much for that." 48 | * "Adding to the confusion: theres multiple executors, and they have a bit of lock in. Async libraries being dependent on which executor version I use is also confusing for newcomers. In other langs, it seems like one just uses everything from the stdlib and everything is compatible" 49 | * "That kind of gave me a lot of hesitation/fomo in the beginning, because it felt like I had to make some big choices around my tech stack that I felt I would be stuck with later. I ended up chatting about this in the discord & researching for multiple days before getting started." 50 | * "Also, due to there not being a "blessed" approach, I don't know if I'm working with some misconceptions around async in rust, and will end up discovering I will need to redo large parts of what I wrote." 51 | -------------------------------------------------------------------------------- /src/design_docs.md: -------------------------------------------------------------------------------- 1 | # 🔬 Design documents 2 | 3 | The **design documents** (or "design docs", more commonly) describe potential designs. These docs vary greatly in terms of their readiness to be implemented: 4 | 5 | * Early on, they describe a vague idea for a future. Often this takes the shape of capturing constraints on the solution, rather than the solution itself. 6 | * When a feature is getting ready to ship, they can evolve into a full blown RFC, with links to tracking issues or other notes. 7 | 8 | ## Early stage design docs 9 | 10 | In the early stages, design docs are meant to capture interesting bits of "async design space". They are often updated to capture the results of a fruitful conversation or thread which uncovered contraints or challenges in solving a particular problem. They will capture a combination of the following: 11 | 12 | * use cases; 13 | * interesting aspects to the design; 14 | * alternatives; 15 | * interactions with other features. 16 | 17 | ## Late stage design docs 18 | 19 | As a design progresses, the doc should get more and more complete, until it becomes something akin to an RFC. (Often, at that point, we will expand the design document into a directory, adding an actual RFC draft and other contents; those things can live in this repo or elsewhere, depending.) Once we decide to put a design doc onto the roadmap, it will also contain links to tracking issues or other places to track the status. -------------------------------------------------------------------------------- /src/design_docs/async_closures.md: -------------------------------------------------------------------------------- 1 | # 📦 Async closures 2 | -------------------------------------------------------------------------------- /src/design_docs/async_drop.md: -------------------------------------------------------------------------------- 1 | # 🗑️ Async drop 2 | -------------------------------------------------------------------------------- /src/design_docs/async_fn_in_traits.md: -------------------------------------------------------------------------------- 1 | # 🧬 Async fn in traits 2 | -------------------------------------------------------------------------------- /src/design_docs/async_lifecycle.md: -------------------------------------------------------------------------------- 1 | # ♻️ Async lifecycle 2 | -------------------------------------------------------------------------------- /src/design_docs/async_main.md: -------------------------------------------------------------------------------- 1 | # 🎇 Async main 2 | -------------------------------------------------------------------------------- /src/design_docs/async_read_write.md: -------------------------------------------------------------------------------- 1 | # 📝 AsyncRead, AsyncWrite traits 2 | -------------------------------------------------------------------------------- /src/design_docs/channels.md: -------------------------------------------------------------------------------- 1 | # 📺 Async aware channels 2 | -------------------------------------------------------------------------------- /src/design_docs/completion_based_futures.md: -------------------------------------------------------------------------------- 1 | # ⏳ Completion-based futures 2 | -------------------------------------------------------------------------------- /src/design_docs/generator_syntax.md: -------------------------------------------------------------------------------- 1 | # ⚡ Generator syntax 2 | -------------------------------------------------------------------------------- /src/design_docs/join.md: -------------------------------------------------------------------------------- 1 | # 🤝 Join combinator 2 | -------------------------------------------------------------------------------- /src/design_docs/mutex.md: -------------------------------------------------------------------------------- 1 | # 🔒 Mutex (future-aware) 2 | -------------------------------------------------------------------------------- /src/design_docs/select.md: -------------------------------------------------------------------------------- 1 | # 🤷‍♀️ Select combinator 2 | -------------------------------------------------------------------------------- /src/design_docs/sink_trait.md: -------------------------------------------------------------------------------- 1 | # 🚰 Sink trait 2 | -------------------------------------------------------------------------------- /src/design_docs/stream.md: -------------------------------------------------------------------------------- 1 | # ☔ Stream trait 2 | -------------------------------------------------------------------------------- /src/design_docs/yield_safe.md: -------------------------------------------------------------------------------- 1 | # ⚠️ Yield-safe lint 2 | -------------------------------------------------------------------------------- /src/glossary.md: -------------------------------------------------------------------------------- 1 | # Glossary 2 | 3 | If you follow discussions in the async ecosystem you're likely to find acronyms being tossed around, many of which refer to language features that are in-progress. 4 | What follows is a best-effort list of those acronyms and links to resources discussing the language features. 5 | 6 | ## AFIT - async fn in trait 7 | - [RFC #3185](https://github.com/rust-lang/rfcs/pull/3185) 8 | - Allows static dispatch of `async fn`s in traits 9 | 10 | ## RPITIT - return position `impl trait` in traits 11 | - [RFC #3425](https://github.com/rust-lang/rfcs/pull/3425) 12 | - Allows `impl trait` as a return type in trait definitions 13 | 14 | ## RTN - return type notation 15 | - [Tracking Issue: rust#109417](https://github.com/rust-lang/rust/issues/109417) 16 | - Experimental, pre-RFC feature providing bounds for return types on `async fn`s 17 | 18 | ## TAIT - type alias `impl trait` 19 | - [Tracking Issue: rust#63063](https://github.com/rust-lang/rust/issues/63063) 20 | - Allows the ability to write `type Foo = impl Bar` 21 | -------------------------------------------------------------------------------- /src/triage.md: -------------------------------------------------------------------------------- 1 | # 🔍 Triage meetings 2 | 3 | ## When, where 4 | 5 | The weekly triage meeting is held on [Zulip] at [11:30 US Eastern time on every other Monday][everytimezone]. 6 | For the date of the next triage meeting, see the [meetings page]. 7 | 8 | [meetings page]: ./meetings.md 9 | [everytimezone]: https://everytimezone.com/s/c3abbec9 10 | [Zulip]: ./welcome.md#zulip 11 | 12 | ## So you want to fix a bug? 13 | 14 | If you're interested in fixing bugs, there is no need to wait for the triage meeting. 15 | Take a look at the [mentored async-await bugs that have no assignee][bugs]. 16 | Every mentored bug should have a few comments. 17 | If you see one you like, you can add the `@rustbot claim` comment into the bug and start working on it! 18 | Feel to reach out to the mentor on [Zulip] to ask questions. 19 | 20 | [bugs]: https://github.com/rust-lang/rust/issues?q=is%3Aopen+label%3AE-mentor+label%3AA-async-await+no%3Aassignee 21 | 22 | ## Project board 23 | 24 | The [project board] tracks various bugs and other work items for the async foundation group. 25 | It is used to drive the triage process. 26 | 27 | [project board]: https://github.com/orgs/rust-lang/projects/2 28 | 29 | ## Triage process 30 | 31 | In our weekly triage meetings, we take new issues assigned [`A-async-await`] and categorize them. 32 | The process is: 33 | 34 | - Review the [project board], from right to left: 35 | - Look at what got **Done**, and celebrate! :tada: 36 | - Review **In progress** issues to check we are making progress and there is a clear path to finishing (otherwise, move to the appropriate column) 37 | - Review **Blocked** issues to see if there is anything we can do to unblock 38 | - Review **Claimed** issues to see if they are in progress, and if the assigned person still intends to work on it 39 | - Review **To do** issues and assign to anyone who wants to work on something 40 | - Review [uncategorized issues] 41 | - Mark `P-low`, `P-medium`, or `P-high` 42 | - Add `P-high` and _assigned_ `E-needs-mentor` issues to the [project board] 43 | - Mark `AsyncAwait-triaged` 44 | - If there's still a shortage of **To do** issues, review the list of [`P-medium`] or [`P-low`] issues for candidates 45 | 46 | ## Mentoring 47 | 48 | If an issue is a good candidate for mentoring, mark `E-needs-mentor` and try to find a mentor. 49 | 50 | Mentors assigned to issues should write up mentoring instructions. 51 | **Often, this is just a couple lines pointing to the relevant code.** 52 | Mentorship doesn't require intimate knowledge of the compiler, just some familiarity and a willingness to look around for the right code. 53 | 54 | After writing instructions, mentors should un-assign themselves, add `E-mentor`, and remove `E-needs-mentor`. 55 | On the project board, if a mentor is assigned to an issue, it should go to the **Claimed** column until mentoring instructions are provided. 56 | After that, it should go to **To do** until someone has volunteered to work on it. 57 | 58 | [`A-async-await`]: https://github.com/rust-lang/rust/labels/A-async-await 59 | [uncategorized issues]: https://github.com/search?q=org%3Arust-lang+is%3Aissue+label%3AA-async-await+is%3Aopen+-label%3AAsyncAwait-Triaged&type=Issues 60 | [`P-medium`]: https://github.com/search?q=org%3Arust-lang+is%3Aissue+label%3AAsyncAwait-Triaged+label%3AP-medium+is%3Aopen&type=Issues 61 | [`P-low`]: https://github.com/search?q=org%3Arust-lang+is%3Aissue+label%3AAsyncAwait-Triaged+label%3AP-low+is%3Aopen&type=Issues 62 | -------------------------------------------------------------------------------- /src/vision.md: -------------------------------------------------------------------------------- 1 | # 🔮 The vision 2 | 3 | [cc]: ./vision/characters.md 4 | [Grace]: ./vision/characters/grace.md 5 | [Alan]: ./vision/characters/alan.md 6 | [gba]: ./vision/characters/grace.md#variant-a-networking-systems 7 | [gba]: ./vision/characters/grace.md#variant-b-embedded 8 | [sq]: ./vision/status_quo.md 9 | [gsq]: ./vision/status_quo/gracy_deploys_her_service.md 10 | [cok]: https://en.wikipedia.org/wiki/Curse_of_knowledge 11 | [sf]: ./vision/shiny_future.md 12 | [roadmap]: ./vision/roadmap.md 13 | 14 | ## What is this 15 | 16 | We believe Rust can become one of the most popular choices for building distributed systems, ranging from embedded devices to foundational cloud services. Whatever they're using it for, we want all developers to love using Async Rust. For that to happen, we need to move Async Rust beyond the "MVP" state it's in today and make it accessible to everyone. 17 | 18 | This document is a collaborative effort to build a shared vision for Async Rust. **Our goal is to engage the entire community in a collective act of the imagination:** how can we make the end-to-end experience of using Async I/O not only a pragmatic choice, but a *joyful* one? 19 | 20 | ## 🚧 Under construction! Help needed! 🚧 21 | 22 | The first version of this document is not yet complete, but it's getting very close! We are in the process of finalizing the set of ["status quo"](./vision/status_quo.md) and ["shiny future"](./vision/shiny_future.md) stories and the details of the [proposed roadmap](./vision/roadmap.md). The current content however is believed to be relatively final, at this point we are elaborating and improving it. 23 | 24 | ## Where we are and where we are going 25 | 26 | The "vision document" starts with a [cast of characters][cc]. Each character is tied to a particular Rust value (e.g., performance, productivity, etc) determined by their background; this background also informs the expectations they bring when using Rust. [Grace], for example, wants to keep the same level of performance she currently get with C, but with the productivity benefits of memory safety. [Alan], meanwhile, is hoping Rust will give him higher performance without losing the safety and ergonomics that he enjoys with garbage collected languages. 27 | 28 | For each character, we write ["status quo" stories][sq] that describe the challenges they face as they try to achieve their goals (and typically fail in dramatic fashion!), **These stories are not fiction.** They are an amalgamation of the real experiences of people using Async Rust, as reported to us by interviews, blog posts, and tweets. Writing these stories helps us gauge the cumulative impact of the various papercuts and challenges that one encounters when using Async Rust. 29 | 30 | The ultimate goal of the vision doc, of course, is not just to tell us where we are now, but where we are going and how we will get there. For this, we include ["shiny future" stories][sf] that tell us how those same characters will fare in a few years time, when we've had a chance to improve the Async Rust experience. 31 | 32 | ## The vision drives the work 33 | 34 | The vision is not just idle speculation. It is the central document that we use to organize ourselves. When we think about our [roadmap](./vision/roadmap.md) for any given year, it is always with the aim of moving us closer to the vision we lay out here. 35 | 36 | ## Involving the whole community 37 | 38 | The async vision document provides a forum where the Async Rust community can plan a great overall experience for Async Rust users. Async Rust was intentionally designed not to have a "one size fits all" mindset, and we don't want to change that. Our goal is to build a shared vision for the end-to-end experience while retaining the loosely coupled, exploration-oriented ecosystem we have built. 39 | -------------------------------------------------------------------------------- /src/vision/characters.md: -------------------------------------------------------------------------------- 1 | # 🙋‍♀️ Cast of characters 2 | 3 | ## What is this? 4 | 5 | We've created four characters that we use to guide our thinking. These characters are the protagonists of our [status quo] and [shiny future] stories, and they help us to think about the different kinds of priorities and expectations that people bring to Async Rust. Having names and personalities also makes the stories more fun and approachable. 6 | 7 | [Alan]: ./characters/alan.md 8 | [Grace]: ./characters/grace.md 9 | [Niklaus]: ./characters/niklaus.md 10 | [Barbara]: ./characters/barbara.md 11 | [status quo]: ./status_quo.md 12 | [shiny future]: ./shiny_future.md 13 | 14 | ## The characters 15 | 16 | * [Alan]: the experienced "GC'd language" developer, new to Rust 17 | * *Top priority*: performance -- that's what he is not getting from current GC'd language 18 | * *Expectations*: absence of memory safety bugs (he gets that now from his GC), strong ecosystem, great tooling 19 | * [Grace]: the systems programming expert, new to Rust 20 | * *Top priority*: memory safety -- that's what she is not getting from C/C++ 21 | * *Expectations*: able to do all the things she's used to from C/C++ 22 | * [Niklaus]: new programmer from an unconventional background 23 | * *Top priority*: accessibility -- he's learning a lot of new things at once 24 | * *Expectations*: community -- the community enabled him to have early success, and he is excited to have it support him and him grow more 25 | * [Barbara]: the experienced Rust developer 26 | * *Top priority*: overall productivity and long-term maintenance -- she loves Rust, and wants to see it extended to new areas; she has an existing code base to maintain 27 | * *Expectations*: elegance and craftsmanship, fits well with Rust 28 | 29 | ## 🤔 Frequently Asked Questions 30 | 31 | ### Where do the names come from? 32 | Famous programming language designers and theorists. [Alan Turing], [Grace Hopper], [Niklaus Wirth], and [Barbara Liskov]. 33 | 34 | ### I don't see myself in these characters. What should I do? 35 | Come to Zulip and talk to us about it! Maybe they need to be adjusted! 36 | 37 | ### I see myself in more than one of these characters! 38 | Yeah, me too. 39 | 40 | [Alan Turing]: https://en.wikipedia.org/wiki/Alan_Turing 41 | [Grace Hopper]: https://en.wikipedia.org/wiki/Grace_Hopper 42 | [Niklaus Wirth]: https://en.wikipedia.org/wiki/Niklaus_Wirth 43 | [Barbara Liskov]: https://en.wikipedia.org/wiki/Barbara_Liskov 44 | -------------------------------------------------------------------------------- /src/vision/characters/alan.md: -------------------------------------------------------------------------------- 1 | # 🙋‍♀️ Cast of characters 2 | 3 | ## Alan: the experienced "GC'd language" developer, new to Rust 4 | 5 | ### Variant A: Dynamic languages 6 | 7 | Alan has been programming for years. He has built systems in Ruby on Rails, node.js, and used Django too. Lately he's been learning Rust and he is tinkering with integrating Rust into some of his projects to get better performance and reliability. He's also building some projects entirely in Rust. 8 | 9 | ### Variant B: Java 10 | 11 | Alan works at a Java shop. They run a number of network services built in Java, along with some that use Kotlin or Scala. He's very familiar with the Java ecosystem and the tooling that the JVM offers. He's also sometimes had to tweak his code to work around garbage collector latencies or to reduce overall memory usage. He's curious to try porting some systems to Rust to see how it works. 12 | 13 | ### Variant C: Kotlin 14 | 15 | Alan is developing networking programs in Kotlin. He loves Kotlin for its expressive syntax and clean integration with Java. Still, he sometimes encounters problems running his services due to garbage collection latencies or overall memory usage. He's heard that Rust can be fun to use too, and is curious to try it out. 16 | 17 | ### Variant D: Go 18 | 19 | Alan develops a distributed database in Go, enjoying its simplicity and first-class treatment of concurrency. He's successfully built a transactional database that handles over 100K QPS. Intrigued by Rust's promise of "fearless concurrency", Alan tries Rust for more efficient use of memory and CPU. He's curious what classes of errors Rust async prevents and how Rust guarantees its safety without sacrificing the speed. 20 | 21 | ## 🤔 Frequently Asked Questions 22 | 23 | ### What does Alan want most from Async Rust? 24 | * The promise of better performance and memory usage than the languages he's been using. Rust's safety guarantees are important too; he's considered using C++ in the past but always judged the maintenance burden would be too high. 25 | 26 | ### What expectations does Alan bring from his current environment? 27 | * A focus on ease of use, a strong ecosystem, and great tooling. 28 | -------------------------------------------------------------------------------- /src/vision/characters/barbara.md: -------------------------------------------------------------------------------- 1 | # 🙋‍♀️ Cast of characters 2 | 3 | ## Barbara: the experienced Rust developer 4 | 5 | Barbara has been using Rust since the 0.1 release. She remembers some of the crazy syntax in Ye Olde Rust of Yore and secretly still misses the `alt` keyword (don't tell anyone). Lately she's maintaining various projects in the async space. 6 | 7 | [axes]: ../characters.md#axes 8 | 9 | ## 🤔 Frequently Asked Questions 10 | 11 | ### What does Barbara want most from Async Rust? 12 | * She is using Rust for its feeling of productivity, and she expects Async Rust to continue in that tradition. 13 | * She maintains several existing projects, so stability is important to her. 14 | 15 | ### What expectations does Barbara bring from her current environment? 16 | * She wants a design that feels like the rest of Rust. 17 | * She loves Rust and she expects Async Rust to share its overall values. 18 | -------------------------------------------------------------------------------- /src/vision/characters/grace.md: -------------------------------------------------------------------------------- 1 | # 🙋‍♀️ Cast of characters 2 | 3 | ## Grace: the systems programming expert, new to Rust 4 | 5 | Grace has been writing C and C++ for a number of years. She's accustomed to hacking lots of low-level details to coax the most performance she can from her code. She's also experienced her share of epic debugging sessions resulting from memory errors in C. She's intrigued by Rust: she likes the idea of getting the same control and performance she gets from C but with the productivity benefits she gets from memory safety. She's currently experimenting with introducing Rust into some of the systems she works on, and she's considering Rust for a few greenfield projects as well. 6 | 7 | [axes]: ../characters.md#axes 8 | 9 | ## 🤔 Frequently Asked Questions 10 | 11 | ### What does Grace want most from Async Rust? 12 | Grace is most interested in memory safety. She is comfortable with C and C++ but she's also aware of the maintenance burden that arises from the lack of memory safety. 13 | 14 | ### What expectations does Grace bring from her current environment? 15 | * Grace expects to be able to get the same performance she used to get from C or C++. 16 | * Grace is accustomed to various bits of low-level tooling, such as gdb or perf. It's nice if Rust works reasonably well with those tools, but she'd be happy to have access to better alternatives if they were available. She's happy using `cargo` instead of `make`, for example. 17 | -------------------------------------------------------------------------------- /src/vision/characters/niklaus.md: -------------------------------------------------------------------------------- 1 | # 🙋‍♀️ Cast of characters 2 | 3 | ## Niklaus: new programmer from an unconventional background 4 | 5 | He's always been interested in programming but doesn't have experience with it. He's been working as a tech writer and decided to dip his toe in by opening PRs to improve the documentation for one of the libraries he was playing with. The feedback was positive so he fixed a small bug. He's now considering getting involved in a deeper way. 6 | 7 | [axes]: ../characters.md#axes 8 | 9 | ## 🤔 Frequently Asked Questions 10 | 11 | ### What does Niklaus want most from Async Rust? 12 | * Niklaus values accessibility. He's learning a lot of new things at once and it can be overwhelming. 13 | 14 | ### What expectations does Niklaus bring from his current environment? 15 | * Niklaus expects a strong and supportive community. The Rust community enabled him to have early success, and he is excited to have it support him and for it to help him grow more. 16 | -------------------------------------------------------------------------------- /src/vision/how_it_feels.md: -------------------------------------------------------------------------------- 1 | # How using async Rust ought to feel (and why it doesn't today) 2 | 3 | This section is, in many ways, the most important. It aims to identify the way it should feel to use Async Rust. 4 | 5 | ## Consistent: "just add async/await" 6 | 7 | Async Rust should be a small delta atop Sync Rust. People who are familiar with sync Rust should be able to leverage what they know to make adopting Async Rust straightforward. Porting a sync code base to async should be relatively smooth: just add async/await, adopt the async variants of the various libraries, and you're done. 8 | 9 | ## Reliable: "if it compiles, it works" 10 | 11 | One of the great things about Rust is the feeling of "if it compiles, it works". This is what allows you to do a giant refactoring and find that the code runs on the first try. It is what lets you deploy code that uses parallelism or other fancy features without exhausting fuzz testing and worry about every possible corner case. 12 | 13 | ## Empowering: "complex stuff feels easy" 14 | 15 | Rust's great strength is taking formerly complex, wizard-like things and making them easy to do. In the case of async, that means letting people use the latest and greatest stuff, like io-uring. It also means enabling parallelism and complex scheduling patterns very easily. 16 | 17 | ## Performant: "ran well right out of the box" 18 | 19 | Rust code tends to perform "quite well" right out of the box. You don't have to give up the "nice things" in the language, like closures or high-level APIs, in order to get good performance and tight memory usage. In fact, those high-level APIs often perform as well or better than what you would get if you wrote the code yourself. 20 | 21 | ## Productive: "great crates for every need, just mix and match" 22 | 23 | Being able to leverage a large ecosystem of top-notch crates is a key part of what makes Rust (and most any modern language) productive. When using async Rust, you should be able to search crates.io and find crates that cover all kinds of things you might want to do. You should be able to add those crates to your `Cargo.toml` and readily connect them to one another without surprising hiccups. 24 | 25 | ## Transparent and tunable: "it's easy to diagnose deadlocks and performance bottlenecks" 26 | 27 | Using Rust means most things work and perform well by default, but of course it can't prevent all problems. When you do find bugs, you need to be able to easily track what happened and figure out how to fix it. When your performance is subpar, you need to be able to peek under the covers and understand what's going on so that you can tune it up. In synchronous Rust, this means integrating with but also improving on existing tooling like debuggers and profilers. In asynchronous Rust, though, there's an extra hurdle, because the terms that users are thinking in (asynchronous tasks etc) exist within the runtime, but are not the same terms that synchronous debuggers and profilers expose. There is a need for more customized tooling to help users debug problems without having to map between the async concept and the underlying implementation. 28 | 29 | ## Control: "I can do all the weird things" 30 | 31 | Part of what's great about Rust is that it lets you get into explore all the corner cases. Want to target the kernel? Develop embedded systems using async networking without any operating system? Run on WebAssembly? No problem, we can do that. 32 | 33 | ## Interoperable: "integrating with C++, node.js, etc is easy" 34 | 35 | Much like C, Rust aims to be a "lingua franca", something you can integrate into your existing systems on a piecemeal basis. In synchronous Rust, this means that functions can "speak" the C ABI and Rust structures can be compiled with C-compatible layouts, and that we use native system functionality like the default memory allocator or the native threading APIs. In _asynchronous_ Rust, it means that we are able to integrate into other systems, like C++ futures, Grand Central Dispatch, or JavaScript promises. 36 | -------------------------------------------------------------------------------- /src/vision/how_to_vision.md: -------------------------------------------------------------------------------- 1 | # ❓ How to vision 2 | 3 | ## How you can help 4 | 5 | | When | What | 6 | | --- | --- | 7 | | 🛑 **Coming soon** | Participate in discussions and development towards [roadmap] goals | 8 | | 🛑 **Coming soon** | Take ownership of "help wanted" goals from the [roadmap] | 9 | | ⚠️ **Winding down** | Propose new ["status quo" stories][hvsq] or [comment] on existing PRs | 10 | | ⚠️ **Winding down** | Propose new ["shiny future" stories][hvsf] or [comment] on existing PRs | 11 | | 🛑 Coming soon | Vote for the [awards] on the status quo and shiny future stories! | 12 | 13 | ## Making the vision real 14 | 15 | We are currently working towards implementing the async vision described in the [shiny future] section. On the [roadmap] page, you can get an overview of the major goals that are part of implementing that future and how we have divided up the work. Each of the goals also has several initiatives, and those initiatives have upcoming milestones. If you'd like to participate in an initiative, you can find the appropriate Zulip stream and see if they are looking for help! 16 | 17 | ### Goal and initiative owners 18 | 19 | Each top-level goal and initiative in the [roadmap] has an **owner**. The owner of the top-level goal manages the goal overall, while the owner of an initiative manages the "nitty gritty" design work (for example, preparing the [evaluation](./how_to_vision/evaluations.md), authoring any RFCs required, or supervising the implementation). You can learn more about the [responsibilities of owners](./how_to_vision/owners.md) in this page. If you have questions about whether you can help out with a goal or an initiative, the owner is probably the one to talk to. 20 | 21 | [responsibilities of owners]: ./how_to_vision/owners.md 22 | 23 | ### Help wanted goals 24 | 25 | Some of the top-level goals are marked with ✋, which means "help wanted". Those goals are looking for an owner. If you think you might be interested, you can read about the [responsibilities of owners] and contact the [wg leads]. 26 | 27 | ### Stakeholders 28 | 29 | While we always encourage feedback from the broader public, many of our initiatives also have identified sets of [stakeholders]. These are people who are specially consulted as part of the process to give feedback on the design and implementation. They can be representatives from major projects in the ecosystem, production users, or other sorts of experts. 30 | 31 | [stakeholders]: ./how_to_vision/stakeholders.md 32 | 33 | ## Living document 34 | 35 | Although many of the pieces are complete, the vision doc is a living document and it will never be done. During the brainstorming period, we had a [lot of stories submitted](./submitted_stories.md) and we are now in the process of "harmonizing" those into a small set of [status quo] and [shiny future] narratives, each based around a representative [project] and the same set of [characters]. If you'd like to help out with that, contact the [wg leads]. 36 | 37 | We also plan to regularly revisit the vision once we've made significant progress on implementation or if new information has come to light. 38 | 39 | ### Submitting status quo and shiny future story PRs 40 | 41 | Although the brainstorming period has ended, we are still open to new PRs, particularly if they cover space that has not been well covered: 42 | 43 | * [Templates and instructions for status quo stories can be found here.][hvsq] 44 | * [Templates and instructions for shiny future stories can be found here.][hvsf] 45 | 46 | ### Wait, did somebody say awards? 47 | 48 | Yes! We are planning on giving [awards] in various categories for folks who write [status quo](./how_to_vision/status_quo.md) and [shiny future](./how_to_vision/shiny_future.md) PRs. The precise categories are TBD. Check out the [awards] page for more details. 49 | 50 | 51 | [hvsq]: ./how_to_vision/status_quo.md 52 | [hvsf]: ./how_to_vision/shiny_future.md 53 | [Vote]: ./how_to_vision/awards.md 54 | [Vote]: ./how_to_vision/awards.md#Vote 55 | [comment]: ./how_to_vision/comment.md 56 | [awards]: ./how_to_vision/awards.md 57 | [wg leads]: ../welcome.md#leads 58 | [repo]: https://github.com/rust-lang/wg-async 59 | [open "status quo" issues]: https://github.com/rust-lang/wg-async/labels/status-quo-story-ideas 60 | [roadmap]: ./roadmap.md 61 | [status quo]: ./status_quo.md 62 | [shiny future]: ./shiny_future.md 63 | [project]: ./project.md 64 | [characters]: ./characters.md 65 | -------------------------------------------------------------------------------- /src/vision/how_to_vision/awards.md: -------------------------------------------------------------------------------- 1 | # ❓ How to vision: Awards 2 | 3 | At the end of the [brainstorming period][htvbp], we'll also vote on various awards to give to the status quo and shiny future PRs that were submitted. 4 | 5 | ## Award categories 6 | 7 | These are the award categories: 8 | 9 | * Most humorous story 10 | * Most creative story 11 | * Most supportive -- who left the most helpful comments? 12 | * Most prolific -- who wrote the most stories? 13 | * Most unexpected -- which status quo story (or shiny future) took you by surprise? 14 | * Most painful "status quo" story 15 | * Most ambitious "shiny future" story 16 | * Most extensive FAQ 17 | 18 | However, if you have an idea for another award category, we are happy to take suggestions. One rule: the awards can't be negative (e.g., no "most unrealistic"), and they can't be about which thing is "best". That would work against the brainstorming spirit. 19 | 20 | [htvbp]: ../how_to_vision.md#brainstorming 21 | 22 | ## Voting 23 | 24 | At the end of the [brainstorming period][htvbp], we're going to have a voting session to select which PRs and people win the awards. The winners will be featured in a blog post. 🏆 25 | -------------------------------------------------------------------------------- /src/vision/how_to_vision/comment.md: -------------------------------------------------------------------------------- 1 | # ❓ How to vision: Constructive comments 2 | 3 | Figuring out the future is tricky business. We all know the internet is not always a friendly place. We want this discussion to be different. 4 | 5 | ## Be respectful and supportive 6 | 7 | Writing a "status quo" or "shiny future" story is an act of bravery and vulnerability. In the status quo, we are asking people to talk about the things that they or others found hard, to admit that they had trouble figuring something out. In the case of the shiny future, we're asking people to put out half-baked ideas so that we can find the seeds that will grow into something amazing. It doesn't take much to make that go wrong. 8 | 9 | ## Comment to understand or improve, not to negate or dissuade 10 | 11 | > “Most people do not listen with the intent to understand; they listen with the intent to reply.” 12 | > 13 | > -- [Stephen Covey](https://www.franklincovey.com/the-7-habits/habit-5/) 14 | 15 | The golden rule is that when you leave a comment, you are looking to understand or improve the story. 16 | 17 | For status quo stories, remember that these are true stories about people's experiences -- they can't be *wrong* (though they could be inaccurate). It may be that somebody tries for days to solve a problem that would've been easy if they had just known to call a particular method. That story is not wrong, it's an opportunity to write a shiny future story in which you explain how they would've learned about that method, or perhaps about how that method would become unnecessary. 18 | 19 | For shiny future stories, even if you don't like the idea, you should ask comments with the goal of better understanding what the author likes about it. Understanding that may give you an idea for how to get those same benefits in a way that you are happier with. It's also valid to encourage the author to elaborate on the impact their story will have on different characters. 20 | 21 | ## You might just want to write your own story 22 | 23 | Remember, opening your own PR is free (In fact, we're giving an [award] for being "most prolific"). If you find that you had a really different experience than a status quo story, or you have a different idea for a shiny future, consider just writing your own PR instead of commenting negatively on someone else's. The goal of the brainstorming phase is to put a lot of alternatives, so that we can look for opportunities to combine them and make something with the best of both. 24 | 25 | [award]: ./awards.md 26 | 27 | ## Good questions for status quo stories 28 | 29 | Here are some examples of good questions for "status quo" stories: 30 | 31 | * Tell me more about this step. What led NAME to do X? 32 | * What do you think OTHER_NAME would have done here? 33 | * Can you be more specific about this point? What library did they use? 34 | 35 | ## Good questions for shiny future stories 36 | 37 | Here are some examples of good questions for "shiny future" stories: 38 | 39 | * How does NAME do X in this future? 40 | * It seems like this would interfere with X, which is important for application A. How would NAME handle that case in this future? 41 | 42 | You should not be afraid to raise technical concerns -- we need to have a robust technical discussion! But do so in a way that leaves room to find an answer that satisfies both of you. 43 | -------------------------------------------------------------------------------- /src/vision/how_to_vision/evaluations.md: -------------------------------------------------------------------------------- 1 | # Writing an evaluation 2 | 3 | When an initiative involves a complex design task, the [initiative owner] begins by writing an **evaluation**. The evaluation documents the various design options and their tradeoffs, and also includes a recommendation. Evaluations are posted publicly and presented to the relevant Rust teams, which will discuss with the [owners] and [stakeholders] ultimately make a choice on how to proceed. 4 | 5 | The current draft for each evaluation will be maintained in some git repository, often a dedicated repository for the initiative. The repository will also list the [stakeholders] associated with that particular effort. 6 | 7 | ## Getting feedback 8 | 9 | Developing an evaluation consists of first preparing an initial draft by surveying initial work and then taking the following steps (repeat until satisfied): 10 | 11 | * Review draft in meetings with stakeholders 12 | * These meetings can be a small, productive group of people 13 | * Often better to have multiple stakeholders together so people can brainstorm together, but 1:1 may be useful too 14 | * Present the draft to the teams and take feedback 15 | * Review issues raised on the repo (see below) 16 | * Adjust draft in response to the above comments 17 | 18 | ## Issues on the repo 19 | 20 | In addition to the active outreach to stakeholders, anyone can submit feedback by opening issues on the repositories storing the draft evaluations. These reposies will have issue categories with templates that categorize the feedback and provide some structure. For example: 21 | 22 | * Experience report 23 | * Proposal feedback 24 | * Crazy new idea 25 | 26 | [initiative owner]: ./owners.md 27 | [owners]: ./owners.md 28 | [stakeholders]: ./stakeholders.md 29 | 30 | -------------------------------------------------------------------------------- /src/vision/how_to_vision/owners.md: -------------------------------------------------------------------------------- 1 | # Owning a goal or initiative 2 | 3 | This page describes the roles and responsibilities associated with being the **owner** of an item on the [roadmap](../roadmap.md). Roadmap items fall into two categories, top-level goals and initiatives. In both cases, being an owner means that you are responsible for ensuring that the item gets done, but the details of owning a top-level goal are different from owning an initiative. 4 | 5 | ## Summary 6 | 7 | Goal owners are responsible for splitting their area into a set of **initiatives**. These can be active or on hold. 8 | 9 | They are also responsible for ensuring that for each initiative: 10 | 11 | - An owner is assigned 12 | - A landing page exists 13 | - Milestones are defined on the landing page 14 | - Stakeholders are identified and looped in at the proper stages 15 | 16 | Finally, they are expected to attend sprint meetings. 17 | 18 | ## Sprint meetings 19 | 20 | We are organizing the working group in **two week sprints**. This means that every two weeks we have a sprint planning meeting. **All goal owners are expected to attend!** Initiative owners or other contributors are welcome as well. 21 | 22 | The purpose of the sprint planning meeting is to check-in on the progress towards the milestones for each initiative and to see if they need to be adjusted. It's also a chance to raise interesting questions or get advice about tricky things or unexpected problems, as well as to celebrate our progress. 23 | 24 | ## Owning a top-level goal 25 | 26 | As the owner of a **top-level goal** your role is to figure out overall plan for how that goal will be achieved and to track progress. This means breaking up the goal into different initiatives, finding owners for those initiatives (which can be you!), and helping those owners to plan milestones. You are also generally responsible for staying on top of the state of things and updating other owners as to new or interesting developments. 27 | 28 | ## Owning an initiative 29 | 30 | Our definition of [initiative] is precisely the same as that used by the Rust lang team: it corresponds to a some active effort with a clear goal or deliverable(s). As the owner of an initiative, your role is to ensure that the work gets done (Which doesn't necessarily mean you do it yourself, it may be that you instead coordinate with volunteers or other implementors). You also guide the design of the deliverables within the initiative. 31 | 32 | As in the lang team process, the role of the owner is not to make the final decision (that belongs to the relevant rust team(s)), but to develop the "menu" of design choices, elaborate the tradeoffs involved, and make recommendations. For particularly complex designs, these evaluations will take the form of [evaluation documents] and are developed in collaboration with a defined set of [stakeholders]. 33 | 34 | [initiative]: https://lang-team.rust-lang.org/initiatives.html 35 | [initiative owners]: https://lang-team.rust-lang.org/initiatives/process/roles/owner.html 36 | [evaluation documents]: ./evaluations.md 37 | 38 | ### Making a landing page 39 | 40 | Each initiative should have a landing page, linked to from the [roadmap]. This can be a page on this website or a dedicated repo. 41 | 42 | For in-progress initiatives the landing page should include, or have pointers to: 43 | 44 | - Goals and impact of the initiative 45 | - Milestones 46 | - Design notes and documentation 47 | - Links to any organizing tools, such as a project board 48 | - The initiative owner 49 | - The current set of [stakeholders] and the area(s) they represent 50 | - Notes on how to get involved 51 | - For landing pages not on this website, a link back to the overall [roadmap] 52 | 53 | For making a dedicated repo, it's recommended to use this [initiative template][template] as a starting point. 54 | 55 | [roadmap]: ../roadmap.md 56 | [template]: https://github.com/rust-lang/initiative-template 57 | 58 | ### Planning initiative milestones 59 | 60 | When you own an initiative, you should work with the owner of the top-level goal and others to plan out a series of **milestones** around the initiative. These milestones correspond to the various steps you need to take to complete the initiative. 61 | 62 | Milestones are not fixed and they frequently change as you progress. They usually start out quite vague, such as "author an RFC", and then get more precise as you learn more about what is required: "figure out the design for X", "implement feature Y". We update the status and set of milestones for each sprint status meeting. 63 | 64 | [stakeholders]: ./stakeholders.md -------------------------------------------------------------------------------- /src/vision/how_to_vision/stakeholders.md: -------------------------------------------------------------------------------- 1 | # Stakeholders 2 | 3 | Many initiatives in the [roadmap] have an associated set of **stakeholders**. The role of a stakeholder is as follows: 4 | 5 | * They are consulted by the owner over the course of working on the initiative. 6 | * They do not have veto power; that belongs to the team. 7 | * When they do raise concerns, those concerns should either be addressed in the design or discussed explicitly in the FAQ. 8 | 9 | Stakeholders can be: 10 | 11 | * Domain experts (perhaps from other languages) 12 | * Representatives from major libraries 13 | * Production users 14 | 15 | Stakeholders can be selected in coordination with the async working group leads. Potential new stakeholders can also get in touch with the owner. 16 | 17 | ## Feedback on the design 18 | 19 | One role for stakeholders is to give feedback on the design as it progresses. Stakeholders are thus consulted in course of preparing evaluation docs or RFCs. 20 | 21 | ## Experimenting with the implementation 22 | 23 | Another role for stakeholders is evaluating the implemenation. This is partiularly important for production users. Stakeholders might, for example, agree to port their code to use the nightly version of the feature and adapt it as the design evolves. 24 | 25 | ## Goals of the stakeholder program 26 | 27 | The goal of the stakeholder program is to make Rust's design process even more inclusive. We have observed that existing mechanisms like the RFC process or issue threads are often not a very good fit for certain categories of users, such as production users or the maintainers of large libraries, as they are not able to keep up with the discussion. As a result, they don't participate, and we wind up depriving ourselves of valuable feedback. The stakeholder program looks to supplement those mechanisms with direct contact. 28 | 29 | Another goal is to get more testing: one problem we have observed is that features are often developed and deployed on nightly, but production users don't really want to try them out until they hit stable! We would like to get some commitment from people to give things a try so that we have a better chance of finding problems before stabilization. 30 | 31 | We want to emphasize that we welcome design feedback from **all Rust users**, regardless of whether you are a named stakeholder or not. If you're using async Rust, or have read through the designs and have a question or idea for improvement, please feel free to open an issue on the appropriate repository. 32 | -------------------------------------------------------------------------------- /src/vision/projects.md: -------------------------------------------------------------------------------- 1 | # ⚡ Projects 2 | 3 | ## What is this? 4 | 5 | This section describes various sample projects that are referenced in our stories. Each project is meant to represent some domain that we are targeting. 6 | 7 | ## List of projects 8 | 9 | See the sidebar for the full list. 10 | 11 | ## Don't find a project like yours here? 12 | 13 | Don't despair! This is just a list of fun projects that we've needed for stories. If you'd like to add a project for your story, feel free to do so! Note though that you may find that some existing project has the same basic characteristics as your project, in which case it's probably better to reuse the existing project. 14 | -------------------------------------------------------------------------------- /src/vision/projects/DistriData.md: -------------------------------------------------------------------------------- 1 | # ⚡ Projects: DistriData (Generic Infrastructure) 2 | 3 | ## What is this? 4 | 5 | This is a sample project for use within the various ["status quo"] or ["shiny future"] stories. 6 | 7 | ["status quo"]: ../status_quo.md 8 | ["shiny future"]: ../shiny_future.md 9 | 10 | ## Description 11 | 12 | DistriData is the latest in containerized, micro-service distributed database technology. Developed completely in the open as part of Cloud Native Computing Foundation, this utility is now deployed in a large portion of networked server applications across the entire industry. Since it's so widely used, DistriData has to balance flexibility with having sensible defaults. 13 | 14 | ## 🤔 Frequently Asked Questions 15 | 16 | ### **What makes DistriData different from others?** 17 | * This project is meant to be used in many different ways in many different projects, and is not unique to any one application. 18 | * Many of those using this project will not even need or want to know that it's written in Rust. 19 | 20 | ### **Does DistriData require a custom tailored runtime?** 21 | DistriData's concerns are at a higher level than the runtime. A fast, reliable, and resource conscious general purpose runtime will serve DistriData's needs. 22 | 23 | ### **How much of this project is likely to be built with open source components from crates.io?** 24 | Yes, while DistriData receives many contributions, it's important to the team that when possible they utilize existing technologies that developers are already familiar with to ensure that contributing to the project is easy. 25 | 26 | ### **What is of most concern to this project?** 27 | It needs to be resource conscious, fast, reliable, but above all else it needs to be easy to run, monitor, and maintain. 28 | 29 | ### **What is of least concern to this project?** 30 | While DistriData is resource conscious, it's not resource *starved*. There's no need to make life difficult to save on a memory allocation here or there. 31 | -------------------------------------------------------------------------------- /src/vision/projects/MonsterMesh.md: -------------------------------------------------------------------------------- 1 | # ⚡ Projects: MonsterMesh (embedded sensors) 2 | 3 | ## What is this? 4 | 5 | This is a sample project for use within the various ["status quo"] or ["shiny future"] stories. 6 | 7 | ["status quo"]: ../status_quo.md 8 | ["shiny future"]: ../shiny_future.md 9 | 10 | ## Description 11 | 12 | "MonsterMesh" is a sensor mesh on microcontrollers using Rust. The nodes communicate wirelessly to relay their results. These sensors are built using very constrained and low power hardware without operating system, so the code is written in a `#[no_std]` environment and is very careful about available resources. 13 | 14 | ## 🤔 Frequently Asked Questions 15 | 16 | ### **What makes embedded projects like MonsterMesh different from others?** 17 | * Embedded developers need to write error-free applications outside of the comfort zone of an operating system. Rust helps to prevent many classes of programming errors at compile time which inspires confidence in the software quality and and cuts time intensive build-flash-test iterations. 18 | * Embedded developers needs good hardware abstraction. Frameworks in other languages do not provide the sophisticated memory mapped IO to safe type abstraction tooling which have been created by the Rust teams. 19 | * Embedded developers care about hard real time capabilities; the concept of "you only pay for what you use" is very important in embedded applications. The combination of the inherently asynchronous interrupt handling of microcontrollers with the Rust async building blocks are a perfect match to effortlessly create applications with hard realtime capabilities. 20 | * Embedded developers are particularly appreciative of strong tooling support. The availability of the full environment via `rustup` and the integration of the full toolchain with `cargo` and `build.rs` make her very happy because she can focus on what she does best instead of having regular fights with the environment. 21 | 22 | ### **Does MonsterMesh require a custom tailored runtime?** 23 | Yes! The tradeoffs for an embedded application like MonsterMesh and a typical server are very different. Further, most server-grade frameworks are not `#[no_std]` compatible and far exceeded the available footprint on the sensor nodes. 24 | 25 | ### **How much of this project is likely to be built with open source components from crates.io?** 26 | Having no operating system to provide abstractions to it, MonsterMesh will contain all the logic it needs to run. Much of this, especially around the hardware-software-interface is unlikely to be unique to MonsterMesh and will be sourced from crates.io. However, the further up the stack one goes, the more specialized the requirements will become. 27 | 28 | ### **How did you pick the name?** 29 | So glad you asked! Please watch this [entertaining video](https://www.youtube.com/watch?v=vNuVifA7DSU). 30 | -------------------------------------------------------------------------------- /src/vision/projects/SLOW.md: -------------------------------------------------------------------------------- 1 | # ⚡ Projects: SLOW (Protocol implementation) 2 | 3 | ## What is this? 4 | 5 | This is a sample project for use within the various ["status quo"] or ["shiny future"] stories. 6 | 7 | ["status quo"]: ../status_quo.md 8 | ["shiny future"]: ../shiny_future.md 9 | 10 | ## Description 11 | 12 | SLOW is an open source implementation of a fancy new protocol. This protocol uses a mix of TCP and UDP packets and is designed to operate particularly well over high latency, low throughput links. 13 | 14 | ## 🤔 Frequently Asked Questions 15 | 16 | ### **What makes this project different from others?** 17 | SLOW is a library, not an application. 18 | 19 | ### **Does this project require a custom tailored runtime?** 20 | Ideally, SLOW would be developed in an independent way that permits it to be used across many runtimes in a variety of different environments. 21 | 22 | ### **How much of this project is likely to be built with open source components from crates.io?** 23 | SLOW builds on other generic libraries available from crates.io. For example, it would like to make use of compression algorithms that others have written, or to use future adapters. 24 | 25 | ### **What is of most concern to this project?** 26 | Uh, I don't really know! If you develop software like this, maybe open a PR and tell me! --nikomatsakis 27 | 28 | ### **What is of least concern to this project?** 29 | Uh, I don't really know! If you develop software like this, maybe open a PR and tell me! --nikomatsakis 30 | 31 | ### **Why is this called SLOW?** 32 | It's like [QUIC](https://en.wikipedia.org/wiki/QUIC), but slow! Get it? Get it? :D 33 | -------------------------------------------------------------------------------- /src/vision/projects/TrafficMonitor.md: -------------------------------------------------------------------------------- 1 | # ⚡ Projects: TrafficMonitor (Custom Infrastructure) 2 | 3 | ## What is this? 4 | 5 | This is a sample project for use within the various ["status quo"] or ["shiny future"] stories. 6 | 7 | ["status quo"]: ../status_quo.md 8 | ["shiny future"]: ../shiny_future.md 9 | 10 | ## Description 11 | 12 | TrafficMonitor is a utility written by AmoogleSoft, a public cloud provider, for monitoring network traffic as it comes into its data centers to prevent things like distributed denial-of-service attacks. It monitors *all* network traffic, looking for patterns, and deciding when to take action against certain threat vectors. TrafficMonitor runs across almost all server racks in a data center, and while it does run on top of an operating system, it is resource conscious. It's also extremely important that TrafficMonitor stay running and handle network traffic with as few "hiccups" as possible. TrafficMonitor is highly tuned to the needs of AmoogleSoft's cloud offering and won't run anywhere else. 13 | 14 | ## 🤔 Frequently Asked Questions 15 | 16 | ### **What makes networking infrastructure projects like TrafficMonitor different from others?** 17 | * Networking infrastructure powers entire datacenters or even public internet infrastructure, and as such it is imperative that it run without failure. 18 | * It is also extremely important that such projects take few resources as possible. Being on an operating system and large server racks *may* mean that using the standard library is possible, but memory and CPU usage should be kept to a minimum. 19 | * This project is worked on by software developers with different backgrounds. Some are networking infrastructure experts (usually using C) while others have experience in networked applications (usually using GCed languages like Java, Go, or Node). 20 | 21 | ### **Does TrafficMonitor require a custom tailored runtime?** 22 | Maybe? TrafficMonitor runs on top of a full operating system and takes full advantage of that operating systems networking stack. It's possible that a runtime meant for server workloads will work with TrafficMonitor. 23 | 24 | ### **How much of this project is likely to be built with open source components from crates.io?** 25 | * TrafficMonitor is highly specialized to the internal workings of AmoogleSoft's public cloud offering. Thus, "off-the-shelf" solutions will only work if they're highly flexible and highly tuneable. 26 | * TrafficMonitor is central to AmoogleSoft's success meaning that getting things "just right" is much more important than having something from crates.io that mostly works but requires little custom tuning. 27 | 28 | ### **What is of most concern to this project?** 29 | * Reliability is the number one concern. This infrastructure is at the core of the business - it needs to work extremely reliable. A close second is being easily monitorible. If something goes wrong, AmoogleSoft needs to know very quickly what the issue is. 30 | * AmoggleSoft is a large company with many existing custom tooling for building, monitoring, and deploying its software. TrafficMonitor has to play nicely in a world that existed long before it came around. 31 | 32 | ### **What is of least concern to this project?** 33 | AmoogleSoft is a large company with time and resources. High-level frameworks that remove control in favor of peak developer productivity is not what they're after. Sure, the easier things are to get working, the better, but that should not be at the sacrifice of control. 34 | -------------------------------------------------------------------------------- /src/vision/projects/YouBuy.md: -------------------------------------------------------------------------------- 1 | # ⚡ Projects: YouBuy (Traditional Server Application) 2 | 3 | ## What is this? 4 | 5 | This is a sample project for use within the various ["status quo"] or ["shiny future"] stories. 6 | 7 | ["status quo"]: ../status_quo.md 8 | ["shiny future"]: ../shiny_future.md 9 | 10 | ## Description 11 | 12 | YouBuy is a growing e-commerce website that now has millions of users. The team behind YouBuy is struggling to keep up with traffic and keep server costs low. Having originally written YouBuy in a mix of Ruby on Rails and Node, the YouBuy team decides to rewrite many parts of their service in Rust which they've investigated and found to be performant while still allowing for high levels of abstraction they're used to. 13 | 14 | ## 🤔 Frequently Asked Questions 15 | 16 | ### **What makes YouBuy and other server applications different from others?** 17 | * Many server applications are written in languages with garbage collectors. Many of the things that Rust forces users to care about are not first order concerns for those working on server applications (e.g., memory management, stack vs heap allocations, etc.). 18 | * Many server applications are written in languages without static type checking. The developers of YouBuy don't have much experience with statically typed languages and some of the developers early in their Rust learning journeys expressed frustration that they found it hard to get their programs to compile especially when using async constructs. 19 | 20 | ### **Does YouBuy require a custom tailored runtime?** 21 | YouBuy should be perfectly fine with a runtime from crates.io. In fact, their concern isn't at the runtime level but at the high-level server framework level. 22 | 23 | ### **How much of this project is likely to be built with open source components from crates.io?** 24 | YouBuy is in fierce competition with many other e-commerce sites. Therefore, the less that YouBuy engineers have to write themselves, the better. Ideally, YouBuy can focus 100% of its energy on features that differentiate it from its competition and none of its time on tweaking its networking stack. 25 | 26 | ### **What is of most concern to this project?** 27 | It seems like YouBuy is always on the verge of either becoming the next billion-dollar company with hundreds of millions of users or completely going out of business. YouBuy needs to be able to move fast and focus on the application business logic. 28 | 29 | ### **What is of least concern to this project?** 30 | Since moving fast is of primary concern, the ins and outs of the underlying networking stack are only of concern when something goes wrong. The hope is that that rarely if ever happens and when it does, it's easy to find the source of the issue. 31 | -------------------------------------------------------------------------------- /src/vision/projects/template.md: -------------------------------------------------------------------------------- 1 | # ⚡ Projects: NAME (DOMAIN) 2 | 3 | *This is a template for adding new projects. See the [instructions] for more details on how to add new project!* 4 | 5 | [instructions]: ../projects.md 6 | 7 | ## What is this? 8 | 9 | This is a sample project for use within the various ["status quo"] or ["shiny future"] stories. 10 | 11 | ["status quo"]: ../status_quo.md 12 | ["shiny future"]: ../shiny_future.md 13 | 14 | ## Description 15 | 16 | *Give a fun description of the project here! Include whatever details are needed.* 17 | 18 | ## 🤔 Frequently Asked Questions 19 | 20 | ### **What makes this project different from others?** 21 | 22 | ### **Does this project require a custom tailored runtime?** 23 | 24 | ### **How much of this project is likely to be built with open source components from crates.io?** 25 | 26 | ### **What is of most concern to this project?** 27 | 28 | ### **What is of least concern to this project?** 29 | -------------------------------------------------------------------------------- /src/vision/roadmap/async_fn.md: -------------------------------------------------------------------------------- 1 | # Async fn everywhere 2 | 3 | ## Impact 4 | 5 | * To a first-order approximation, any place that you can write some sort of Rust function or closure, you should be able to make it asynchronous: 6 | * [in traits and closures, including the Drop trait](https://rust-lang.github.io/async-fundamentals-initiative/) 7 | * in [main and tests](./async_fn/async_main_and_tests.md) 8 | * You should be able to [easily create futures that heap allocate their storage](./async_fn/boxable.md), both for performance tuning and for scenarios like recursive functions 9 | -------------------------------------------------------------------------------- /src/vision/roadmap/async_fn/async_main_and_tests.md: -------------------------------------------------------------------------------- 1 | # Async main and tests 2 | 3 | ## Impact 4 | 5 | * Able to write `#[test]` that easily use async functions. 6 | * In the case of portable libraries, end users are able to re-run test suites with distinct runtimes. 7 | 8 | ## Milestones 9 | 10 | > Able to write `async fn main` and `#[test] async fn` just like you would in synchronous code. 11 | 12 | This initiative is **on hold** while we investigate mechanisms for [portability across runtimes](../portable.md). 13 | 14 | -------------------------------------------------------------------------------- /src/vision/roadmap/async_fn/boxable.md: -------------------------------------------------------------------------------- 1 | # Boxable async fn 2 | 3 | ## Impact 4 | 5 | * Able to easily cause some async functions, blocks, or closures to allocate their stack space lazilly when called (by 'boxing' it) 6 | * Combined with profiler or other tooling support, this can help to tune the size of futures 7 | * Boxed async blocks allows particular *portions* of a function to be boxed, e.g. cold paths 8 | 9 | ## Milestones 10 | 11 | | Milestone | State | Key participants | 12 | | --- | --- | --- | 13 | | Author [evaluation doc] | 💤 | | 14 | | [Feature complete] implementation | 💤 | | 15 | 16 | [evaluation doc]: ./roadmap/stages.html#evaluation 17 | [stabilize]: https://lang-team.rust-lang.org/initiatives/process/stages/stabilized.html 18 | [feature complete]: https://lang-team.rust-lang.org/initiatives/process/stages/feature_complete.html 19 | 20 | ## Design notes 21 | 22 | Example might be to use a decorator: 23 | 24 | ```rust 25 | #[boxed] 26 | async fn foo() { } 27 | ``` 28 | 29 | This does not have to desugar to `-> Box>`; it can instead desugar to `Box`, or perhaps a nominal type to permit recursion. 30 | 31 | Another approach is the `box` keyword: 32 | 33 | ```rust 34 | box async fn foo() { } 35 | ``` 36 | 37 | We can apply the keyword modifier to async blocks and closures: 38 | 39 | ```rust 40 | fn foo() -> BoxFuture { 41 | box async { ... } 42 | } 43 | ``` 44 | 45 | ```rust 46 | async fn stuff(s: impl AsyncIterator) { 47 | s.map(box async |x| { ... }) 48 | } 49 | ``` 50 | 51 | This is useful for breaking up future types to make them more shallow. 52 | -------------------------------------------------------------------------------- /src/vision/roadmap/async_iter.md: -------------------------------------------------------------------------------- 1 | # Flexible async iteration 2 | 3 | ## Impact 4 | 5 | * Able to create and compose iterators that await async results 6 | * Able to create complex parallel or concurrent schedules that work reliably 7 | -------------------------------------------------------------------------------- /src/vision/roadmap/async_iter/generators.md: -------------------------------------------------------------------------------- 1 | # Generators 2 | 3 | ## Impact 4 | 5 | * Able to write iterators (and async iterators) with ease, comparable to languages like Python or JavaScript 6 | * Able to extend the resulting iterators with "optimization" traits like `ExactSizeIterator` for maximum efficiency -------------------------------------------------------------------------------- /src/vision/roadmap/async_iter/traits.md: -------------------------------------------------------------------------------- 1 | # Async iteration 2 | 3 | ## Impact 4 | 5 | * Able to write code that takes "something iterable" 6 | * Able to use combinators similar to synchronous `Iterator` 7 | * Able to construct complex, parallel schedules that [can refer to borrow data](../borrowed_data_and_cancellation.md) 8 | 9 | ## Requires 10 | 11 | * [inline async functions](../async_fn/inline_async_fn.md), but it is possible to prototype and experiment without that 12 | * [borrowed data and cancellation](../borrowed_data_and_cancellation.md), but it is possible to prototype and experiment without that 13 | 14 | ## Design notes 15 | 16 | The async iterator trait can leverage [inline async functions](../async_fn_everywhere/inline_async_fn.md): 17 | 18 | ```rust 19 | #[repr(inline_async)] 20 | trait AsyncIterator { 21 | type Item; 22 | 23 | async fn next(&mut self) -> Self::Item; 24 | } 25 | ``` 26 | 27 | Note the name change from `Stream` to `AsyncIterator`. 28 | 29 | One implication of this change is that pinning is no longer necessary when driving an async iterator. For example, one could now write an async iterator that recursively walks through a set of URLs like so (presuming `std::async_iter::from_fn` and [async closures](https://rust-lang.github.io/async-fundamentals-initiative/design-discussions/async_closures.html)): 30 | 31 | ```rust 32 | fn explore(start_url: Url) -> impl AsyncIterator { 33 | let mut urls = vec![start_url]; 34 | std::async_iter::from_fn(async move || { 35 | if let Some(url) = urls.pop() { 36 | let mut successor_urls = fetch_successor_urls(url).await; 37 | urls.extend(successor_urls); 38 | Some(url) 39 | } else { 40 | None 41 | } 42 | }) 43 | } 44 | ``` 45 | 46 | ### Parallel async iteration 47 | 48 | We should have combinators like `buffered` that enable *parallel* async iteration, similar to the parallel iterators offered by [rayon]. The core operation here is `for_each` (which processes each item in the iterator): 49 | 50 | ```rust 51 | trait ParAsyncIter { 52 | type Item; 53 | 54 | async fn for_each(&mut self, op: impl AsyncFn(Self::Item)); 55 | } 56 | ``` 57 | 58 | The `buffered` combinator would be implemented by creating an internal scope and spawning tasks into it as needed. -------------------------------------------------------------------------------- /src/vision/roadmap/async_overloading.md: -------------------------------------------------------------------------------- 1 | # Async overloading 2 | 3 | ## Impact 4 | 5 | * By default, function definitions can be compiled into either sync or async mode 6 | * Able to overload a function with two variants, one for sync and one for async 7 | 8 | ## Design notes 9 | 10 | This is a highly speculative deliverable. However, it would be great if one were able to write code that is neither sync nor sync, but potentially *either*. Further, one should be able to provide *specialized* variants that perform the same task but in slightly different ways; this would be particularly useful for primitives like TCP streams. 11 | 12 | ### Monomorphize 13 | 14 | The way to think of this is that every function has an implicit generic parameter indicating its *scheduler mode*. When one writes `fn foo()`, that is like creating a generic impl: 15 | 16 | ```rust 17 | impl Fn<(), SM> for Foo 18 | where 19 | SM: SchedulerMode, 20 | { 21 | ... 22 | } 23 | ``` 24 | 25 | When one writes `async fn` or `sync fn`, those are like providing specific impls: 26 | 27 | ```rust 28 | impl Fn<(), AsyncSchedulerMode> for Foo { 29 | ... 30 | } 31 | 32 | impl Fn<(), SchedulerMode> for Foo { 33 | ... 34 | } 35 | ``` 36 | 37 | Further, by default, when you call a function, you invoke it in the same scheduler mode as the caller. 38 | 39 | ### Implications for elsewhere 40 | 41 | * If we had this feature, then having distinct modules like `use std::io` and `use std::async_io` would not be necessary. 42 | * Further, we would want to design our traits and so forth to have a "common subset" of functions that differ only in the presence or absence of the keyword `async`. 43 | 44 | ### Related work 45 | 46 | * [SE-0296: Allow overloads that differ only in async](https://github.com/apple/swift-evolution/pull/1392) 47 | * [Async Overloading (Yoshua Wuyts, 2021)](https://blog.yoshuawuyts.com/async-overloading/) 48 | -------------------------------------------------------------------------------- /src/vision/roadmap/documentation.md: -------------------------------------------------------------------------------- 1 | # Documentation 2 | 3 | ## Impact 4 | 5 | * Quality, easily findable documentation to help folks get started with async Rust 6 | 7 | ## Requires 8 | 9 | * [Async book](./documentation/async_book.md) 10 | -------------------------------------------------------------------------------- /src/vision/roadmap/documentation/async_book.md: -------------------------------------------------------------------------------- 1 | # Async book 2 | 3 | ## Impact 4 | 5 | * Centralized documentation explainined how Async Rust works 6 | * Docs explain how to get started, identify common patterns, and cover concepts that are common to all or most runtimes 7 | -------------------------------------------------------------------------------- /src/vision/roadmap/polish.md: -------------------------------------------------------------------------------- 1 | # Polish 2 | 3 | ## Impact 4 | 5 | * Users can predict and understand why the compiler raises error messages. Errors are aligned with an experienced user's intuition about how Rust works. 6 | * Error messages identify common misconceptions, suggest solutions, and are generally on par with sync Rust. 7 | * Errors not only show that there is a problem, they help the user to fix it and to learn more about Rust (possibly directing the user to other documentation). 8 | * The compiler may suggest crates from the ecosystem to help solve problems when appropriate. 9 | * Lints guide the user away from common errors and help them both to get started with async Rust and to maintain async Rust programs over time. 10 | * Rust's async implementation is high quality and reflects an attention to detail. 11 | * No internal compiler errors 12 | * Compiler analysis and code generation passes are precise and not unnecessarily conservative. 13 | * Integration with low-level tooling and the like is high-quality. 14 | * The generated code from the compiler is high quality and performant. 15 | 16 | ## 🛠️ How to Help 17 | 18 | The goal of a highly polished async experience in Rust has many details and touches many aspects of the project, including both the async area in particular and the Rust project in general. 19 | This means there are lots of ways to get involved! 20 | 21 | The weekly [triage meeting] primarily focuses on polish issues, so that is a great place to get to know people already working on the project and find out what people are actively working on. 22 | We meet over Zulip, so feel free to just lurk, or chime in if you want to. 23 | See the [triage meeting] page for details about when the meeting happens and how to join. 24 | 25 | Even outside of regularly scheduled meetings, you are welcome to hang out in the Async Working Group's [Zulip stream]. 26 | There are usually a few people active there who are happy to discuss async-related topics. 27 | 28 | If you are looking for a specific area to help, there are several places where we track work. 29 | 30 | * The [Initiatives](#initiatives) list down below. 31 | * The Async Work Group [Project Board]. The "On Deck" column is a good place to start looking. 32 | * Issues on the [wg-async repo]. These tend to relate to project organization and longer term objectives. 33 | * Issues on the [Rust repo]. Specifically, issues tagged [AsyncAwait-Polish], [A-async-await]. Issues that are also tagged with E-mentor will have mentoring instructions, which are usually pointers to specific points in the code where changes will be needed to fix the issue. 34 | 35 | Finally, a great way to contribute is to point out any rough edges you come across with writing async Rust. 36 | This can be done either through issues on the [Rust repo], or by starting a topic on our [Zulip stream]. 37 | Examples of rough edges that we are interested in include confusing error messages or places where Rust behaved in a way you found surprising or counter-intuitive. 38 | Knowing about these issues helps to ensure we are fixing the right things. 39 | 40 | [A-async-await]: https://github.com/rust-lang/rust/labels/A-async-await 41 | [AsyncAwait-Polish]: https://github.com/rust-lang/rust/labels/AsyncAwait-Polish 42 | [Project Board]: https://github.com/orgs/rust-lang/projects/2 43 | [Rust repo]: https://github.com/rust-lang/rust/issues 44 | [Triage meeting]: ../../triage.md 45 | [wg-async repo]: https://github.com/rust-lang/wg-async/issues 46 | [Zulip stream]: https://rust-lang.zulipchat.com/#narrow/stream/187312-wg-async 47 | 48 | ## Initiatives 49 | 50 | | Initiative | State | Key participants | 51 | | --- | --- | --- | 52 | | [Error messages] | 💤 | | 53 | | Lint: [Must not suspend] | 🦀 | [Gus Wynn] | 54 | | Lint: [Blocking in async context] | 💤 | | 55 | | Lint: [Large copies], large generators | 💤 | | 56 | | [Cleaner async stacktraces] | 💤 | | 57 | | [Precise generator captures] | 🦀 | [eholk] | 58 | | [Sync and async behave the same] | 💤 | | 59 | 60 | [eholk]: https://github.com/eholk/ 61 | [Lang team]: https://www.rust-lang.org/governance/teams/lang 62 | [Blocking in async context]: ./polish/lint_blocking_fns.md 63 | [Large copies]: ./polish/lint_large_copies.md 64 | [Must not suspend]: ./polish/lint_must_not_suspend.md 65 | [RFC]: https://rust-lang.github.io/rfcs/3014-must-not-suspend-lint.html 66 | [Precise generator captures]: ./polish/precise_generator_captures.md 67 | [Gus Wynn]: https://github.com/guswynn 68 | [Error messages]: ./polish/error_messages.md 69 | [Cleaner async stacktraces]: ./polish/stacktraces.md 70 | [Sync and async behave the same]: ./polish/sync_and_async.md 71 | -------------------------------------------------------------------------------- /src/vision/roadmap/polish/error_messages.md: -------------------------------------------------------------------------------- 1 | # Error messages for most confusing scenarios 2 | 3 | ## Impact 4 | 5 | * Errors not only show that there is a problem, they help the user to fix it and to learn more about Rust (possibly directing the user to other documentation). 6 | 7 | ## Design notes 8 | 9 | Of course there are an infinite number of improvements one could make. The point of this deliverable is to target the *most common* situations and confusions people see in practice. The final list is still being enumerated: 10 | 11 | * Confusing error: Immutable reference to future is not a future [rust-lang/rust#87211](https://github.com/rust-lang/rust/issues/87211) 12 | -------------------------------------------------------------------------------- /src/vision/roadmap/polish/lint_blocking_fns.md: -------------------------------------------------------------------------------- 1 | # Lint blocking fns 2 | 3 | ## Impact 4 | 5 | * Identify calls to blocking functions from within async functions and guide the user to an async replacement. 6 | 7 | ## Milestones 8 | 9 | | Milestone | Status | Key Participants | 10 | | --- | --- | --- | 11 | | RFC proposed and accepted | 💤 | | 12 | | Implemented | 💤 | | 13 | -------------------------------------------------------------------------------- /src/vision/roadmap/polish/lint_large_copies.md: -------------------------------------------------------------------------------- 1 | # Lint large copies 2 | 3 | ## Impact 4 | 5 | * Identify when large types are being copied and issue a warning. This is particularly useful for large futures, but applies to other Rust types as well. 6 | 7 | ## Milestones 8 | 9 | | Milestone | Status | Key Participants | 10 | | --- | --- | --- | 11 | | [Lang team] initiative proposal | 💤 | | 12 | | Implemented | 💤 | | 13 | 14 | ## Design notes 15 | 16 | This is already implemented in experimental form. We would also need easy and effective ways to reduce the size of a future, though, such as [deliv_boxable](../async_fn/boxable.md). 17 | 18 | [Lang team]: https://www.rust-lang.org/governance/teams/lang 19 | -------------------------------------------------------------------------------- /src/vision/roadmap/polish/lint_must_not_suspend.md: -------------------------------------------------------------------------------- 1 | # Lint must not suspend 2 | 3 | ## Impact 4 | 5 | * Warnings when values which ought not to be live over an await are, well, live over an `await`. 6 | * Example: lock guards. 7 | 8 | ## Milestones 9 | 10 | | Milestone | Status | Key Participants | 11 | | --- | --- | --- | 12 | | Implemented the [RFC] | ✅ | [Gus Wynn] | 13 | | [Improve drop range tracking] | 🦀 | [Eric Holk] | 14 | | Stabilize the lint | 💤 |[Gus Wynn] | 15 | 16 | [RFC]: https://rust-lang.github.io/rfcs/3014-must-not-suspend-lint.html 17 | [Improve drop range tracking]: https://github.com/rust-lang/rust/pull/91032 18 | [Gus Wynn]: https://github.com/guswynn 19 | [Eric Holk]: https://github.com/eholk 20 | -------------------------------------------------------------------------------- /src/vision/roadmap/polish/precise_generator_captures.md: -------------------------------------------------------------------------------- 1 | # Precise Generator Captures 2 | 3 | ## Impact 4 | 5 | * Users can predict and understand why the compiler raises error messages. Errors are aligned with an experienced user's intuition about how Rust works. 6 | * Compiler analysis and code generation passes are precise and not unnecessarily conservative. 7 | 8 | ## Milestones 9 | 10 | | Milestone | Status | Key Participants | 11 | | --- | --- | --- | 12 | | Prototyped | 🦀 | [eholk] | 13 | | Documented in Rust Reference | 🦀 | [eholk] | 14 | | [Lang team] initiative proposal | 💤 | [eholk] | 15 | | [Lang team] signoff | 💤 | [Lang team] | 16 | | Stabilized | 💤 | [eholk] | 17 | 18 | [eholk]: https://github.com/eholk/ 19 | [Lang team]: https://www.rust-lang.org/governance/teams/lang 20 | -------------------------------------------------------------------------------- /src/vision/roadmap/polish/stacktraces.md: -------------------------------------------------------------------------------- 1 | # Stacktraces 2 | 3 | ## Impact 4 | 5 | * Async stacktraces contain only the information that people need to figure out what has happened, and are free of extraneous or runtime-internal details 6 | * Users are able to recover the full, unabridged stacktrace if needed 7 | 8 | See the [design notes] for details about the current state of async stack traces and proposals for how to improve them. 9 | 10 | [design notes]: ../../../design_docs/async_stack_traces.md 11 | -------------------------------------------------------------------------------- /src/vision/roadmap/portable.md: -------------------------------------------------------------------------------- 1 | # Portable across runtimes, easy to switch 2 | 3 | ## Impact 4 | 5 | * Able to grab libraries from crates.io and mix-and-match them with confidence, no matter what runtime or other libraries you are using 6 | * Able to easily author libraries that can be combined with other libraries and are independent of runtime 7 | * Able to easily change applications between runtimes to explore new possibilities 8 | * Able to easily author new runtimes that try out a new execution strategy or for some new environment and have them interoperate with most extant libraries, without the need to change those libraries 9 | * Able to find runtimes that fit a wide variety of scenarios and use patterns 10 | -------------------------------------------------------------------------------- /src/vision/roadmap/portable/runtime.md: -------------------------------------------------------------------------------- 1 | # Runtime 2 | 3 | ## Impact 4 | 5 | * Able to write simple, non-generic async Rust code that performs common operations like opening TCP sockets, sending UDP packets, accessing files, sleeping, and spawning tasks, but which is not specific to a particular runtime. 6 | * Able to retarget code that relies on these APIs across different runtimes with no effort. 7 | 8 | ## Design notes 9 | 10 | When writing sync code, it is possible to simply _access_ I/O and other facilities without needing to thread generics around: 11 | 12 | ```rust 13 | fn load_socket_addr() -> Result> { 14 | Ok(std::fs::read_to_string("address.txt")?.parse()?) 15 | } 16 | ``` 17 | 18 | This code will work no matter what operating system you run it on. 19 | 20 | Similarly, if you don't mind hard-coding your runtime, one can use `tokio` or `async_std` in a similar fashion 21 | 22 | ```rust 23 | // Pick one: 24 | // 25 | // use tokio as my_runtime; 26 | // use async_std as my_runtime; 27 | 28 | async fn load_socket_addr() -> Result> { 29 | Ok(my_runtime::fs::read_to_string("address.txt").await?.parse()?) 30 | } 31 | ``` 32 | 33 | Given suitable traits in the stdlib, it would be possible to write generic code that feels similar: 34 | 35 | ```rust 36 | async fn load_socket_addr() -> Result> { 37 | Ok(F::read_to_string("address.txt").await?.parse()?) 38 | } 39 | ``` 40 | 41 | Alternatively, that might be done with `dyn` trait: 42 | 43 | ```rust 44 | async fn load_socket_addr(fs: &dyn AsyncFs)) -> Result> { 45 | Ok(F::read_to_string("address.txt").await?.parse()?) 46 | } 47 | ``` 48 | 49 | Either approach is significantly more annoying, both as the author of the library and for folks who invoke your library. 50 | 51 | ### Preferred experience 52 | 53 | The ideal would be that you can write an async function that is "as easy" to use as a non-async one, and have it be portable across runtimes: 54 | 55 | ```rust 56 | async fn load_socket_addr() -> Result> { 57 | Ok(std::async_fs::read_to_string("address.txt").await?.parse()?) 58 | } 59 | ``` 60 | 61 | ### But how to achieve it? 62 | 63 | The basic idea is to extract out a "core API" of things that a runtime must provide and to make those functions available as part of the `Context` that `Async` values are invoked with. To avoid the need for generics and monomorphization, this would have to be based purely on `dyn` values. This interface ought to be compatible with no-std runtimes as well, which imposes some challenges. 64 | 65 | ## Frequently asked questions 66 | 67 | ### What about async overloading? 68 | 69 | Good question! The [async overloading](../async_overloading.md) feature may be another, better route to this same goal. At minimum it implies that `std::async_fs` etc might not be the right names (although those modules could be deprecated and merged going forward). 70 | 71 | It definitely suggests that the names and signatures of all functions, methods, and types should be kept very strictly analogous. In particular, sync APIs should be a subset of async APIs. 72 | 73 | ### What about cap-std? 74 | 75 | It's interesting to observe that the `dyn` approach is feeling very close to [cap-std](https://blog.sunfishcode.online/introducing-cap-std/). That might be worth taking into consideration. Some targets, like wasm, may well prefer if we took a more "capability oriented" approach. 76 | 77 | ### What about spawning and scopes? 78 | 79 | Given that spawning should occur through scopes, it may be that we don't need a `std::async_thread::spawn` API so much as standards for scopes. 80 | 81 | ### What about evolving the API? 82 | 83 | We will want to be able to start with a small API and grow it. How is that possible, given that the *implementation* of the API lives in external runtimes? 84 | 85 | ### What methods are needed? 86 | 87 | We need to cover the things that exist in the sync stdlib 88 | 89 | * spawn, spawn-blocking 90 | * timers (sleep) 91 | * TCP streams, UDP sockets 92 | * file I/O 93 | * channels and other primitives 94 | * mutexes? 95 | -------------------------------------------------------------------------------- /src/vision/roadmap/portable/spawn.md: -------------------------------------------------------------------------------- 1 | # Async spawn, spawn-blocking 2 | 3 | ## Impact 4 | 5 | * Able to write libraries or applications that use a trait to spawn async or blocking tasks without referring to a particular runtime 6 | * Able to use the trait in a dyn-safe fashion -------------------------------------------------------------------------------- /src/vision/roadmap/portable/timers.md: -------------------------------------------------------------------------------- 1 | # Async timer 2 | 3 | ## Impact 4 | 5 | * Able to write libraries or applications that use a trait to create a timer without referring to a particular runtime 6 | * Able to use the trait in a dyn-safe fashion -------------------------------------------------------------------------------- /src/vision/roadmap/scopes/capability.md: -------------------------------------------------------------------------------- 1 | # Capability 2 | 3 | ## Impact 4 | 5 | * The ability to create async tasks that can be safely given access to borrowed data, similar to crossbeam or rayon scopes 6 | * There are potentially multiple routes with which this can be accomplished 7 | 8 | ## Design notes 9 | 10 | Today's `Future` trait lacks one fundamental capability compared to synchronous code: there is no (known?) way to "block" your caller and be sure that the caller will not continue executing until you agree. In synchronous code, you can use a closure and a destructor to achieve this, which is the technique used for things like `rayon::scope` and crossbeam's scoped threads. In async code, because the `Future` trait has a safe poll function, it is always possible to poll it part way and then `mem::forget` (or otherwise leak) the value; this means that one cannot have parallel threads executing and using those references. 11 | 12 | Async functions are commonly written with borrowed references as arguments: 13 | 14 | ```rust 15 | async fn do_something(db: &Db) { ... } 16 | ``` 17 | 18 | but important utilities like `spawn` and `spawn_blocking` require `'static` tasks. Without "unfogettable" traits, the only way to circumvent this is with mechanisms like `FuturesUnordered`, which is then subject to footguns as described in [Barbara battles buffered streams](https://rust-lang.github.io/wg-async/vision/status_quo/barbara_battles_buffered_streams.html). 19 | 20 | There are two main approaches under consideration to address this issue: 21 | 22 | * [Introducing a new trait for futures, Async](./capability/variant_async_trait.md) 23 | * [Introducing a new "default" trait, Leak](./capability/variant_leak.md) that can be used to prevent values from leaking 24 | * If we say that scopes cannot be leaked, and the scope defines `AsyncDrop`, then we can (presumably) be sure that its destructor will run. 25 | -------------------------------------------------------------------------------- /src/vision/roadmap/scopes/capability/variant_async_trait.md: -------------------------------------------------------------------------------- 1 | # Variant: Async trait 2 | 3 | As proposed in https://github.com/Matthias247/rfcs/pull/1, one way to solve this is to introduce a new future trait with an unsafe poll method: 4 | 5 | ```rust 6 | trait Async { 7 | type Output; 8 | 9 | /// # Unsafe conditions 10 | /// 11 | /// * Once polled, cannot be moved 12 | /// * Once polled, destructor must execute before memory is deallocated 13 | /// * Once polled, must be polled to completion 14 | /// 15 | /// FIXME: Have to specify how to manage panic. 16 | unsafe fn poll( 17 | &mut self, 18 | context: &mut core::task::Context<'_>, 19 | ) -> core::task::Poll; 20 | } 21 | ``` 22 | 23 | This would then require "bridging impls" to convert the (now likely deprecated, or at least repurposed) Future trait: 24 | 25 | ```rust 26 | impl Async for F { .. } // impl A 27 | ``` 28 | 29 | which in turn creates an interesting question, since if we wish to have a single combinator that is usable with either trait, specialization would be required: 30 | 31 | ```rust 32 | impl Future for Combinator { .. } // impl B 33 | impl Async for Combinator { .. } // impl C 34 | 35 | // Coherence error: Given some type `F1: Future`, 36 | // two ways to show that `Combinator: Async`. 37 | ``` 38 | 39 | ## Bridging 40 | 41 | Introduce "bridge impls" like the following: 42 | 43 | ```rust 44 | impl Async for F where F: Future { 45 | 46 | } 47 | ``` 48 | 49 | Newer runtimes will be built atop the `Async` trait, but older code will still work with them, since everything that implements `Future` implements `Async`. 50 | 51 | #### Combinators 52 | 53 | One tricky case has to do with bridging combinators. If you have a combinator like `Join`: 54 | 55 | ```rust 56 | struct Join { ... } 57 | 58 | impl Future for Join 59 | where 60 | A: Future, 61 | B: Future, 62 | { } 63 | ``` 64 | 65 | This combinator cannot then be used with `Async` values. You cannot (today) add a second impl like the following for coherence reasons: 66 | 67 | ```rust 68 | impl Async for Join 69 | where 70 | A: Async, 71 | B: Async, 72 | { } 73 | ``` 74 | 75 | The problem is that this second impl creates multiple routes to implement `Async` for `Join` where `A` and `B` are futures. These routes are of course equivalent, but the compiler doesn't know that. 76 | 77 | ### Solution A: Don't solve it 78 | 79 | We might simply introduce new combinators for the `Async` trait. Particularly given the move to [scoped threads](./scoped.md) it is likely that the set of combinators would want to change anyhow. 80 | 81 | ### Solution B: Specialization 82 | 83 | Specialization can be used to resolve this, and it would be a great feature for Rust overall. However, specialization has a number of challenges to overcome. Some related articles: 84 | 85 | - [Maximally minimal specialization](https://smallcultfollowing.com/babysteps/blog/2018/02/09/maximally-minimal-specialization-always-applicable-impls/) 86 | - [Supporting blanket impls in specialization](https://smallcultfollowing.com/babysteps/blog/2016/10/24/supporting-blanket-impls-in-specialization/) 87 | -------------------------------------------------------------------------------- /src/vision/roadmap/scopes/capability/variant_leak.md: -------------------------------------------------------------------------------- 1 | # Variant: Leak trait 2 | 3 | (Requires elaboration) 4 | -------------------------------------------------------------------------------- /src/vision/roadmap/scopes/scope_api.md: -------------------------------------------------------------------------------- 1 | # API 2 | 3 | ## Impact 4 | 5 | * Able to create hierarchical scopes, easily spawn async & blocking tasks within those scopes, and propagate cancellation. 6 | * Able to select any runtime to back the API 7 | 8 | -------------------------------------------------------------------------------- /src/vision/roadmap/testing.md: -------------------------------------------------------------------------------- 1 | # Testing 2 | 3 | ## Impact 4 | 5 | * Async applications need the ability to write tests that let them simulate and mock the outside world 6 | * Ability to test edge cases: 7 | * Long latencies 8 | * Dropped connections 9 | * Funky schedules 10 | 11 | ## Design notes 12 | 13 | At the moment, this is an "experimentation" area, but it represents a common need without well-established, widely used solutions. -------------------------------------------------------------------------------- /src/vision/roadmap/threadsafe_portability.md: -------------------------------------------------------------------------------- 1 | # Threadsafe portability 2 | 3 | ## Impact 4 | 5 | * Able to write code that can be easily made `Send` or not `Send` 6 | * The resulting code is able to switch between helper types, like `Rc` and `Arc`, appropriately. -------------------------------------------------------------------------------- /src/vision/roadmap/tooling.md: -------------------------------------------------------------------------------- 1 | # Tooling 2 | 3 | ## Impact 4 | 5 | * Tooling that gives insight into the state of async runtimes 6 | * How many tasks are running and what is their state 7 | * What are tasks blocked on and why? 8 | * Where is memory allocated? 9 | * Where is CPU time spent? 10 | * Flamegraph of where things spend their time 11 | * Perf-style profile of where things spend their time 12 | * Tooling should also allow you to limit profiles to a particular request or to requests that meet particular criteria (e.g., coming from a particular source) 13 | * Tooling should detect common hazards and identify them, suggesting fixes 14 | * Tasks that clone a `Waker` but don't trigger it 15 | * Tasks that don't respond to a request to cancellation for a long time 16 | * Outlier tasks that sleep for a very long time without being awoken 17 | * Tooling should permit "always on" profiling that can be used in production 18 | * Tooling can provide profile-based feedback: 19 | * Where to "heap-allocate" futures 20 | * Poll functions that execute for a long time without yielding 21 | * Imbalanced workloads across cores 22 | * Tooling can be either customized or integrated into existing tools like perf, gdb, lldb, etc, as appropriate -------------------------------------------------------------------------------- /src/vision/roadmap/tooling/console.md: -------------------------------------------------------------------------------- 1 | # Console 2 | 3 | https://github.com/tokio-rs/console 4 | 5 | * Able to get live information about the state of the runtime and async tasks from a running program. 6 | * How many tasks are running and what is their state 7 | * Detect common hazards and identify them 8 | * Requires opt-in at program source level 9 | * https://hackmd.io/yjtVRDPjQJq05p_8L0-5UA 10 | * Incurs some amount of overhead 11 | * Todo: quantify overhead incurred 12 | -------------------------------------------------------------------------------- /src/vision/roadmap/tooling/crashdump.md: -------------------------------------------------------------------------------- 1 | # Crashdump 2 | 3 | * Able to get information about the state of the runtime and async tasks from crashdumps. -------------------------------------------------------------------------------- /src/vision/shiny_future.md: -------------------------------------------------------------------------------- 1 | # ✨ Shiny future 2 | 3 | This page represents a complete vision for where we want async to go. This vision is what we believe to be the best way to achieve the [experiences](./how_it_feels.md) that we want async to provide. 4 | 5 | ## Work in progress 6 | 7 | Note that while a lot of the steps needed are fairly clear, several of them also have [significant unknowns or points of controversy](./unresolved_questions.md). We have attempted to highlight those and expect to be working through those points as we go. 8 | 9 | ## Certainty levels 10 | 11 | - 🌈 -- Implemented and stable 12 | - 🌞 -- Everything is looking good 13 | - 🌤️ -- Still some stuff to figure out, but unlikely to see major changes in the design 14 | - 🌥️ -- Got one or two solid leads, but still have to figure out if it will work 15 | - 🌧️ -- No clear path yet, this may not even be a good idea 16 | 17 | ## Key aspects of the future 18 | 19 | * 🌤️ If you know sync Rust, getting started in Async Rust is straightforward ([more](roadmap/async_fn.md)]) 20 | * 🌤️ Mostly, you change `fn` to `async fn`, add some calls to await, and change over to other parts of the stdlib, though supporting `dyn Trait` requires making some choices, particularly in a no-std environment 21 | * 🌤️ It still has that "if it compiles, it generally works, and it runs pretty darn fast" feeling 22 | * 🌤️ Destructors and cleanup also work the same way as in sync Rust, thanks to `Drop` to `AsyncDrop` 23 | * 🌤️ No need to write poll functions or to interact with pin except in quite specialized scenarios 24 | * 🌤️ High-quality documentation and tutorials helps you to get started and learn the ropes 25 | * 🌤️ The docs also identify common patterns for structuring your async programs and their advantages and disadvantages 26 | * 🌥️ Tooling and debugger integration gives insight into the behavior of your program 27 | * 🌥️ Easy to get a snapshot of overall acitivity (e.g. to find out what tasks or exist or why a task is blocked) 28 | * 🌥️ Easy to see aggregate performance trends over time (e.g., number of active connections, waiting connections, etc) 29 | * 🌥️ Easy to profile things in terms of your async tasks (e.g., to get a flamegraph of a specific connection) 30 | * 🌥️ Variety of high-quality runtimes available in cargo, and it's easy to change between them: 31 | * 🌧️ When you use things from the standard library, they work across runtimes automatically 32 | * 🌥️ There are standardized, foundational traits for common operations like I/O, spawning tasks, timers 33 | * 🌥️ Hierarchical scopes allow you to easily spawn parallel and concurrent tasks 34 | * 🌥️ These can reference borrowed data, enabling easy parallel processing of async iterators (think "async rayon") 35 | * 🌥️ Cancellation works well and without surprises 36 | * 🌥️ When cancellation is requested, it propagates to subtasks within a scope 37 | * 🌧️ I/O operations and the like begin to fail, so that cancellation is automatic and flows through familiar error paths 38 | * 🌥️ If desired, you can "opt-in" to synchronous cancellation, in which case any await becomes a cancellation point. This allows your `async fn` to be used with `select` without spawning a task. 39 | 40 | ## Learn more 41 | 42 | Check out... 43 | 44 | * [The user's manual of the future](./shiny_future/users_manual.md) 45 | 46 | ## Where did all the stories go? 47 | 48 | The full set of "submitted" shiny future stories [have been moved here](./submitted_stories/shiny_future.md). 49 | -------------------------------------------------------------------------------- /src/vision/status_quo.md: -------------------------------------------------------------------------------- 1 | # 😱 Status quo 2 | 3 | ## Where did all the stories go? 4 | 5 | The full set of "submitted" status quo stories [have been moved here](./submitted_stories/status_quo.md). This area will be used for a "combined" status quo story which has not yet been written! 6 | -------------------------------------------------------------------------------- /src/vision/submitted_stories.md: -------------------------------------------------------------------------------- 1 | # 💝 Appendix: Submitted stories 2 | 3 | This appendix contains the full list of [status quo](./submitted_stories/status_quo.md) and [shiny future](./submitted_stories/shiny_future.md) stories that were submitted by users as part of the vision doc construction. The lessons and ideas from these stories have been incorporated into the current [roadmap]. 4 | 5 | [roadmap]: ./roadmap.md -------------------------------------------------------------------------------- /src/vision/submitted_stories/shiny_future.md: -------------------------------------------------------------------------------- 1 | # ✨ Shiny future: Where we want to get to 2 | 3 | ## 🚧 Under construction! Help needed! 🚧 4 | 5 | We are still in the process of drafting the vision document. The stories you see on this page are examples meant to give a feeling for how a shiny future story looks; you can expect them to change. See the ["How to vision"][htv] page for instructions and details. 6 | 7 | [htv]: ../how_to_vision.md 8 | 9 | ## What it this 10 | 11 | The "shiny future" is here to tell you what we are trying to build over the next 2 to 3 years. That is, it presents our "best guess" as to what will look like a few years from now. When describing specific features, it also embeds links to [design notes] that describe the constraints and general plans around that feature. 12 | 13 | 🧐 You may also enjoy reading the [blog post] announcing the brainstorming effort. 14 | 15 | [design notes]: ../design_notes.md 16 | [blog post]: https://blog.rust-lang.org/2021/04/14/async-vision-doc-shiny-future.html 17 | 18 | ### Think big -- too big, if you have to 19 | 20 | You'll notice that the ideas in this document are **maximalist and ambitious**. They stake out an opinionated position on how the ergonomics of Async I/O should feel. This position may not, in truth, be attainable, and for sure there will be changes along the way. Sometimes the realities of how computers actually work may prevent us from doing all that we'd like to. That's ok. This is a dream and a goal. 21 | 22 | We fully expect that the designs and stories described in this document will change as we work towards realizing them. When there are areas of particular uncertainty, we use the Frequently Asked Questions and the design docs to call them out. 23 | 24 | ## Where are the stories? 25 | 26 | We haven't written these yet! 27 | -------------------------------------------------------------------------------- /src/vision/submitted_stories/shiny_future/barbara_wants_async_rw.md: -------------------------------------------------------------------------------- 1 | # ✨ Barbara Wants Async Read Write 2 | 3 | ## 🚧 Warning: Draft status 🚧 4 | 5 | This is a draft "shiny future" story submitted as part of the brainstorming period. It is derived from what actual Rust users wish async Rust should be, and is meant to deal with some of the challenges that Async Rust programmers face today. 6 | 7 | If you would like to expand on this story, or adjust the answers to the FAQ, feel free to open a PR making edits (but keep in mind that, as peoples needs and desires for async Rust may differ greatly, shiny future stories [cannot be wrong]. At worst they are only useful for a small set of people or their problems might be better solved with alternative solutions). Alternatively, you may wish to [add your own shiny vision story][htvsq]! 8 | 9 | ## The story 10 | 11 | Character: Barbara. 12 | 13 | Barbara is the creator of a `sans-io` library for Rust. She designed her library to 14 | integrate with `async` and her goal was to make it runtime agnostic; so that it could 15 | be as broadly used as possible. Unfortunately, when she first wrote the library `async` 16 | did not have a standard abstraction for Buffered IO. So her first implementation did 17 | not use buffered IO. When she tried to update her library to use buffered IO so as to 18 | improve performance she was confronted with the problem that each runtime had its own 19 | implementation and abstractions. The result was several unavoidable compromises on her 20 | runtime-agnostic design goals. She was able to achieve her performance improvements 21 | but only with runtime specific implementations; leaving her with a larger more complex 22 | code base. 23 | 24 | But today is a fantastic day for Barbara. The Rust async team has recently released 25 | the latest version of `async` and part of that release was a standard Buffered Async 26 | Read/Write abstraction. Since then, several runtimes have been updated to implement 27 | the new abstraction and Barbara refactored the buffered IO module to use this new 28 | abstraction and she deprecated the runtime specific solutions. Today is the day that 29 | Barbara gets to release her new version of `sans-io` which takes full advantage of the 30 | buffered Async Read/Write abstractions now defined in `async`. The result is a library 31 | that maintains the same performance gains that it had with the runtime specific modules 32 | while greatly reducing the amount of code. 33 | 34 | ## 🤔 Frequently Asked Questions 35 | 36 | *NB: These are generic FAQs. Feel free to customize them to your story or to add more.* 37 | 38 | ### What status quo stories are you retelling? 39 | 40 | *Link to status quo stories if they exist. If not, that's ok, we'll help find them.* 41 | 42 | ### What are the key attributes of this shiny future? 43 | 44 | - Just like AsyncRead/AsyncWrite there are no standard traits for buffered I/O 45 | 46 | - This is made worse by the fact that there isn’t even ecosystem traits for buffered writes. 47 | 48 | - There are no standard (or even present in futures-io) concrete types for async buffered I/O. 49 | 50 | - Each major runtime has their own async BufReader, BufWriter types. 51 | 52 | - All the issues with creating runtime agnostic libraries are very present here. (TODO: link with runtime agnostic lib story) 53 | std::io doesn’t have a BufWrite trait for sync I/O. 54 | 55 | - It’s less of an issue than in async Rust because of the existence of the standardized std::io::BufWriter. 56 | 57 | 58 | 59 | ### What is the "most shiny" about this future? 60 | 61 | *Thing about Rust's core "value propositions": performance, safety and correctness, productivity. Which benefit the most relative to today?* 62 | This benefits productivity and correctness the most. The problem is not performance, in particular, as each runtime provides buffered IO solutions. The problem is that they are inconsistent and not compatible. This means that writing code that is compatible with any async runtime becomes both: much more difficult and much more likely to be wrong when runtimes change. 63 | 64 | ### What are some of the potential pitfalls about this future? 65 | 66 | *Thing about Rust's core "value propositions": performance, safety and correctness, productivity. Are any of them negatively impacted? Are there specific application areas that are impacted negatively? You might find the sample [projects] helpful in this regard, or perhaps looking at the goals of each [character].* 67 | - Having a design which makes it difficult for existing runtimes to make their buffered IO types compatible or to migrate their runtimes over to the new designs. 68 | 69 | ### Did anything surprise you when writing this story? Did the story go any place unexpected? 70 | 71 | *The act of writing shiny future stories can uncover things we didn't expect to find. Did you have any new and exciting ideas as you were writing? Realize some complications that you didn't foresee?* 72 | The most surprising thing is that there is a buffered read type in `futures` but no buffered *write* type in `futures`. I would expect both or neither. 73 | 74 | ### What are some variations of this story that you considered, or that you think might be fun to write? Have any variations of this story already been written? 75 | 76 | *Often when writing stories, we think about various possibilities. Sketch out some of the turning points here -- maybe someone will want to turn them into a full story! Alternatively, if this is a variation on an existing story, link back to it here.* 77 | No variations. 78 | 79 | ### What are some of the things we'll have to figure out to realize this future? What projects besides Rust itself are involved, if any? (Optional) 80 | 81 | *Often the 'shiny future' stories involve technical problems that we don't really know how to solve yet. If you see such problems, list them here!* 82 | 83 | 84 | 85 | [character]: ../../characters.md 86 | [comment]: ../../how_to_vision/comment.md 87 | [status quo stories]: ../status_quo.md 88 | [Alan]: ../../characters/alan.md 89 | [Grace]: ../../characters/grace.md 90 | [Niklaus]: ../../characters/niklaus.md 91 | [Barbara]: ../../characters/barbara.md 92 | [projects]: ../../projects.md 93 | [htvsq]: ../shiny_future.md 94 | [cannot be wrong]: ../../how_to_vision/comment.md#comment-to-understand-or-improve-not-to-negate-or-dissuade 95 | -------------------------------------------------------------------------------- /src/vision/submitted_stories/shiny_future/template.md: -------------------------------------------------------------------------------- 1 | # ✨ Shiny future stories: template 2 | 3 | *This is a template for adding new "shiny future" stories. To propose a new shiny future PR, do the following:* 4 | 5 | * *Create a new file in the [`shiny_future`] directory named something like `Alan_loves_foo.md` or `Grace_does_bar_and_its_great.md`, and start from [the raw source from this template]. You can replace all the italicized stuff. :)* 6 | * *Do not add a link to your story to the [`SUMMARY.md`] file; we'll do it after merging, otherwise there will be too many conflicts.* 7 | 8 | *For more detailed instructions, see the [How To Vision: Shiny Future] page!* 9 | 10 | [How To Vision: Shiny Future]: ../shiny_future.md 11 | [the raw source from this template]: https://raw.githubusercontent.com/rust-lang/wg-async/master/src/vision/shiny_future/template.md 12 | [`shiny_future`]: https://github.com/rust-lang/wg-async/tree/master/src/vision/shiny_future 13 | [`SUMMARY.md`]: https://github.com/rust-lang/wg-async/blob/master/src/SUMMARY.md 14 | 15 | 16 | ## 🚧 Warning: Draft status 🚧 17 | 18 | This is a draft "shiny future" story submitted as part of the brainstorming period. It is derived from what actual Rust users wish async Rust should be, and is meant to deal with some of the challenges that Async Rust programmers face today. 19 | 20 | If you would like to expand on this story, or adjust the answers to the FAQ, feel free to open a PR making edits (but keep in mind that, as peoples needs and desires for async Rust may differ greatly, shiny future stories [cannot be wrong]. At worst they are only useful for a small set of people or their problems might be better solved with alternative solutions). Alternatively, you may wish to [add your own shiny vision story][htvsq]! 21 | 22 | ## The story 23 | 24 | *Write your story here! Feel free to add subsections, citations, links, code examples, whatever you think is best.* 25 | 26 | ## 🤔 Frequently Asked Questions 27 | 28 | *NB: These are generic FAQs. Feel free to customize them to your story or to add more.* 29 | 30 | ### What status quo stories are you retelling? 31 | 32 | *Link to status quo stories if they exist. If not, that's ok, we'll help find them.* 33 | 34 | ### What are the key attributes of this shiny future? 35 | 36 | *Summarize the main attributes of the design you were trying to convey.* 37 | 38 | ### What is the "most shiny" about this future? 39 | 40 | *Thing about Rust's core "value propositions": performance, safety and correctness, productivity. Which benefit the most relative to today?* 41 | 42 | ### What are some of the potential pitfalls about this future? 43 | 44 | *Thing about Rust's core "value propositions": performance, safety and correctness, productivity. Are any of them negatively impacted? Are there specific application areas that are impacted negatively? You might find the sample [projects] helpful in this regard, or perhaps looking at the goals of each [character].* 45 | 46 | ### Did anything surprise you when writing this story? Did the story go any place unexpected? 47 | 48 | *The act of writing shiny future stories can uncover things we didn't expect to find. Did you have any new and exciting ideas as you were writing? Realize some complications that you didn't foresee?* 49 | 50 | ### What are some variations of this story that you considered, or that you think might be fun to write? Have any variations of this story already been written? 51 | 52 | *Often when writing stories, we think about various possibilities. Sketch out some of the turning points here -- maybe someone will want to turn them into a full story! Alternatively, if this is a variation on an existing story, link back to it here.* 53 | 54 | ### What are some of the things we'll have to figure out to realize this future? What projects besides Rust itself are involved, if any? (Optional) 55 | 56 | *Often the 'shiny future' stories involve technical problems that we don't really know how to solve yet. If you see such problems, list them here!* 57 | 58 | 59 | 60 | [character]: ../../characters.md 61 | [comment]: ../../how_to_vision/comment.md 62 | [status quo stories]: ../status_quo.md 63 | [Alan]: ../../characters/alan.md 64 | [Grace]: ../../characters/grace.md 65 | [Niklaus]: ../../characters/niklaus.md 66 | [Barbara]: ../../characters/barbara.md 67 | [projects]: ../../projects.md 68 | [htvsq]: ../shiny_future.md 69 | [cannot be wrong]: ../../how_to_vision/comment.md#comment-to-understand-or-improve-not-to-negate-or-dissuade 70 | -------------------------------------------------------------------------------- /src/vision/submitted_stories/status_quo/alan_builds_a_cache.md: -------------------------------------------------------------------------------- 1 | # 😱 Status quo stories: Alan tries to cache requests, which doesn't always happen. 2 | 3 | ## 🚧 Warning: Draft status 🚧 4 | 5 | This is a draft "status quo" story submitted as part of the brainstorming period. It is derived from real-life experiences of actual Rust users and is meant to reflect some of the challenges that Async Rust programmers face today. 6 | 7 | If you would like to expand on this story, or adjust the answers to the FAQ, feel free to open a PR making edits (but keep in mind that, as they reflect peoples' experiences, status quo stories [cannot be wrong], only inaccurate). Alternatively, you may wish to [add your own status quo story][htvsq]! 8 | 9 | 10 | ## The story 11 | 12 | [Alan][] is working on an HTTP server. The server makes calls to some other service. The performance of the downstream service is somewhat poor, so Alan would like to implement some basic caching. 13 | 14 | [Alan]: ../../characters/alan.md 15 | 16 | Alan writes up some code which does the caching: 17 | 18 | ```rust 19 | async fn get_response(&mut self, key: String) { 20 | // Try to get the response from cache 21 | if let Some(cached_response) = self.cache.get(key) { 22 | self.channel.send(cached_response).await; 23 | return; 24 | } 25 | 26 | // Get the response from the downstream service 27 | let response = self.http_client.make_request(key).await; 28 | self.channel.send(response).await; 29 | 30 | // Store the response in the cache 31 | self.cache.set(key, response); 32 | } 33 | ``` 34 | 35 | Alan is happy with how things are working, but notices every once in a while the downstream service hangs. To prevent that, Alan implements a timeout. 36 | 37 | He remembers from the documentation for his favorite runtime that there is the `race` function which can kick off two futures and polls both until one completes (similar to tokio's [select](https://docs.rs/tokio/1.5.0/tokio/macro.select.html) and async-std's [race](https://docs.rs/async-std/1.9.0/async_std/future/trait.Future.html#method.race) for example). 38 | 39 | 40 | ```rust 41 | runtime::race(timeout(), get_response(key)).await 42 | ``` 43 | 44 | ## The bug 45 | 46 | Alan ships to production but after several weeks he notices some users complaining that they receive old data. 47 | 48 | Alan looks for help. The compiler unfortunately doesn't provide any hints. He turns to his second best friend clippy, who cannot help either. 49 | Alan tries debugging. He uses his old friend `println!`. After hours of working through, he notices that sometimes the line that sets the response in the cache never gets called. 50 | 51 | ## The solution 52 | 53 | Alan goes to [Barbara][] and asks why in the world that might be ⁉️ 54 | 55 | 💡 Barbara looks through the code and notices that there is an await point between sending the response over the channel and setting the cache. 56 | 57 | Since the `get_response` future can be dropped at each available await point, it may be dropped *after* the http request has been made, but *before* the response has successfully been sent over the channel, thus not executing the remaining instructions in the function. 58 | 59 | This means the cache might not be set. 60 | 61 | Alan fixes it by setting the cache before sending the result over the channel. 🎉 62 | 63 | ```rust 64 | async fn get_response(&mut self, key: String) { 65 | // ... cache miss happened here 66 | 67 | // We perform the HTTP request and our code might continue 68 | // after this .await once the HTTP request is complete 69 | let response = self.http_client.make_request(key).await; 70 | 71 | // Immediately store the response in the cache 72 | self.cache.set(key, response); 73 | 74 | self.channel.send(response).await; 75 | } 76 | ``` 77 | 78 | ## 🤔 Frequently Asked Questions 79 | 80 | ### **What are the morals of the story?** 81 | 82 | * Futures can be "canceled" at any await point. Authors of futures must be aware that after an await, the code might not run. 83 | * This is similar to `panic` safety but way more likely to happen 84 | * Futures might be polled to completion causing the code to work. But then many years later, the code is changed and the future might conditionally not be polled to completion which breaks things. 85 | * The burden falls on the user of the future to poll to completion, and there is no way for the lib author to enforce this - they can only document this invariant. 86 | * Diagnosing and ultimately fixing this issue requires a fairly deep understanding of the semantics of futures. 87 | * Without a Barbara, it might be hard to even know where to start: No lints are available, Alan is left with a normal debugger and `println!`. 88 | 89 | ### **What are the sources for this story?** 90 | The relevant sources of discussion for this story have been gathered [in this github issue](https://github.com/rust-lang/wg-async/issues/65). 91 | 92 | ### **Why did you choose Alan to tell this story?** 93 | Alan has enough experience and understanding of push based async languages to make the assumptions that will trigger the bug. 94 | 95 | ### **How would this story have played out differently for the other characters?** 96 | This story would likely have played out the same for almost everyone but Barbara, who has probably been bitten by that already. 97 | The debugging and fixing time would however probably have varied depending on experience and luck. 98 | -------------------------------------------------------------------------------- /src/vision/submitted_stories/status_quo/alan_wants_prefetch_iterator.md: -------------------------------------------------------------------------------- 1 | # 😱 Status quo stories: Alan wants an async iterator with prefetch 2 | 3 | ## 🚧 Warning: Draft status 🚧 4 | 5 | This is a draft "status quo" story submitted as part of the brainstorming period. It is derived from real-life experiences of actual Rust users and is meant to reflect some of the challenges that Async Rust programmers face today. 6 | 7 | If you would like to expand on this story, or adjust the answers to the FAQ, feel free to open a PR making edits (but keep in mind that, as they reflect peoples' experiences, status quo stories [cannot be wrong], only inaccurate). Alternatively, you may wish to [add your own status quo story][htvsq]! 8 | 9 | ## The story 10 | 11 | Alan once wrote a data processing microservice in a GC'd language which was designed for high throughput. Now he wants to write it in Rust and have strong ownership model. 12 | 13 | The original service consumes messages from a source stream (e.g. Kafka), process them and produces results to another stream and/or saves them to a database. Since the service acquires some data from other sources like external services and its own PostgreSQL database, Alan batches incoming messages to acquire as much as possible data from that sources with minimal overhead. 14 | 15 | Since messages might arrive with some delays between them, or can end at some point for a while, their number is unknown, there's an async iterator which reads the input stream and waits some time before producing a batch if the next message isn't immediately ready. 16 | 17 | Alan explored `FutureExt` from `async-std` and found no evidence that it's possible to wait for multiple futures returning different results (it's not possible for `ValueTask`s in .NET, but it worked well with `Task`s which can be awaited multiple times). Later he was suggested to use an `enum` and the `race` method to achive his goal: 18 | 19 | ```rust 20 | enum Choices { 21 | A(A), 22 | B(B), 23 | C(C), 24 | } 25 | 26 | // convert each future into the type `Choices<...>`: 27 | let future_a = async move { A(future_a.await) }; 28 | let future_b = async move { B(future_b.await) }; 29 | let future_c = async move { C(future_c.await) }; 30 | 31 | // await the race: 32 | match future_a.race(future_b).race(future_c).await { 33 | A(a) => ..., 34 | B(b) => ...., 35 | C(c) => ..., 36 | } 37 | ``` 38 | 39 | While that helped Alan, it was completely unobvious to him. He expected to see a macro accepting futures and producing a new future to be awaited: 40 | 41 | ```rust 42 | match race!(future_a, future_b, future_c).await { 43 | // ... 44 | } 45 | ``` 46 | 47 | Having `join!` would be nice too for Alan, so he can avoid binding variables to futures which later shall be awaited: 48 | 49 | ```rust 50 | // How it's now 51 | let future_a = do_async_a(); 52 | let future_b = do_async_b(); 53 | let future_c = do_async_c(); 54 | 55 | let result_a = future_a.await; 56 | let result_b = future_b.await; 57 | let result_c = future_c.await; 58 | 59 | // How it could be 60 | let (result_a, result_b, result_c) = join!(future_a, future_b, future_c).await; 61 | ``` 62 | 63 | ## 🤔 Frequently Asked Questions 64 | 65 | *Here are some standard FAQ to get you started. Feel free to add more!* 66 | 67 | ### **What are the morals of the story?** 68 | * Even though Alan had experience writing async code in other languages, he had a hard time figuring out how to do relatively simple things in Rust, like joining or racing on futures of different types. 69 | 70 | ### **What are the sources for this story?** 71 | Personal experience of the author. 72 | 73 | ### **Why did you choose Alan to tell this story?** 74 | As a backend developer in a GC'd language, Alan writes async code every day. He wants to gain the maximum performance and have memory safety at the same time. 75 | 76 | ### **How would this story have played out differently for the other characters?** 77 | *In some cases, there are problems that only occur for people from specific backgrounds, or which play out differently. This question can be used to highlight that.* 78 | 79 | [character]: ../../characters.md 80 | [status quo stories]: ../status_quo.md 81 | [Alan]: ../../characters/alan.md 82 | [Grace]: ../../characters/grace.md 83 | [Niklaus]: ../../characters/niklaus.md 84 | [Barbara]: ../../characters/barbara.md 85 | [htvsq]: ../status_quo.md 86 | [cannot be wrong]: ../../how_to_vision/comment.md#comment-to-understand-or-improve-not-to-negate-or-dissuade 87 | -------------------------------------------------------------------------------- /src/vision/submitted_stories/status_quo/aws_engineer.md: -------------------------------------------------------------------------------- 1 | # 😱 Status quo stories: Status quo of an AWS engineer 2 | 3 | ## 🚧 Warning: Draft status 🚧 4 | 5 | This is a draft "status quo" story submitted as part of the brainstorming period. It is derived from real-life experiences of actual Rust users and is meant to reflect some of the challenges that Async Rust programmers face today. 6 | 7 | If you would like to expand on this story, or adjust the answers to the FAQ, feel free to open a PR making edits (but keep in mind that, as they reflect peoples' experiences, status quo stories [cannot be wrong], only inaccurate). Alternatively, you may wish to [add your own status quo story][htvsq]! 8 | 9 | ## The story 10 | 11 | This tells the story of Alan, an engineer who works at AWS. 12 | 13 | * [Writing a Java-based service at AWS](aws_engineer/writing_a_java_based_service.md): Alan is accustomed to using many convenient tools for writing Java-based services. 14 | * [Getting started with Rust](aws_engineer/getting_started_with_rust.md): Alan gets tapped to help spin up a new project on a tight timeline. He hasn't used Rust before, so he starts trying to setup an environment and learn the basics. 15 | * [Coming from Java](aws_engineer/coming_from_java.md): Alan finds that some of the patterns he's accustomed to from Java don't translate well to Rust. 16 | * [Exploring the ecosystem](aws_engineer/ecosystem.md): The Rust ecosystem has a lot of useful crates, but they're hard to find. "I don't so much find them as stumble upon them by accident." 17 | * At first, Rust feels quite ergonomic to Alan. The async-await system seems pretty slick. But as he gets more comfortable with Rust, he starts to encounter situations where he can't quite figure out how to get things setup the way he wants, and he has to settle for suboptimal setups: 18 | * [Juggling error handling](aws_engineer/juggling_error_handling.md): Alan tries to use `?` to process errors in a stream. 19 | * [Failure to parallelize](aws_engineer/failure_to_parallelize.md): Alan can't figure out how to parallelize a loop. 20 | * [Borrow check errors](aws_engineer/borrow_check_errors.md): Alan tries to write code that fills a buffer and returns references into it to the caller, only to learn that Rust's borrow checker makes that pattern difficult. 21 | * As Alan goes deeper into Async Rust, he learns that its underlying model can be surprising. One particular [deadlock](aws_engineer/solving_a_deadlock.md) takes him quite a long time to figure out. 22 | * [Encountering pin](aws_engineer/encountering_pin.md): Wrapping streams, `AsyncRead` implementations, and other types requires using `Pin` and it is challenging. 23 | * [Figuring out the best option](aws_engineer/figuring_out_the_best_option.md): Alan often encounters cases where he doesn't know what is the best way to implement something. He finds he has to implement it both ways to tell, and sometimes even then he can't be sure. 24 | * [Testing his service](aws_engineer/testing_the_service.md): Alan invents patterns for Dependency Injection in order to write tests. 25 | * [Using the debugger](aws_engineer/using_the_debugger.md): Alan wishes for a smoother debugging experience. 26 | * [Missed Waker leads to lost performance](aws_engineer/missed_waker_leads_to_lost_performance.md): Alan finds his service his not as fast as the reference server; the problem is ultimately due to a missed `Waker`, which was causing his streams to wake up much later than it should've. 27 | * [Debugging performance problems](aws_engineer/debugging_performance_problems.md): Alan finds more performance problems and tries to figure out their cause using tooling like `perf`. It's hard. 28 | * [Getting ready to deploy](aws_engineer/getting_ready_to_deploy.md): Alan prepares to deply the service. 29 | * [Using JNI](aws_engineer/using_jni.md): Alan uses JNI to access services that are only available using Java libraries. 30 | 31 | ## 🤔 Frequently Asked Questions 32 | 33 | ### **What are the morals of the story?** 34 | 35 | * Building services in Rust can yield really strong results, but a lot of hurdles remain: 36 | * 'If it compiles, it works' is not true: there are lots of subtle variations. 37 | * Debugging correctness and performance problems is hard, and the tooling is not what folks are used to. 38 | * Few established patterns to things like DI. 39 | * The ecosystem has a lot of interesting things in it, but it's hard to navigate. 40 | 41 | ### **What are the sources for this story?** 42 | 43 | This story is compiled from discussions with service engineers in various AWS teams. 44 | 45 | ### **Why did you choose Alan to tell this story?** 46 | 47 | Because Java is a very widely used language at AWS. 48 | 49 | ### **How would this story have played out differently for the other characters?** 50 | 51 | Most parts of it remain the same; the main things that were specific to Java are some of the patterns Alan expected to use. Similarly, few things are specific to AWS apart from some details of the setup. 52 | 53 | [character]: ../../characters.md 54 | [status quo stories]: ../status_quo.md 55 | [Alan]: ../../characters/alan.md 56 | [Grace]: ../../characters/grace.md 57 | [Niklaus]: ../../characters/niklaus.md 58 | [Barbara]: ../../characters/barbara.md 59 | [htvsq]: ../status_quo.md 60 | [cannot be wrong]: ../../how_to_vision/comment.md#comment-to-understand-or-improve-not-to-negate-or-dissuade 61 | -------------------------------------------------------------------------------- /src/vision/submitted_stories/status_quo/aws_engineer/borrow_check_errors.md: -------------------------------------------------------------------------------- 1 | # Status quo of an AWS engineer: Borrow check errors 2 | 3 | Alan has more or less gotten the hang of the borrow checker, but sometimes it still surprises him. One day, he is working on a piece of code in DistriData. There are a set of connections: 4 | 5 | ```rust= 6 | struct Connection { 7 | buffer: Vec, 8 | } 9 | ``` 10 | 11 | and each `Connection` has the ability to iterate through various requests. These requests return subslices of the data in the connection: 12 | 13 | ```rust= 14 | struct Request<'a> { 15 | headers: Vec<&'a u8>, 16 | } 17 | ``` 18 | 19 | He writes a routine to get the next request from the connection. It begins by reading data into the internal buffer and then parsing from that buffer and returning the request ([playground](https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=6d8f2e7349e25677b25c527964842de8)): 20 | 21 | ```rust= 22 | impl Connection { 23 | pub async fn read_next(&mut self) -> Request<'_> { 24 | loop { 25 | self.read_into_buffer(); 26 | 27 | // can't borrow self.buffer, even though we only hang on to it in the 28 | // return branch 29 | match Request::try_parse(&self.buffer) { 30 | Some(r) => return r, 31 | None => continue, 32 | } 33 | } 34 | } 35 | 36 | async fn read_into_buffer(&mut self) { 37 | self.buffer.push(1u8); 38 | } 39 | } 40 | ``` 41 | 42 | This code, however, doesn't build. He gets the following error: 43 | 44 | ``` 45 | error[E0502]: cannot borrow `*self` as mutable because it is also borrowed as immutable 46 | --> src/lib.rs:15:12 47 | | 48 | 13 | pub async fn read_next(&mut self) -> Request<'_> { 49 | | - let's call the lifetime of this reference `'1` 50 | 14 | loop { 51 | 15 | self.read_into_buffer().await; 52 | | ^^^^^^^^^^^^^^^^^^^^^^^ mutable borrow occurs here 53 | ... 54 | 19 | match Request::try_parse(&self.buffer) { 55 | | ------------ immutable borrow occurs here 56 | 20 | Some(r) => return r, 57 | | - returning this value requires that `self.buffer` is borrowed for `'1` 58 | ``` 59 | 60 | This is confusing. He can see that there is a mutable borrow occurring, and an immutable one, but it seems like they occur at disjoint periods of time. Why is the compiler complaining? 61 | 62 | After asking on `#rust` in the AWS Slack, he learns that this is a pattern that Rust's borrow checker just can't support. It gets confused when you return data from functions and winds up producing errors that aren't necessary. Apparently there's some [research project named after a Hamlet play](https://github.com/rust-lang/polonius/) that might help, but that isn't going to help him now. The slack channel points him at the [ouroboros](https://github.com/joshua-maros/ouroboros) project and he eventually uses it to work around the problem ([playground](https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=59b2cb72529e58c13ab00eee1e9c0435)). 63 | -------------------------------------------------------------------------------- /src/vision/submitted_stories/status_quo/aws_engineer/coming_from_java.md: -------------------------------------------------------------------------------- 1 | # Status quo of an AWS engineer: Coming from Java 2 | 3 | At first, Alan is trying to write Rust code as if it were Java. He's accustomed to avoiding direct dependencies between types and instead modeling everything with an `interface`, so at first he creates a lot of Rust traits. He quickly learns that `dyn Trait` can be kind of painful to use. 4 | 5 | He also learns that Rust doesn't really permit you to add references willy nilly. It was pretty common in Java to have a class that was threaded everywhere with all kinds of references to various parts of the system. This pattern often leads to borrow check errors in Rust. 6 | 7 | He gets surprised by parallelism. He wants a concurrent hash map but can't find one in the standard library. There are a lot of crates on crates.io but it's not clear which would be best. He decides to use a mutex-protected lock. 8 | 9 | He is surprised because futures in Java correspond to things executed in parallel, but in Rust they don't. It takes him some time to get used to this. Eventually he learns that a Rust future is more akin to a java *callable*. -------------------------------------------------------------------------------- /src/vision/submitted_stories/status_quo/aws_engineer/debugging_performance_problems.md: -------------------------------------------------------------------------------- 1 | # Status quo of an AWS engineer: Debugging overall performance loss 2 | 3 | Alan's service is working better and better, but performance is still lagging from where he hoped it would be. It seems to be about 20% slower than the Java version! After [calling in Barbara to help him diagnose the problem](https://rust-lang.github.io/wg-async/vision/status_quo/alan_iteratively_regresses.html), Alan identifies one culprit: Some of the types in Alan's system are really large! The system seems to spend a surprising amount of time just copying bytes. Barbara helped Alan diagnose this by showing him some hidden rustc flags, tinkering with his perf setup, and a few other tricks. 4 | 5 | There is still a performance gap, though, and Alan's not sure where it could be coming from. There are a few candidates: 6 | 7 | * Perhaps they are not using tokio's scheduler optimally. 8 | * Perhaps the memory allocation costs introduced by the `#[async_trait]` are starting to add up. 9 | 10 | Alan tinkers with jemalloc and finds that it does improve performance, so that's interesting, but he'd like to have a better understanding of *why*. 11 | -------------------------------------------------------------------------------- /src/vision/submitted_stories/status_quo/aws_engineer/ecosystem.md: -------------------------------------------------------------------------------- 1 | # Status quo of an AWS engineer: Exploring the ecosystem 2 | 3 | Alan finds that cargo is a super powerful tool, but he finds it very hard to find crates to use. He doesn't really feel he discovers crates so much as "falls upon" them by chance. For example, he happened to see a stray mention of `cargo bloat` in the internals form, and that turned out to be exactly what he needed. He finds the `async-trait` crate in a similar way. He's happy these tools exist, but he wishes he had more assurance of finding them; he wonders what useful things are out there that he *doesn't* know about. 4 | 5 | In some cases, there are a lot of choices and it's really hard to tell which is best. Alan spent some time evaluating crates that do md5 hashing, for example, and found tons of choices. He does some quick performance testing and finds huge differences: openssl seems to be the fastest, so he takes that, but he is worried he may have missed some crates. 6 | 7 | He had decided to use tokio because it was the thing that everyone else is using. But he gradually learns that there are more runtimes out there. Sometimes, when he adds a crate, he finds that it is bringing in a new set of dependencies that don't seem necessary. 8 | 9 | He also gets confused by the vast array of options. `tokio` seems to have an `AsyncRead` trait, for example, but so does `futures` -- which one should he use? 10 | 11 | He's heard of other runtimes and he might like to be able to try them out, but it would be too much work. Occasionally he winds up with multiple versions of the same crate, which can lead either to compilation or runtime errors. For example, when rusoto upgraded to a new version of tokio, this spilled over into confusing huge error messages from the rusoto builder owing to subtle trait and type mismatches. Fortunately the recent tokio 1.0 release promises to solve some of those problems. 12 | -------------------------------------------------------------------------------- /src/vision/submitted_stories/status_quo/aws_engineer/encountering_pin.md: -------------------------------------------------------------------------------- 1 | # Status quo of an AWS engineer: Encountering pin 2 | 3 | As Alan is building the server, he encounters a case where he wants to extend a stream of data to track some additional metrics. The stream implements [`AsyncRead`]. He thinks, "Ah, I'll just make a wrapper type that can extend any `AsyncRead`." He opens up the rustdoc, though, and realizes that this may be a bit tricky. "What is this `self: Pin<&mut Self>`?" notation, he thinks. He had vaguely heard of `Pin` when skimming the docs for futures and things but it was never something he had to work with directly before. 4 | 5 | [`AsyncRead`]: https://docs.rs/tokio/1.5.0/tokio/io/trait.AsyncRead.html 6 | 7 | Alan's experiences here are well documented in [Alan hates writing a Stream](https://rust-lang.github.io/wg-async/vision/status_quo/alan_hates_writing_a_stream.html). Suffice to say that, at long last, he does it to work, but he does not feel he really understands what is going on. Talking with his coworkers on slack he notes, "Mostly I just add `Pin` and whatever else the compiler asks for until it works; then I pray it doesn’t crash." :crossed_fingers: 8 | 9 | *References:* 10 | 11 | * [Alan hates writing a Stream](../alan_hates_writing_a_stream.html) 12 | * ["Pin and suffering", by faster-than-lime](https://fasterthanli.me/articles/pin-and-suffering) 13 | -------------------------------------------------------------------------------- /src/vision/submitted_stories/status_quo/aws_engineer/figuring_out_the_best_option.md: -------------------------------------------------------------------------------- 1 | # Status quo of an AWS engineer: Figuring out the best option 2 | 3 | Sometime after working on `AsyncRead`, Alan stumbles over the `async-trait` crate. This crate offers a macro that will let him add `async fn` to traits. He's excited about this because it seems like it would allow him to rewrite some of the custom `AsyncRead` impls in a cleaner way. The only problem is that he can't really judge what the implications are going to be -- will it be faster? Slower? It's hard to tell until it's done. He feels like this comes up a lot in Rust: he is forced to make a choice and see it all the way through to the end before he can decide whether he likes it (or if it will work at all: sometimes he encounters a compiler error part of the way through that he just can't figure out how to resolve). It's particularly frustrating in Async Rust where [there seem to be so many options to choose from](https://rust-lang.github.io/wg-async/vision/status_quo/barbara_needs_async_helpers.html). 4 | -------------------------------------------------------------------------------- /src/vision/submitted_stories/status_quo/aws_engineer/getting_ready_to_deploy.md: -------------------------------------------------------------------------------- 1 | # Status quo of an AWS engineer: Getting ready to deploy the service 2 | 3 | The next morning, Alan is talking to his team. The service is more-or-less working, although there is room to improve performance. It's time to talk about the Operational Readiness Review (ORR). Before any service can be put into production at AWS, it needs to pass an ORR. This is a stringent process where experienced senior engineers grill the team about all kinds of things that could go wrong and how they would handle them. These plans are gathered into a document that can be consulted should the need arise. 4 | 5 | Alan has been through a few ORRs in his time at AWS. They're always stressful, but they're usually not that big a deal. A lot of the worst cases are handled automatically by the Java frameworks that Alan is accustomed to working with: for example, they have connection timeouts, or facilities for logging particular kinds of events. For the stuff that is not automatic, there are known "best practices" that can help. 6 | 7 | For Rust, there are a lot of unknowns. The standard servers don't exist, and Alan's team has had to roll their own. There aren't nearly as many tools for performance monitoring or other sorts of improvements. Alan's team is treading new ground by deploying a Rust-based service, and they know they have to budget extra time to manage that. -------------------------------------------------------------------------------- /src/vision/submitted_stories/status_quo/aws_engineer/getting_started_with_rust.md: -------------------------------------------------------------------------------- 1 | # Status quo of an AWS engineer: Getting started with Rust 2 | 3 | For his latest project, Alan is rewriting a core component of [DistriData]. They are trying to move on a tight deadline. 4 | 5 | The component that they are rewriting was implemented in Java, but it was having difficulty with high tail latencies and other performance hiccups. The team has an idea for a new architecture that will be more efficient, and they would like to reduce resource usage by adopting Rust. 6 | 7 | Getting started with Rust is a bit different than what he is used to. There's not much infrastructure. They still define their service interface using the same modeling language, but there is no tooling to generate a server from it. 8 | 9 | [DistriData]: ../../projects/DistriData.html 10 | 11 | ## IDE setup 12 | 13 | Of course, the very first thing Alan does it to tweak his IDE setup. He's happy to learn that IntelliJ has support for Rust, since he is accustomed to the keybindings and it has great integration with Brazil, AWS's internal build system. 14 | 15 | Still, as he plays around with Rust code, he realizes that the support is not nearly at the level of Java. Autocomplete often gets confused. For example, when there are two traits with the same name but coming from different crates, IntelliJ often picks the wrong one. It also has trouble with macros, which are very common in async code. Some of Alan's colleagues switch to VSCode, which is sometimes better but has many of the same problems; Alan decides to stick with IntelliJ. 16 | 17 | ## Building the first server 18 | 19 | Alan asks around the company to learn more about how Async Rust works and he is told to start with the tokio tutorial and the Rust book. He also joins the company slack channel, where he can ask questions. The tokio tutorial is helpful and he is feeling relatively confident. 20 | 21 | ## Missing types during Code review 22 | 23 | One problem Alan finds has to do with AWS's internal tooling (although it would be the same in most places). When browsing Rust code in the IDE, there are lots of tips to help in understanding, such as tooltips showing the types of variables and the like. In code reviews, though, there is only the plain text. Rust's type inference is super useful and make the code compact, but it can be hard to tell what's going on when you just read the plain source. 24 | -------------------------------------------------------------------------------- /src/vision/submitted_stories/status_quo/aws_engineer/missed_waker_leads_to_lost_performance.md: -------------------------------------------------------------------------------- 1 | # Status quo of an AWS engineer: Missed `Waker` leads to lost performance 2 | 3 | Once the server is working, Alan starts to benchmark it. He is not really sure what to expect, but he is hoping to see an improvement in performance relative to the baseline service they were using before. To his surprise, it seems to be running slower! 4 | 5 | After trying a few common tricks to improve performance without avail, Alan wishes -- not for the first time -- that he had better tools to understand what was happening. He decides instead to add more metrics and logs in his service, to understand where the bottlenecks are. Alan is used to using a well-supported internal tool (or a mature open source project) to collect metrics, where all he needed to do was pull in the library and set up a few configuration parameters. 6 | 7 | However, in Rust, there is no widely-used, battle-tested library inside and outside his company. Even less so in an async code base! So Alan just used what seemed to be the best options: tracing and metrics crate, but he quickly found that they couldn't do a few of the things he wants to do, and somehow invoking the metrics is causing his service to be even slower. Now, Alan has to debug and profile his metrics implementation before he can even fix his service. (Cue another story on how that's difficult...) 8 | 9 | After a few days of poking at the problem, Alan notices something odd. It seems like there is often a fairly large delay between the completion of a particular event and the execution of the code that is meant to respond to that event. Looking more closely, he realizes that the code for handling that event fails to trigger the `Waker` associated with the future, and hence the future never wakes up. 10 | 11 | As it happens, this problem was hidden from him because that particular future was combined with a number of others. Eventually, the other futures get signalled, and hence the event does get handled -- but less promptly than it should be. He fixes the problem and performance is restored. 12 | 13 | "I'm glad I had a baseline to compare this against!", he thinks. "I doubt I would have noticed this problem otherwise." 14 | 15 | -------------------------------------------------------------------------------- /src/vision/submitted_stories/status_quo/aws_engineer/solving_a_deadlock.md: -------------------------------------------------------------------------------- 1 | # Status quo of an AWS engineer: Solving a deadlock 2 | 3 | Alan logs into work the next morning to see a message in Slack: 4 | 5 | > Alan, I've noticed that the code to replicate the shards across the hosts is sometimes leading to a deadlock. Any idea what's going on? 6 | 7 | This is the same code that Alan tried to parallelize earlier. He pulls up the function, but everything *seems* correct! It's not obvious what the problem could be. 8 | 9 | ```rust= 10 | // Prepare the outgoing HTTP requests to each host: 11 | let mut host_senders: Vec = vec![]; 12 | let mut host_futures = FuturesUnordered::new(); 13 | for host in hosts { 14 | let (sender, body) = hyper::body::Body::channel(); 15 | host_senders.push(sender); 16 | host_futures.push(create_future_to_send_request(body)); 17 | } 18 | 19 | // Send each chunk from each shared to each host: 20 | while let Some(chunks) = shards.next().await { 21 | let chunk_futures = chunks 22 | .into_iter() 23 | .zip(&mut host_senders) 24 | .map(|(chunk, sender)| sender.send_data(chunk)); 25 | 26 | join_all(chunk_futures) 27 | .await 28 | .into_iter() 29 | .collect::, _>>()?; 30 | } 31 | 32 | // Wait for all HTTP requests to complete, aborting on error: 33 | loop { 34 | match host_futures.next().await { 35 | Some(Ok(response)) => handle_response(response)?, 36 | Some(Err(e)) => return Err(e).map_err(box_err)?, 37 | None => return Ok(()), 38 | } 39 | } 40 | ``` 41 | 42 | He tries to reproduce the deadlock. He is able to reproduce the problem readily enough, but only with larger requests. He had always used small tests before. He connects to the process with the debugger but he can't really make heads or tails of what tasks seem to be stuck (see [Alan tries to debug a hang](https://rust-lang.github.io/wg-async/vision/status_quo/alan_tries_to_debug_a_hang.html) or [Barbara wants async insights](https://rust-lang.github.io/wg-async/vision/status_quo/barbara_wants_async_insights.html)). He resorts to sprinkling logging everywhere. 43 | 44 | At long last, he starts to see a pattern emerging. From the logs, he sees the data from each chunk is being sent to the hyper channel, but it never seems to be sent over the HTTP connection to the backend hosts. He is pretty confused by this -- he thought that the futures he pushed into `host_futures` should be taking care of sending the request body out over the internet. He goes to talk to Barbara -- [who, as it happens, has been through this very problem in the past](https://rust-lang.github.io/wg-async/vision/status_quo/barbara_battles_buffered_streams.html) -- and she explains to him what is wrong. 45 | 46 | "When you push those futures into `FuturesUnordered`", she says, "they will only make progress when you are actually awaiting on the stream. With the way the loop is setup now, the actual sending of data won't start until that third loop. Presumably your deadlock is because the second loop is blocked, waiting for some of the data to be sent." 47 | 48 | "Huh. That's...weird. How can I fix it?", asks Alan. 49 | 50 | "You need to spawn a separate task," says Barbara. "Something like this should work." She modifies the code to spawn a task that is performing the third loop. That task is spawned before the second loop starts: 51 | 52 | ```rust= 53 | // Prepare the outgoing HTTP requests to each host: 54 | let mut host_senders: Vec = vec![]; 55 | let mut host_futures = FuturesUnordered::new(); 56 | for host in hosts { 57 | let (sender, body) = hyper::body::Body::channel(); 58 | host_senders.push(sender); 59 | host_futures.push(create_future_to_send_request(body)); 60 | } 61 | 62 | // Make sure this runs in parallel with the loop below! 63 | let send_future = tokio::spawn(async move { 64 | // Wait for all HTTP requests to complete, aborting on error: 65 | loop { 66 | match host_futures.next().await { 67 | Some(Ok(response)) => handle_response(response)?, 68 | Some(Err(e)) => break Err(e)?, 69 | None => break Ok(()), 70 | } 71 | } 72 | }); 73 | 74 | // Send each chunk from each shared to each host: 75 | while let Some(chunks) = shards.next().await { 76 | let chunk_futures = chunks 77 | .into_iter() 78 | .zip(&mut host_senders) 79 | .map(|(chunk, sender)| sender.send_data(chunk)); 80 | 81 | join_all(chunk_futures) 82 | .await 83 | .into_iter() 84 | .collect::, _>>()?; 85 | } 86 | 87 | send_future.await 88 | ``` 89 | 90 | "Oof", says Alan, "I'll try to remember that!" 91 | -------------------------------------------------------------------------------- /src/vision/submitted_stories/status_quo/aws_engineer/testing_the_service.md: -------------------------------------------------------------------------------- 1 | # Status quo of an AWS engineer: Testing the service 2 | 3 | At first, Alan is content to test by hand. But once the server is starting to really work, he realizes he needs to do unit testing. He wants to do something like [Mockito](https://site.mockito.org/) in Rust, so he starts searching the internet to find out what the options are. To his surprise, he learns that there doesn't seem to be any comparable framework in Rust. 4 | 5 | One option he considers is making all of his functions generic. For example, he could create a trait to model, for example, the network, so that he can insert artificial pauses and other problems during testing: 6 | 7 | ```rust 8 | trait Network { 9 | ... 10 | } 11 | ``` 12 | 13 | Writing such a trait is fairly complicated, but even if he wrote it, he would have to make all of his structs and functions generic: 14 | 15 | ```rust 16 | struct MyService { 17 | ... 18 | } 19 | ``` 20 | 21 | Alan starts threading these parameters through the code and quickly gets overwhelmed. 22 | 23 | He decides instead to test his real code without any mocking. He and his team start building a load-testing framework, they call it "simworld". They need to be able to inject network errors, control timing, and force other unusual situations. 24 | 25 | Building simworld takes a lot of time, but it is very useful, and they start to gain some confidence in their code. 26 | -------------------------------------------------------------------------------- /src/vision/submitted_stories/status_quo/aws_engineer/using_jni.md: -------------------------------------------------------------------------------- 1 | # Status quo of an AWS engineer: Using JNI 2 | 3 | One other problem that Alan's team has encountered is that some of the standard libraries they would use at AWS are only available in Java. After some tinkering, Alan's team decides to stand-up a java server as part of their project. The idea is that the server can accept the connections and then use JNI to invoke the Rust code; having the Rust code in process means it can communicate directly with the underlying file descriptor and avoid copies. 4 | 5 | They stand up the Java side fairly quickly and then spend a bit of time experimenting with different ways to handle the "handoff" to Rust code. The first problem is keeping the tokio runtime alive. Their first attempt to connect using JNI was causing the runtime to get torn down. But they figure out that they can store the Runtime in a static variable. 6 | 7 | Next, they find having Rust code access Java objects is quite expensive; it's cheaper to pass bytebuffers at the boundary using protobuf. They try a few options for serialization and deserialization to find which works best. 8 | 9 | Overall, the integration with the JNI works fairly smoothly for them, but they wish there had been some documented pattern to have just shown them the best way to set things up, rather than them having to discover it. -------------------------------------------------------------------------------- /src/vision/submitted_stories/status_quo/aws_engineer/using_the_debugger.md: -------------------------------------------------------------------------------- 1 | # Status quo of an AWS engineer: Using the debugger 2 | 3 | Even though the code is starting to work, they soon uncover a test that is not behaving as it ought to. Alan decides to try loading the Rust code into the debugger. He quickly realizes that the debugger is showing him the raw threads that are used to implement his service, and not the tasks and things that the service uses at a logical level, but that's not a problem for what he's doing right now. He sets a breakpoint on a particular line of code that corresponds to the place where things seem to be going wrong. 4 | 5 | At first, the debugger seems to be working great, but Alan soon realizes that the experiences is a far cry from what he is used to with IntelliJ and Java code. Stepping through the code is unpredictable; it's not always obvious what function the will be stepping into. More than once Alan is confronted with a screen full of assembly. "No thank you," he thinks, and just avoids stepping into that function in the future. He finds that he often cannot print the values of variables ('variable optimized out', says the debugger) or execute code dynamically. Sometimes he is able to print them but instead of seeing something useful, he gets a bunch of random pointer values. 6 | 7 | Alan gives up on the debugger. He starts to thread printfs and logging statements throughout his code. The [`tracing`] crate is pretty useful. Eventually, he is able to find and fix the problem and get his test case passing. -------------------------------------------------------------------------------- /src/vision/submitted_stories/status_quo/aws_engineer/writing_a_java_based_service.md: -------------------------------------------------------------------------------- 1 | # Status quo of an AWS engineer: Writing a Java-based service 2 | 3 | Alan has been working at AWS for the last six years. He's accustomed to a fairly standard workflow for launching Java-based services: 4 | 5 | * Write a description of the service APIs using a modeling language like [Smithy](https://awslabs.github.io/smithy/). 6 | * Submit the description to a webpage, which gives a standard service implementation based on [netty](https://netty.io/). Each of the API calls in the modeling language has a function with a `/* TODO */` comment to fill in. 7 | * As Alan works with his team to fill in each of those items, he makes use of a number of standard conventions: 8 | * Mocking with projects like [mockito](https://site.mockito.org/) to allow for unit testing of specific components. 9 | * Alan uses a variety of nice tools: 10 | * Advanced IDEs like IntelliJ, which offer him suggestions as he works. 11 | * Full-featured, if standard, debuggers; he can run arbitrary code, mutate state, step into and out of functions with ease. 12 | * Tools for introspecting the VM state to get heap usage information and other profiling data. 13 | * Performance monitoring frameworks 14 | * As Alan is preparing to launch his service, he has to conduct an Operational Readiness Review (ORR): 15 | * This consists of a series of detailed questions covering all kinds of nasty scenarios that have arisen in deployments past. For each one, he has to explain how his service will handle it. 16 | * For most of them, the standard framework has pre-generated code that covers it, or he is able to use standard patterns. 17 | -------------------------------------------------------------------------------- /src/vision/submitted_stories/status_quo/barbara_anguishes_over_http.md: -------------------------------------------------------------------------------- 1 | # 😱 Status quo stories: Barbara Anguishes Over HTTP 2 | 3 | ## 🚧 Warning: Draft status 🚧 4 | 5 | This is a draft "status quo" story submitted as part of the brainstorming period. It is derived from real-life experiences of actual Rust users and is meant to reflect some of the challenges that Async Rust programmers face today. 6 | 7 | If you would like to expand on this story, or adjust the answers to the FAQ, feel free to open a PR making edits (but keep in mind that, as they reflect people's experiences, status quo stories [cannot be wrong], only inaccurate). Alternatively, you may wish to [add your own status quo story][htvsq]! 8 | 9 | ## The story 10 | 11 | Barbara is starting a new project, working together with Alan. They want to write a Rust library and as part of it they will need to make a few HTTP calls to various web services. While HTTP is part of the responsibilities of the library it is by no means the only thing the library will need to do. 12 | 13 | As they are pair programming, they get the part of the library where HTTP will be involved and Alan asks Barbara, "OK, how do I make an HTTP request?". 14 | 15 | As an experienced async Rust developer Barbara has been dreading this question from the start of the project. She's tempted to ask "How long do you have?", but she quickly gathers herself and starts to outline the various considerations. She starts with a relatively simple question: "Should we use an HTTP library with a sync interface or an async interface?". 16 | 17 | Alan, who comes from a JavaScript background, remembers the transition from callbacks to async/await in that language. He assumes Rust is merely making its transition to async/await, and it will eventually be the always preferred choice. He hesitates and asks Barbara: "Isn't async/await always better?". Barbara, who can think of many scenarios where a blocking, sync interface would likely be better, weighs whether going done the rabbit-hole of async vs sync is the right way to spend their time. She decides instead to try to directly get at the question of whether they should use async for this particular project. She knows that bridging sync and async can be difficult, and so there's another question they need to answer first: "Are we going to expose a sync or an async interface to the users of our library?". 18 | 19 | Alan, still confused about when using a sync interface is the right choice, replies as confident as he can: "Everybody wants to use async these days. Let's do that!". He braces for Barbara's answer as he's not even sure what he said is actually true. 20 | 21 | Barbara replies, "If we expose an async API then we need to decide which async HTTP implementation we will use". As she finishes saying this, Barbara feels slightly uneasy. She knows that it is possible to use a sync HTTP library and expose it through an async API, but she fears totally confusing Alan and so decides to not mention this fact. 22 | 23 | Barbara looks over at Alan and sees a blank stare on his face. She repeats the question: "So, which async HTTP implementation should we use?". Alan responds with the only thing that comes to his mind: "which one is the best?" to which Barbara responds "Well, it depends on which async runtime you're using". 24 | 25 | Alan, feeling utterly dejected and hoping that the considerations will soon end tries a new route out of this conversation: "Can we allow the user of the library to decide?". 26 | 27 | Barbara thinks to herself, "Oh boy, we could provide a trait that abstracts over the HTTP request and response and allow the user to provide the implementation for whatever HTTP library they want... BUT, if we ever need any additional functionality that an async runtime needs to expose - like async locks or async timers - we might be forced to pick an actual runtime implementation on behalf of the user... Perhaps, we can put the most popular runtime implementations behind feature flags and let the user chose that way... BUT what if we want to allow plugging in of different runtimes?" 28 | 29 | Alan, having watched Barbara stare off into the distance for what felt like a half-hour, feels bad for his colleague. All he can think to himself is how Rust is so much more complicated that C#. 30 | 31 | ## 🤔 Frequently Asked Questions 32 | 33 | ### **What are the morals of the story?** 34 | 35 | * What is a very mundane and simple decision in many other languages, picking an HTTP library, requires users to contemplate *many* different considerations. 36 | * There is no practical way to choose an HTTP library that will serve most of the ecosystem. Sync/Async, competing runtimes, etc. - someone will always be left out. 37 | * HTTP is a small implementation detail of this library, but it is a HUGE decision that will ultimately be the biggest factor in who can adopt their library. 38 | 39 | ### **What are the sources for this story?** 40 | Based on the author's personal experience of taking newcomers to Rust through the decision making process of picking an HTTP implementation for a library. 41 | 42 | ### **Why did you choose [Barbara][] to tell this story?** 43 | Barbara knows all the considerations and their consequences. A less experienced Rust developer might just make a choice even if that choice isn't the right one for them. 44 | 45 | [Alan]: ../../characters/alan.md 46 | [Grace]: ../../characters/grace.md 47 | [Niklaus]: ../../characters/niklaus.md 48 | [Barbara]: ../../characters/barbara.md 49 | [htvsq]: ../status_quo.md 50 | [cannot be wrong]: ../../how_to_vision/comment.md#comment-to-understand-or-improve-not-to-negate-or-dissuade 51 | -------------------------------------------------------------------------------- /src/vision/submitted_stories/status_quo/barbara_gets_burned_by_select.md: -------------------------------------------------------------------------------- 1 | # 😱 Status quo stories: Template 2 | 3 | ## 🚧 Warning: Draft status 🚧 4 | 5 | This is a draft "status quo" story submitted as part of the brainstorming period. It is derived from real-life experiences of actual Rust users and is meant to reflect some of the challenges that Async Rust programmers face today. 6 | 7 | If you would like to expand on this story, or adjust the answers to the FAQ, feel free to open a PR making edits (but keep in mind that, as they reflect peoples' experiences, status quo stories [cannot be wrong], only inaccurate). Alternatively, you may wish to [add your own status quo story][htvsq]! 8 | 9 | ## The story 10 | 11 | Barbara is working on the [YouBuy] server. In one particular part of the story, she has a process that has to load records from a database on the disk. As she receives data from the database, the data is sent into a channel for later processing. She writes an `async fn` that looks something like this: 12 | 13 | ```rust 14 | async fn read_send(db: &mut Database, channel: &mut Sender<...>) { 15 | loop { 16 | let data = read_next(db).await; 17 | let items = parse(&data); 18 | for item in items { 19 | channel.send(item).await; 20 | } 21 | } 22 | } 23 | ``` 24 | 25 | This database load has to take place while also fielding requests from the user. The routine that invokes `read_send` uses [`select!`](https://docs.rs/futures/0.3.14/futures/macro.select.html) for this purpose. It looks something like this: 26 | 27 | ```rust 28 | let mut db = ...; 29 | let mut channel = ...; 30 | loop { 31 | futures::select! { 32 | _ = read_send(&mut file, &mut channel) => {}, 33 | some_data = socket.read_packet() => { 34 | // ... 35 | } 36 | } 37 | } 38 | ``` 39 | 40 | This setup seems to work well a lot of the time, but Barbara notices that the data getting processed is sometimes incomplete. It seems to be randomly missing some of the rows from the middle of the database, or individual items from a row. 41 | 42 | ### Debugging 43 | 44 | She's not sure what could be going wrong! She starts debugging with print-outs and logging. Eventually she realizes the problem. Whenever a packet arrives on the socket, the `select!` macro will drop the other futures. This can sometime cause the `read_send` function to be canceled in between reading the data from the disk and sending the items over the channel. Ugh! 45 | 46 | Barbara has a hard time figuring out the best way to fix this problem. 47 | 48 | ## 🤔 Frequently Asked Questions 49 | 50 | ### **What are the morals of the story?** 51 | 52 | * Cancellation doesn't always cancel the entire task; particularly with `select!`, it sometimes cancels just a small piece of a given task. 53 | * This is in tension with Rust's original design, which was meant to tear down an entire thread or task at once, precisely because of the challenge of writing exception-safe code. 54 | * Cancellation in Async Rust therefore can require fine-grained recovery. 55 | 56 | ### **What are the sources for this story?** 57 | 58 | This was based on [tomaka's blog post](https://tomaka.medium.com/a-look-back-at-asynchronous-rust-d54d63934a1c), which also includes a number of possible solutions, all of them quite grungy. 59 | 60 | ### **Why did you choose Barbara to tell this story?** 61 | 62 | The problem described here could strike anyone, including veteran Rust users. It's a subtle interaction that is independent of source language. Also, the original person who reported it, tomaka, is a veteran Rust user. 63 | 64 | ### **How would this story have played out differently for the other characters?** 65 | 66 | They would likely have a hard time diagnosing the problem. It really depends on how well they have come to understand the semantics of cancellation. This is fairly independent from programming language background; knowing non-async Rust doesn't help in particular, as this concept is specific to async code. 67 | 68 | ### What is different between this story and other cancellation stories? 69 | 70 | There is already a story, ["Alan builds a cache"] that covers some of the challenges around cancellation. It is quite plausible that those stories could be combined, but the focus of this story is different. The key moral of this story is that certain combinators, notably `select!`, can cause small pieces of a single task to be torn down and canceled. This cancellation can occur for any reason -- it is not always associated with (for example) clients timing out or closing sockets. It might be (as in this story) the result of clients *sending* data! 71 | 72 | This is one key point that makes cancellation in async Rust rather different than panics in sync Rust. Panics in sync Rust generally occur for bugs, to start, and they are typically not meant to be recovered from except at a coarse-grained level. In contrast, as this story shows, cancellation can require fine-grained recovery and for non-bug events. 73 | 74 | 75 | ["Alan builds a cache"]: alan_builds_a_cache.md 76 | [character]: ../../characters.md 77 | [status quo stories]: ../status_quo.md 78 | [Alan]: ../../characters/alan.md 79 | [Grace]: ../../characters/grace.md 80 | [Niklaus]: ../../characters/niklaus.md 81 | [Barbara]: ../../characters/barbara.md 82 | [htvsq]: ../status_quo.md 83 | [cannot be wrong]: ../../how_to_vision/comment.md#comment-to-understand-or-improve-not-to-negate-or-dissuade 84 | [YouBuy]: ../../projects/YouBuy.md -------------------------------------------------------------------------------- /src/vision/submitted_stories/status_quo/barbara_wants_async_insights.md: -------------------------------------------------------------------------------- 1 | # 😱 Status quo stories: Barbara wants Async Insights 2 | 3 | ## 🚧 Warning: Draft status 🚧 4 | 5 | This is a draft "status quo" story submitted as part of the brainstorming period. It is derived from real-life experiences of actual Rust users and is meant to reflect some of the challenges that Async Rust programmers face today. 6 | 7 | If you would like to expand on this story, or adjust the answers to the FAQ, feel free to open a PR making edits (but keep in mind that, as they reflect peoples' experiences, status quo stories [cannot be wrong], only inaccurate). Alternatively, you may wish to [add your own status quo story][htvsq]! 8 | 9 | ## The story 10 | 11 | [Barbara] has an initial prototype of a new service she wrote in sync Rust. She then decides, since the service is extremely I/O bound, to port it to async Rust and her benchmarks have led her to believe that performance is being left on the table. 12 | 13 | She does this by sprinkling `async/.await` everywhere, picking an executor, and moving dependencies from sync to async. 14 | 15 | Once she has the program compiling, she thinks "oh that was easy". She runs it for the first time and surprisingly she finds out that when hitting an endpoint, nothing happens. 16 | 17 | Barbara, always prepared, has already added logging to her service and she checks the logs. As she expected, she sees here that the endpoint handler has been invoked but then... nothing. Barbara exclaims, "Oh no! This was not what I was expecting, but let's dig deeper." 18 | 19 | She checks the code and sees that the endpoint spawns several tasks, but unfortunately those tasks don't have much logging in them. 20 | 21 | Barbara knows that debugging with a traditional debugger is not very fruitful in async Rust. She does a deep dive into the source code and doesn't find anything. Then she adds much more logging, but to her dismay she finds that a particular task seems stuck, but she has no idea why. 22 | 23 | She really wishes that there was a way to get more insight into why the task is stuck. These were the thoughts inside her head at that moment: 24 | * Is it waiting on I/O? 25 | * Is there a deadlock? 26 | * Did she miss some sync code that might still be there and messing with the executor? 27 | 28 | For the I/O question she knows to use some tools on her operating system (lsof). This reveals some open sockets but she's not sure how to act on this. 29 | 30 | She scans the code for any std lib imports that might be blocking, but doesn't find anything. 31 | 32 | After hours of crawling through the code, she notices that her task is receiving a message from a bounded async channel. She changes this to be an unbounded channel and then things start working. 33 | 34 | She wants to know why the code was not working, but unfortunately she has no way to gain insight into this issue. She fears that her task might use too much memory knowing that the channel is unbounded, but she can't really tell. 35 | 36 | She thinks, "Anyhow it is working now, let's see if we got some performance gains." After thorough benchmarking she finds out that she didn't quite get the performance gain she was expecting. "Something is not working, as intended", she thinks. 37 | 38 | ## 🤔 Frequently Asked Questions 39 | 40 | ### **What are the morals of the story?** 41 | * There are very few ways to get insights into running systems. Tracing is state of the art. `console.log` #ftw 42 | * Tracing is a static activity and there's no way to dynamically gain insights. 43 | * While it's possible to find solutions to these issues, often you don't have insight into if those solutions bring new problems. 44 | * Debugging process for non-trivial issues is almost guaranteed to be painful and expensive. 45 | 46 | ### **What are the sources for this story?** 47 | [Issue 75](https://github.com/rust-lang/wg-async/issues/75) 48 | 49 | ### **What are examples of the kinds of things a user might want to have insight into?** 50 | * Custom Events - logging/tracing (Per task?) 51 | * Memory consumption per task. 52 | * I/O handles in waiting state per task. 53 | * Number of tasks and their states over time. 54 | * Wake and drop specific tasks. 55 | * **Denoised** stack traces and/or stack traces that are task aware. 56 | * Who spawned the task? 57 | * Worker threads that are blocked from progressing tasks forward. 58 | * Tasks that are not progressing. 59 | 60 | ### **Why did you choose [Barbara] to tell this story?** 61 | Barbara knows what she's doing, but still there is little way to get insights. 62 | 63 | ### **How would this story have played out differently for the other characters?** 64 | Depending on what languages he was using before, [Alan] would likely have had experience with a stronger tooling story: 65 | * The highly [debuggable](https://youtu.be/JvBT4XBdoUE) BEAM (a VM), for Erlang. 66 | * [Delve](https://github.com/go-delve/delve), the debugging tool for Go. 67 | * Using [Visual Studio](https://devblogs.microsoft.com/visualstudio/how-do-i-debug-async-code-in-visual-studio/) to debug C#. 68 | * Debugging async Java using [IntelliJ](https://www.jetbrains.com/help/idea/debug-asynchronous-code.html). 69 | 70 | [character]: ../../characters.md 71 | [status quo stories]: ../status_quo.md 72 | [Alan]: ../../characters/alan.md 73 | [Grace]: ../../characters/grace.md 74 | [Niklaus]: ../../characters/niklaus.md 75 | [Barbara]: ../../characters/barbara.md 76 | [htvsq]: ../status_quo.md 77 | [cannot be wrong]: ../../how_to_vision/comment.md#comment-to-understand-or-improve-not-to-negate-or-dissuade 78 | -------------------------------------------------------------------------------- /src/vision/submitted_stories/status_quo/barbara_wishes_for_easy_runtime_switch.md: -------------------------------------------------------------------------------- 1 | # 😱 Status quo stories: Barbara wishes for an easy runtime switch 2 | 3 | ## 🚧 Warning: Draft status 🚧 4 | 5 | This is a draft "status quo" story submitted as part of the brainstorming period. It is derived from 6 | real-life experiences of actual Rust users and is meant to reflect some of the challenges that Async 7 | Rust programmers face today. 8 | 9 | ## The story 10 | 11 | [Barbara] has been working on an async codebase for the past 5 years. It is extremely mature and 12 | quite large (in the millions of lines of code). They've been relying on tokio as their async runtime 13 | and the codebase makes heavy use of its rich API. It has served them well over the years and they're 14 | very happy with it. 15 | 16 | Barbara knows about async-std but has never used it. She has wondered for a while how her 17 | application would work and perform if she had used async-std instead. She decides to test it out by 18 | porting her projects from tokio to async-std. 19 | 20 | To their disappointment, they discover many areas, where their choice of runtime permeates the code 21 | base: 22 | 23 | * tokio provides variants of helpers macros and types, like `tokio::select!` and `tokio::Mutex`. 24 | These helpers can be used without the rest of tokio, and there are also alternatives from the 25 | `futures` crate and elsewhere (albeit with subtle differences). 26 | * tokio uses a custom version of `AsyncRead` and `AsyncWrite` traits which differ from the ones used 27 | by other parts of the ecosystem. 28 | * The tokio API is needed to create core runtime operations like timers (`tokio::time::sleep`) and 29 | to launch tasks; there doesn't seem to be a standard way to abstract over those kinds of things in 30 | a runtime-independent way. 31 | * Some of their dependencies (e.g `hyper` and `reqwest`) are tied to tokio. In some cases, there are 32 | configuration options or ways to use those dependencies that don't depend on tokio, but there is 33 | no standard mechanism for that. 34 | 35 | These things aren't specific to tokio. There just doesn't seem to be a lot of consensus in the 36 | ecosystem on how to write "runtime-independent" code and in some cases there aren't any great 37 | options available (e.g., spawning tasks). 38 | 39 | They investigate the possibility of providing some sort of compatibility layer between tokio 40 | and their new runtime of choice but this turns out to not seem like the right way to go as this 41 | compatibility layer would require too much overhead. 42 | 43 | Realizing that the task of porting the entire code base to async-std, will take a lot of effort and 44 | time, Barbara decides to give up. She is very disappointed. 45 | 46 | ## 🤔 Frequently Asked Questions 47 | 48 | ### **What are the morals of the story?** 49 | * Using a certain executor often means using a certain run-time ecosystem. This often locks the user 50 | into that ecosystem. 51 | * Tying yourself to a certain executor means that you are tied to the priorities of that executor. 52 | You may be happy with the run-time ecosystem, but have special needs that the default executor 53 | does not provide. If the executor doesn't have an extensibility model, you're stuck. **Note:** 54 | It is perfectly reasonable for a general purpose executor to not be able or willing to cater for 55 | specialized needs. 56 | * All of this is made worse by that fact that [run-time agnostic libraries] are difficult and 57 | sometimes even impossible to write. 58 | 59 | ### **What are the sources for this story?** 60 | 61 | This story is more of a thought experiment than a recounting of a true story. We just asked 62 | logically what would happen if a team working on code base where it was assumed they could use a 63 | specific runtime decides to use a different runtime. 64 | 65 | ### **Why did you choose Barbara to tell this story?** 66 | The story assumes a Rust programmer that has worked for several years on a large and complex Rust 67 | codebase, so Barbara is the natural choice here. 68 | 69 | ### **How would this story have played out differently for the other characters?** 70 | It wouldn't. If this story happens them, they're on the same level of Rust expertise as Barbara is. 71 | 72 | [Barbara]: ../../characters/barbara.md 73 | [run-time agnostic libraries]: https://github.com/rust-lang/wg-async/issues/45 74 | -------------------------------------------------------------------------------- /src/vision/submitted_stories/status_quo/grace_deploys_her_service.md: -------------------------------------------------------------------------------- 1 | # 😱 Status quo stories: Grace deploys her service and hits obstacles 2 | 3 | [Alan]: ../../characters/alan.md 4 | [Grace]: ../../characters/grace.md 5 | [Niklaus]: ../../characters/niklaus.md 6 | [Barbara]: ../../characters/barbara.md 7 | 8 | [Grace deploys her service and is able to fix problems]: ../shiny_future.md#grace-deploys-her-service-and-is-able-to-fix-problems 9 | 10 | ## 🚧 Warning: Draft status 🚧 11 | 12 | This is a draft "status quo" story submitted as part of the brainstorming period. It is derived from real-life experiences of actual Rust users and is meant to reflect some of the challenges that Async Rust programmers face today. 13 | 14 | ## The story 15 | 16 | When examining her service metrics, [Grace] notices tail latencies in the P99 that exceed their target. She identifies GC in the routing layer as the culprit. Grace follows industry trends and is already aware of Rust and its ecosystem at a high level. She decides to investigate rewriting the routing service in Rust. 17 | 18 | To meet throughput requirements, Grace has already decided to use a thread-per-core model and minimize cross-thread communication. She explores available ecosystem options and finds no option that gets her exactly what she is looking for out of the box. However, she can use Tokio with minimal configuration to achieve her architecture. 19 | 20 | A few months of frantic hacking follow. 21 | 22 | montage of cats typing 23 | 24 | Soon enough, she and her team have a proof of concept working. They run some local stress tests and notice that 5% of requests hang and fail to respond. The client eventually times out. She cannot reproduce this problem when running one-off requests locally. It only shows up when sending above 200 requests-per-second. 25 | 26 | She realizes that she doesn't have any tooling to give her insight into what's going on. She starts to add lots of logging, attempting to tie log entries to specific connections. Using an operating system tool, she can identify the socket addresses for the hung connections, so she also includes the socket addresses in each log message. She then filters the logs to find entries associated with hung connections. Of course, the logs only tell her what the connection managed to do successfully; they don't tell her why it stopped -- so she keeps going back to add more logging until she can narrow down the exact call that hangs. 27 | 28 | Eventually, she identifies that the last log message is right before authenticating the request. An existing C library performs authentication, integrated with the routing service using a custom future implementation. She eventually finds a bug in the implementation that resulted in occasional lost wake-ups. 29 | 30 | She fixes the bug. The service is now working as expected and meeting Grace's performance goals. 31 | 32 | ## 🤔 Frequently Asked Questions 33 | 34 | ### **What are the morals of the story?** 35 | * When coming from a background of network engineering, users will bring their own design choices around architecture. 36 | * Examples: [seastar](http://seastar.io/) and [Glommio](https://www.datadoghq.com/blog/engineering/introducing-glommio/) 37 | * There is a lack of debugging tools for async. 38 | * Writing futures by hand is error prone. 39 | 40 | ### **What are the sources for this story?** 41 | This is based on the experiences of helping a tokio user to diagnose a bug in their code. 42 | 43 | ### **Why did you choose Grace to tell this story?** 44 | * The actual user who experienced this problem fit the profile of Grace. 45 | * The story is focused on the experience of people aiming to use workflows they are familiar with from C in a Rust setting. 46 | 47 | ### **How would this story have played out differently for the other characters?** 48 | Alan or Niklaus may well have had a much harder time diagnosing the problem due to not having as much of a background in systems programming. For example, they may not have known about the system tool that allowed them to find the list of dangling connections. 49 | 50 | ### **Could Grace have used another runtime to achieve the same objectives?** 51 | * Maybe! But in this instance the people this story is based on were using tokio, so that's the one we wrote into the story. 52 | * (If folks want to expand this answer with details of how to achieve similar goals on other runtimes that would be welcome!) 53 | 54 | -------------------------------------------------------------------------------- /src/vision/submitted_stories/status_quo/grace_wants_a_zero_copy_api.md: -------------------------------------------------------------------------------- 1 | # 😱 Status quo stories: Grace wants a zero-copy API 2 | 3 | [Alan]: ../../characters/alan.md 4 | [Grace]: ../../characters/grace.md 5 | [Niklaus]: ../../characters/niklaus.md 6 | [Barbara]: ../../characters/barbara.md 7 | 8 | ## 🚧 Warning: Draft status 🚧 9 | 10 | This is a draft "status quo" story submitted as part of the brainstorming period. It is derived from real-life experiences of actual Rust users and is meant to reflect some of the challenges that Async Rust programmers face today. 11 | 12 | ## The story 13 | 14 | Grace had written lots of operating system code in the past, and up until recently was working on a project using [DPDK](https://www.dpdk.org/) for zero-copy networking. The vast majority of the bugs 15 | that Grace found were related to memory (mis)management, so she is excited for the prospect of trying Rust as part of her new job. 16 | 17 | However, Grace has a hard time getting this to work without heavily resorting to `unsafe` constructs. As she evolves her undertanding of Rust, she looks hopefully at the signature of `poll_read`: 18 | 19 | ```rust 20 | fn poll_read( 21 | self: Pin<&mut Self>, 22 | cx: &mut Context, 23 | buf: &mut [u8] 24 | ) -> Poll> 25 | ``` 26 | 27 | She notices that the buffer is always passed to the invocation, but she can't pass it down to the operating system: because in rust-async tasks can be canceled at any time, which would 28 | free the buffer, those buffers are not guaranteed to be alive throughout the entire operation and there is no good way to extend their lifetime. There needs to be at least one copy! 29 | 30 | Grace hears from her coworkers that they are all using Tokio anyway. But the Tokio traits, although different from the standard traits, are not much better: 31 | 32 | ```rust 33 | fn poll_read( 34 | self: Pin<&mut Self>, 35 | cx: &mut Context<'_>, 36 | buf: &mut ReadBuf<'_> 37 | ) -> Poll>; 38 | ``` 39 | 40 | There's a specialized type for the buffer, but its management and lifetime are still not suitable for zero-copy I/O. 41 | 42 | Grace then came across a famous [blog post](https://boats.gitlab.io/blog/post/io-uring/) from a seasoned developer that mentions 43 | another trait, `AsyncBufRead`, but she immediately identifies two issues with that: 44 | 45 | * There is not a similar trait for writes, which suffer from much the same problem 46 | * Grace's team is already using a plethora of convenience traits built upon these base traits, including `AsyncReadExt` and `AsyncBufReadExt`, 47 | and they all pass a buffer, forcing a copy. 48 | 49 | Grace now has no good choices: she can live with the performance penalty of the copies, which lets her down since she how has the feeling she 50 | could do more with C++, or she can come up with her own specialized traits, which will make her work harder to consume by her team. 51 | 52 | ## 🤔 Frequently Asked Questions 53 | 54 | ### **What are the morals of the story?** 55 | 56 | * The cancellation problem and buffer lifetimes make it impossible to keep a user-provided buffer alive. That makes zero-copy I/O much harder 57 | than it could be. 58 | 59 | ### **What are the sources for this story?** 60 | 61 | * Personal experience. 62 | 63 | ### **Why did you choose Grace to tell this story?** 64 | 65 | * Grace has experience with C/C++, which is still the de-facto language for very low level things like zero-copy. The author had a similar experience 66 | when trying to expose zero-copy APIs. 67 | 68 | ### **How would this story have played out differently for the other characters?** 69 | 70 | * Zero-copy I/O is an important, but fairly niche use case that requires specialized prior knowledge that usually is only found among system-level 71 | programmers. 72 | * That is usually done in C/C++, and Grace is the only one that is very likely to have this experience. 73 | * There is a chance Barbara would have ventured into similar problems. She would likely have had a similar experience than Grace. 74 | 75 | [character]: ../../characters.md 76 | [status quo stories]: ../status_quo.md 77 | [htvsq]: ../status_quo.md 78 | [cannot be wrong]: ../../how_to_vision/comment.md#comment-to-understand-or-improve-not-to-negate-or-dissuade 79 | -------------------------------------------------------------------------------- /src/vision/submitted_stories/status_quo/grace_writes_a_new_fb_service.md: -------------------------------------------------------------------------------- 1 | # 😱 Status quo stories: Template 2 | 3 | ## 🚧 Warning: Draft status 🚧 4 | 5 | This is a draft "status quo" story submitted as part of the brainstorming period. It is derived from real-life experiences of actual Rust users and is meant to reflect some of the challenges that Async Rust programmers face today. 6 | 7 | If you would like to expand on this story, or adjust the answers to the FAQ, feel free to open a PR making edits (but keep in mind that, as they reflect peoples' experiences, status quo stories [cannot be wrong], only inaccurate). Alternatively, you may wish to [add your own status quo story][htvsq]! 8 | 9 | ## The story 10 | 11 | This tells the story of Grace, an engineer working at Facebook on C++ 12 | services. 13 | 14 | * Grace writes C++ services at Facebook, built upon many libraries and support 15 | infrastructure 16 | * Grace's last project had several bad bugs related to memory safety, and she 17 | is motivated to give Rust a shot on a new service she's writing 18 | * First, she must determine if there are Rust bindings to the other FB 19 | services her new service will depend on 20 | * She determines that she'll need to write a binding to the FooDB service 21 | using cxx 22 | * She also determines that several crates she'll need from crates.io aren't 23 | vendored in the FB monorepo, so she'll need to get them and their 24 | dependencies imported. She'll need to address any version conflicts and 25 | special build rules since FB uses Buck and not Cargo to build all code 26 | * While developing her service, Grace discovers that IDE features she's used 27 | to in VS Code don't always work for Rust code 28 | * Grace writes up the performance and safety benefits of her new service after 29 | it's first month of deployment. Despite the tooling issues, the end result 30 | is a success 31 | 32 | ## 🤔 Frequently Asked Questions 33 | 34 | *Here are some standard FAQ to get you started. Feel free to add more!* 35 | 36 | ### **What are the morals of the story?** 37 | 38 | * Building successful Rust services in a company that has lots of existing 39 | tooling and infrastructure can be difficult, as Grace must do extra work 40 | when new ground is tread 41 | * Big companies like Facebook have large monorepos and custom build systems 42 | and the standard Rust tooling may not be useable in that environment 43 | * Facebook has a large team making developer's lives easier, but it is 44 | focused around the most common workflows, and Grace must work a little 45 | harder for now as Rust support is in its early days 46 | * Integrating with existing C++ code is quite important as Grace cannot 47 | rewrite existing services 48 | 49 | ### **What are the sources for this story?** 50 | 51 | This story is compiled from internal discussions with Facebook engineers and 52 | from internal reports of successful Rust projects. 53 | 54 | ### **Why did you choose Grace to tell this story?** 55 | 56 | Both Alan or Grace could be appropriate, but I chose Grace in order to focus 57 | on tooling and C++ service integration issues. 58 | 59 | ### **How would this story have played out differently for the other characters?** 60 | 61 | Had I chosen Alan, a Python programmer at Facebook, there is probably a lot 62 | more learning curve with Rust's async mechanics. Python programmers using 63 | async don't necessarily have analogs for things like `Pin` for example. 64 | 65 | [character]: ../../characters.md 66 | [status quo stories]: ../status_quo.md 67 | [Alan]: ../../characters/alan.md 68 | [Grace]: ../../characters/grace.md 69 | [Niklaus]: ../../characters/niklaus.md 70 | [Barbara]: ../../characters/barbara.md 71 | [htvsq]: ../status_quo.md 72 | [cannot be wrong]: ../../how_to_vision/comment.md#comment-to-understand-or-improve-not-to-negate-or-dissuade 73 | -------------------------------------------------------------------------------- /src/vision/submitted_stories/status_quo/niklaus_wants_to_share_knowledge.md: -------------------------------------------------------------------------------- 1 | # 😱 Status quo stories: Niklaus Wants to Share Knowledge 2 | 3 | 4 | ## 🚧 Warning: Draft status 🚧 5 | 6 | This is a draft "status quo" story submitted as part of the brainstorming period. It is derived from real-life experiences of actual Rust users and is meant to reflect some of the challenges that Async Rust programmers face today. 7 | 8 | If you would like to expand on this story, or adjust the answers to the FAQ, feel free to open a PR making edits (but keep in mind that, as they reflect peoples' experiences, status quo stories [cannot be wrong], only inaccurate). Alternatively, you may wish to [add your own status quo story][htvsq]! 9 | 10 | ## The story 11 | 12 | Niklaus, who sometimes goes by the pen name "Starol Klichols", has authored some long-form documentation about Rust that people have found helpful. One could even go so far as to call this documentation a ["book"][trpl]. 13 | 14 | Niklaus has typically minimized the use of crates in documentation like this as much as possible. Niklaus has limited time to dedicate to keeping the documentation up to date, and given the speed at which the ecosystem sometimes evolves, it's hard to keep up when crates are involved. Also, Niklaus would like to avoid limiting the readership of the documentation to the users of a particular crate only, and would like to avoid any accusations of favoritism. 15 | 16 | But Niklaus would really really like to document async to avoid disappointing [people like Barbara]! 17 | 18 | Niklaus was excited about [the RFC proposing that `block_on` be added to the stdlib][block-on-rfc], because it seemed like that would solve Niklaus' problems. Niklaus would really like to include `async` in a big update to the documentation. No pressure. 19 | 20 | [trpl]: https://doc.rust-lang.org/stable/book/ 21 | [people like Barbara]: https://github.com/rust-lang/wg-async/blame/5ce418ac4076850f515034010cc51b707441f695/src/vision/status_quo/barbara_makes_their_first_steps_into_async.md#L22 22 | [block-on-rfc]: https://github.com/rust-lang/rust/pull/65875 23 | [htvsq]: ../status_quo.md 24 | [cannot be wrong]: ../../how_to_vision/comment.md#comment-to-understand-or-improve-not-to-negate-or-dissuade 25 | 26 | 27 | ## 🤔 Frequently Asked Questions 28 | 29 | ### **What are the morals of the story?** 30 | Writing documentation to go with the language/stdlib for something that is half in the language/stdlib and half in the ecosystem is hard. 31 | This is related to [Barbara's story](https://rust-lang.github.io/wg-async/vision/status_quo/barbara_makes_their_first_steps_into_async.html) about wanting to get started without needing to pick an executor. 32 | There are topics of async that apply no matter what executor you pick, but it's hard to explain those topics without picking an executor to demonstrate with. 33 | We all have too much work to do and not enough time. 34 | 35 | ### **What are the sources for this story?** 36 | * It me and Steve. Surprise! 37 | * [We've wanted to add async to the book for a long time](https://github.com/rust-lang/book/issues/1275). 38 | * So far, we use exactly one crate in the book, `rand`, and a recent update to `rand` caused readers confusion and caused a bunch of work on our part. [Take a look at all the issues linked to this PR](https://github.com/rust-lang/book/pull/2542). I really really really don't want to use more crates in the book. 39 | 40 | ### **Why did you choose *Niklaus* to tell this story?** 41 | Niko said I couldn't add new characters. 42 | 43 | ### **How would this story have played out differently for the other characters?** 44 | I happen to know that the next version of Programming Rust, whose authors might be described as different characters, includes `async` and uses `async-std`. So it's possible to just pick an executor and add async to the book, but I don't wanna. 45 | -------------------------------------------------------------------------------- /src/vision/submitted_stories/status_quo/template.md: -------------------------------------------------------------------------------- 1 | # 😱 Status quo stories: Template 2 | 3 | *This is a template for adding new "status quo" stories. To propose a new status quo PR, do the following:* 4 | 5 | * *Create a new file in the [`status_quo`] directory named something like `Alan_tries_to_foo.md` or `Grace_does_bar.md`, and start from [the raw source from this template]. You can replace all the italicized stuff. :)* 6 | * *Do not add a link to your story to the [`SUMMARY.md`] file; we'll do it after merging, otherwise there will be too many conflicts.* 7 | 8 | *For more detailed instructions, see the [How To Vision: Status Quo] page!* 9 | 10 | *If you're looking for ideas of what to write about, take a look at the [open issues]. You can also [open an issue of your own] to throw out an idea for others.* 11 | 12 | [How To Vision: Status Quo]: ../status_quo.md 13 | [the raw source from this template]: https://raw.githubusercontent.com/rust-lang/wg-async/master/src/vision/status_quo/template.md 14 | [`status_quo`]: https://github.com/rust-lang/wg-async/tree/master/src/vision/status_quo 15 | [`SUMMARY.md`]: https://github.com/rust-lang/wg-async/blob/master/src/SUMMARY.md 16 | [open issues]: https://github.com/rust-lang/wg-async/issues?q=is%3Aopen+is%3Aissue+label%3Astatus-quo-story-ideas 17 | [open an issue of your own]: https://github.com/rust-lang/wg-async/issues/new?assignees=&labels=good+first+issue%2C+help+wanted%2C+status-quo-story-ideas&template=-status-quo--story-issue.md&title= 18 | 19 | 20 | ## 🚧 Warning: Draft status 🚧 21 | 22 | This is a draft "status quo" story submitted as part of the brainstorming period. It is derived from real-life experiences of actual Rust users and is meant to reflect some of the challenges that Async Rust programmers face today. 23 | 24 | If you would like to expand on this story, or adjust the answers to the FAQ, feel free to open a PR making edits (but keep in mind that, as they reflect peoples' experiences, status quo stories [cannot be wrong], only inaccurate). Alternatively, you may wish to [add your own status quo story][htvsq]! 25 | 26 | ## The story 27 | 28 | *Write your story here! Feel free to add subsections, citations, links, code examples, whatever you think is best.* 29 | 30 | ## 🤔 Frequently Asked Questions 31 | 32 | *Here are some standard FAQ to get you started. Feel free to add more!* 33 | 34 | ### **What are the morals of the story?** 35 | *Talk about the major takeaways-- what do you see as the biggest problems.* 36 | 37 | ### **What are the sources for this story?** 38 | *Talk about what the story is based on, ideally with links to blog posts, tweets, or other evidence.* 39 | 40 | ### **Why did you choose *NAME* to tell this story?** 41 | *Talk about the character you used for the story and why.* 42 | 43 | ### **How would this story have played out differently for the other characters?** 44 | *In some cases, there are problems that only occur for people from specific backgrounds, or which play out differently. This question can be used to highlight that.* 45 | 46 | [character]: ../../characters.md 47 | [status quo stories]: ../status_quo.md 48 | [Alan]: ../../characters/alan.md 49 | [Grace]: ../../characters/grace.md 50 | [Niklaus]: ../../characters/niklaus.md 51 | [Barbara]: ../../characters/barbara.md 52 | [htvsq]: ../status_quo.md 53 | [cannot be wrong]: ../../how_to_vision/comment.md#comment-to-understand-or-improve-not-to-negate-or-dissuade 54 | -------------------------------------------------------------------------------- /src/vision/unresolved_questions.md: -------------------------------------------------------------------------------- 1 | # Major unresolved questions or controveries 2 | 3 | This section contains places where there remains significant design work to be done. 4 | It also contains some points of major controversy, where the path is clear, but many 5 | people disagree on whether to take it. These are places where further input can be useful. 6 | 7 | The page for each controversy attempts to summarize the various options available and some of 8 | the tradeoffs involved. 9 | -------------------------------------------------------------------------------- /src/vision/unresolved_questions/async_fn.md: -------------------------------------------------------------------------------- 1 | # How to represent the AsyncFn traits? 2 | 3 | As noted in the [async fn page](https://rust-lang.github.io/async-fundamentals-initiative/), the ["inline async fn"](https://rust-lang.github.io/async-fundamentals-initiative/design-discussions/dyn_async_trait.html) technique cannot represent async closures. 4 | -------------------------------------------------------------------------------- /src/vision/unresolved_questions/await_or_not.md: -------------------------------------------------------------------------------- 1 | # To await or not to await? 2 | 3 | Should we require you to use `.await`? After the epic syntax debates we had, wouldn't it be ironic if we got rid of it altogether, as [carllerche has proposed](https://carllerche.com/2021/06/17/six-ways-to-make-async-rust-easier/)? 4 | 5 | Basic idea: 6 | 7 | - When you invoke an async function, it could await by default. 8 | - You would write `async foo()` to create an "async expression" -- i.e., to get a `impl Async`. 9 | - You might instead write `async || foo()`, i.e., create an async closure. 10 | 11 | Appealing characteristics: 12 | 13 | - **More analogous to sync code.** In sync code, if you want to defer immediately executing something, you make a closure. Same in async code, but it's an async closure. 14 | - **Consistency around async-drop.** If we adopt an [async drop](https://rust-lang.github.io/async-fundamentals-initiative/design-discussions/async_drop.html) proposal, that implies that there will be "awaits" that occur as you exit a block (or perhaps from the control-flow of a `break` or `?`). These will not be signaled with a `.await`. So you can no longer rely on _every_ await point being visible with a keyword. 15 | - **No confusion around remembering to await.** Right now the compiler has to go to some lengths to offer you messages suggesting you insert `.await`. It'd be nice if you just didn't have to remember. 16 | - **Room for optimization.** When you first invoke an async function, it can immediately start executing; it only needs to create a future in the event that it suspends. This may also make closures somewhat smaller. 17 | - This could be partially achieved by adding an optional method on the trait that compiles a version of the fn meant to be used when it is _immediately awaited_. 18 | 19 | But there are some downsides: 20 | 21 | - **Churn.** Introducing a new future trait is largely invisible to users except in that it manifests as version mismatches. Removing the await keyword is a much more visible change. 22 | - **Await points are less visible.** There may be opportunity to introduce concurrency and so forth that is harder to spot when reading the code, particularly outside of an IDE. (In Kotlin, which adopts this model, suspend points are visible in the "gutter" of the editor, but this is not visible when reviewing patches on github.) 23 | - Await points today also indicate where a live `Send` or `Sync` value will affect if the future is send or sync (but with async-drop, this would no longer be true). 24 | - **Async becomes an effect.** In today's Rust, an "async function" desugars into a traditional function that returns a future. This function is called like any other, and hence it can implement the `Fn` traits and so forth. In this "await-less" Rust, an async function is called differently from other functions, because it induces an await. This means that we need to consider `async` as a kind of "effect" (like `unsafe`) in a way that is not today. 25 | - Similarly, how do we handle the case of `fn foo() -> impl Future`? Does that auto-await, or does it require an explicit `await` keyword? 26 | - What happens when you invoke an `async fn` in a sync environment? 27 | 28 | ## Frequently asked questions 29 | 30 | ### How could you do this anyway? Wouldn't it be a massive breaking change? 31 | 32 | It would have to take place over an edition. 33 | -------------------------------------------------------------------------------- /src/vision/unresolved_questions/cancellation.md: -------------------------------------------------------------------------------- 1 | # How best to integrate voluntary cancellation? 2 | -------------------------------------------------------------------------------- /src/vision/unresolved_questions/default_runtime.md: -------------------------------------------------------------------------------- 1 | # Default runtime? 2 | 3 | The [User's Manual of the future](../shiny_future/users_manual.md) suggests that one must still pick a runtime upfront and use a decorator like `#[runtime::main]`. This is "accidental complexity" for people learning async Rust: the choice of runtime is something they are not yet equipped to make. It would be better for users if they could just write `async fn main` and not choose a runtime yet (and then, later, once they are equipped to make the choice, opt for other runtimes). 4 | 5 | However, we also wish to avoid shipping and maintaining a runtime in the Rust stdlib. We want runtimes to live in the ecosystem and evolve over time. If we were to pick a "default runtime", that might favor one runtime at the expense of others. 6 | 7 | Should we pick a default runtime? If so, what criteria do we use to pick one, and how do we manage the technical side of things (e.g., we need to either ship the runtime with rustup or else insert some kind of implicit cargo dependency). 8 | -------------------------------------------------------------------------------- /src/vision/unresolved_questions/portable_without_generics.md: -------------------------------------------------------------------------------- 1 | # Extend stdlib to permit portable async without generics? 2 | -------------------------------------------------------------------------------- /src/welcome.md: -------------------------------------------------------------------------------- 1 | # 👋🏽 Welcome 2 | 3 | Welcome to the home of wg-async! This working group is focused 4 | around implementation/design of the “foundations” for Async I/O. 5 | 6 | You can learn more by reading our [charter]. 7 | 8 | [charter]: ./CHARTER.md 9 | 10 | ## 🛠️ Getting involved 11 | 12 | We have several [meetings] throughout the month. Feel free to stop by then (or any time!) to introduce yourself. We take meeting notes and keep them on our [HackMD](https://hackmd.io/@wg-async). 13 | 14 | If you're interested in fixing bugs, though, there is no need to wait for the meeting! [Take a look at the instructions here.][fix-bugs] 15 | 16 | We are actively working on bringing the [async vision] to reality, so there are lots of ways to help. 17 | Check out the [Roadmap] to see the various things we are working on. 18 | Each of the high level goals should have further instructions for how to get starting helping with that goal in particular. 19 | Look for the 🛠️ icon, which highlights areas where further how to help resources are available. 20 | 21 | [meetings]: ./meetings.md 22 | [fix-bugs]: ./triage.md#so-you-want-to-fix-a-bug 23 | [async vision]: ./vision.md 24 | [Roadmap]: ./vision/roadmap.md 25 | 26 | ## Zulip 27 | 28 | We hold discussions on [the `#wg-async` stream in Zulip](https://rust-lang.zulipchat.com/#narrow/stream/187312-wg-async) 29 | 30 | ## License 31 | 32 | Licensed under either of 33 | 34 | * Apache License, Version 2.0 ([LICENSE-APACHE](LICENSE-APACHE) or ) 35 | * MIT license ([LICENSE-MIT](LICENSE-MIT) or ) 36 | 37 | at your option. 38 | 39 | ### Contribution 40 | 41 | Unless you explicitly state otherwise, any contribution intentionally submitted 42 | for inclusion in this crate by you, as defined in the Apache-2.0 license, shall 43 | be dual licensed as above, without any additional terms or conditions. 44 | -------------------------------------------------------------------------------- /tools/fixlinks/.gitignore: -------------------------------------------------------------------------------- 1 | /target 2 | -------------------------------------------------------------------------------- /tools/fixlinks/Cargo.toml: -------------------------------------------------------------------------------- 1 | [package] 2 | name = "fixlinks" 3 | version = "0.1.0" 4 | authors = ["Tyler Mandry "] 5 | edition = "2018" 6 | 7 | [dependencies] 8 | camino = "1.0" 9 | regex = "1.5" 10 | pathdiff = "0.2" 11 | -------------------------------------------------------------------------------- /triagebot.toml: -------------------------------------------------------------------------------- 1 | [assign] 2 | --------------------------------------------------------------------------------