├── README.md └── rfcs ├── Py3-first.md ├── additional_cypto_cbor_libs.md ├── address_space_overrides.md ├── ahem_webfont.md ├── any.js-globals.md ├── assert_precondition.md ├── assert_precondition_rename.md ├── assert_throws.md ├── async_promise_test.md ├── asynchronous_setup.md ├── bigint_support.md ├── check_layout_wait_for_fonts.md ├── code_of_conduct.md ├── conditional_requirements.md ├── consume_user_activation.md ├── crash_test.md ├── crash_test_navigate.md ├── create-new-test-editor-repo.md ├── css_build_system.md ├── disable_chrome_stability.md ├── docker_github_registry.md ├── expected_fail_message_tag_for_subtests.md ├── fetch_json_et_al.md ├── get_signal.md ├── global-window-module.md ├── hide_test_state.md ├── http2.md ├── https_port_8444.md ├── integrate_wave_test_runner.md ├── interop_2022.md ├── interop_2023.md ├── interop_charter.md ├── manifest-path-trie.md ├── manifest-working-copy.md ├── matrix.md ├── minimize_restore_window.md ├── origin_policy.md ├── origin_policy_revert.md ├── page-size.md ├── polyfill_testing.md ├── print_test.md ├── promise.md ├── proxy_for_connection_testing.md ├── pullpanda_analytics.md ├── py_3.md ├── python-file-handler-local-imports.md ├── python_37.md ├── python_38.md ├── quic.md ├── reftest_simplification.md ├── reftest_variants.md ├── remote_channel.md ├── remove_ie_edge_legacy.md ├── remove_stub.md ├── rename_lint_whitelist.md ├── retry_unexpected.md ├── rfc183_remove_unused_dump_test_results_function_in_testharnessreport_js.md ├── rfc_as_pr.md ├── rfc_exemption_testdriver_method.md ├── shadowrealm-global.md ├── simplify_license.md ├── single_test.md ├── subsuites.md ├── test_arrow_name.md ├── test_rendered_event.md ├── test_scoring_change.md ├── testdriver-features.md ├── testdriver_bidi.md ├── testdriver_cookie_methods.md ├── testdriver_cross_origin.md ├── testdriver_generate_test_report.md ├── testdriver_get_cookie_methods.md ├── testdriver_in_other_types.md ├── testdriver_set_rph_registration_mode.md ├── testdriver_set_spc_transaction_mode.md ├── testharness_include_late_tests.md ├── top_level_subdomain.md ├── variant_name_not_empty.md ├── vendoring.md ├── wait_for.md ├── web_features.md ├── websockets_http2.md ├── websockets_py3.md ├── webtransport_h3_cert_hash_test_server.md ├── webtransport_h3_test_server.md ├── wpt-results-analysis.md ├── wpt_live.md ├── wpt_pr_live_turndown.md ├── wptfyi_enable_view_eq_test.md └── wptserve_py3.md /README.md: -------------------------------------------------------------------------------- 1 | # web-platform-tests RFCs 2 | 3 | Most changes in web-platform-tests can be made directly in pull requests 4 | following the usual review process. 5 | 6 | This RFC (request for comments) process is used to request wider review. 7 | 8 | ## When to use the RFC process 9 | 10 | Use this process for substantial changes which would impact other stakeholders 11 | or users of web-platform-tests. Examples of where it is likely to be useful: 12 | 13 | - Changes in [resources/](https://github.com/web-platform-tests/wpt/tree/master/resources) 14 | which would affect many test authors, such as extending testharness.js. 15 | - Changes in [tools/](https://github.com/web-platform-tests/wpt/tree/master/tools) 16 | which would affect downstream users of web-platform-tests, such as changing 17 | the manifest format or wptserve behavior. 18 | - Adding a new third-party code or service dependency. 19 | - Changing review workflows or policies. 20 | 21 | Cases where the RFC process need *not* be used: 22 | 23 | - Introducing a new top-level directory for a new specification. 24 | - Extending testdriver.js with a method that closely matches a [WebDriver Classic](https://w3c.github.io/webdriver/) endpoint or a [WebDriver BiDi](https://w3c.github.io/webdriver-bidi) command or an event. To notify maintainers of testdriver.js vendor integration, label the pull request `testdriver.js`. (This exemption was introduced by [RFC 127](https://github.com/web-platform-tests/rfcs/blob/master/rfcs/rfc_exemption_testdriver_method.md) and [RFC 185](https://github.com/web-platform-tests/rfcs/blob/master/rfcs/testdriver_bidi.md).) 25 | - Minor changes in behavior in where all call sites are known and accounted 26 | for. 27 | - Behavior-preserving refactoring with a low risk of regressions. 28 | 29 | ## The RFC process 30 | 31 | - Each RFC requires a PR in this repository consisting of a single 32 | markdown file added to the `rfcs` directory setting out the proposed 33 | change. This file must have the PR number as the title and at least 34 | the following sections: 35 | * Summary 36 | * Details 37 | * Risks 38 | - The proposal is discussed by the community and it is assumed that the 39 | proposal will change in accordance with that discussion. 40 | - At least one review approval is required to merge a PR. For changes that 41 | may impact downstream WPT consumers there is an additional requirement 42 | of agreement from those consumers that the proposed change can be 43 | accommodated in their workflows. 44 | - Anyone is welcome to add the PR to the monthly 45 | [WPT infra meeting](https://github.com/web-platform-tests/wpt-notes) to 46 | have a dsicsussion about it. To add an RFC to the agenda, tag the PR with the 47 | `agenda+` label. 48 | - If reviewers or downstream consumers are not responsive after at least two 49 | weeks, the PR should be added to the monthly 50 | [WPT infra meeting](https://github.com/web-platform-tests/wpt-notes) to 51 | ensure it gets discussed. 52 | - After review approval(s) and in the case of no substantive disagreement, the 53 | RFC is considered accepted 1 week after the first approval. If any participant 54 | requests it, more time can be granted to review the proposal. This will not be 55 | less than one week but may be longer (e.g. until the next WPT infra meeting). 56 | However if no feedback is forthcoming by the agreed date, the proposal may 57 | be considered accepted. 58 | - If substantive disagreement remains, then the issue is escalated to the 59 | [core team](https://github.com/orgs/web-platform-tests/teams/wpt-core-team/) 60 | for a decision: 61 | - Only arguments that have already been made on the issue may be taken into 62 | account. 63 | - If meaningful new information is presented, the comment period can be 64 | extended to ensure that all participants have time to consider it. 65 | - When no new information is forthcoming, a decision should be made within 1 66 | additional week. 67 | - The decision can be to defer the issue until a later time. 68 | - If necessary, the core team may decide using a ≥2/3 majority vote. 69 | - An RFC that is accepted is merged. One that is rejected is closed 70 | without merging. 71 | -------------------------------------------------------------------------------- /rfcs/Py3-first.md: -------------------------------------------------------------------------------- 1 | # RFC 65: Switching to "Py3-first" for WPT tests on CI 2 | 3 | ## Summary 4 | 5 | This is to propose switching test suite runs over to Python 3 on the WPT CI (Taskcluster and Azure). At the same time unit tests and infrastructure tests will still be running in Python 2 and Python 3. This is referred to as the “Py3-first" stage in the [Dropping Python 2 support plan](https://github.com/web-platform-tests/rfcs/issues/62). 6 | 7 | The minimum python 3 version supported is 3.6 with main focus on 3.8+. For the foreseeable future it is very likely to continue to run unit tests and infrastructure tests on both 3.6 and 3.8+ versions. The WPT test suite itself would run on 3.8+ in the CI. 8 | 9 | ## Changes on upstream CI 10 | 11 | **1. Switch the test suite runs to use Python 3** 12 | * [Switch wpt run on Taskcluster to Python 3 only](https://github.com/web-platform-tests/wpt/pull/26252) (specifically, Python 3.8) 13 | 14 | * Use Python 3.8.x in Azure Pipelines for [Safari](https://github.com/web-platform-tests/wpt/pull/25044) and [Edge](https://github.com/web-platform-tests/wpt/pull/24952) runs. 15 | 16 | 17 | **2. Keep the current CI setup of unit tests and infrastructure tests running on both Py2 and Py3** 18 | 19 | The unit tests and infrastructure tests are already running on both Py2 and Py3. Outcomes of unit tests on Py2 and Py3 are not fully identical yet. We have created [a Unit Tests Tracksheet](https://docs.google.com/spreadsheets/d/1__dE5_ABazhDl6ONp76tbkgAimj14gD7QeaUYtfsi_w) to track the differences and give explanations. 20 | 21 | 22 | 23 | ## Recommended actions for downstream (browser vendors) 24 | 25 | Switch wpt run to use Python 3, or at least run wptserve with Python 3 as soon as the upstream switches over and stabilizes. This minimizes the risk of exporting new Py2-only handlers to upstream. Waiting until after the upstream completes the switchover reduces churn in case we need to roll back to Py2 temporarily. 26 | 27 | ## Risks 28 | 29 | To assess possible risks and provide evidence for a move to Python 3 being safe, we have produced [a tracking spreadsheet](https://docs.google.com/spreadsheets/d/1cJcSW8PHe3A4m0dtBfjV4nrYLg67LzIznFbZIknDURc/edit#gid=1054524593). It compares the test results of wpt runs on Python2 and Python 3 for all the supported configurations including: 30 | * Linux+Chrome Dev 31 | * Linux+Firefox Nightly 32 | * Windows + Edge Dev 33 | * macOS + Safari Technology Preview 34 | 35 | Most different test results are caused by the fact that the test itself is flaky. For those flaky tests, links to flaky results in [wpt.fyi insights](https://wpt.fyi/insights) or [wpt.fyi recent runs ](https://wpt.fyi/runs) are given. 36 | 37 | Other cases fall into the following categories: 38 | * Different test names in `webdriver`. For some webdriver tests, test names are generated by pytest. If the argument passed to `@pytest.mark.parametrize` contains escape characters (e.g. `\n`, `\t` etc.), it produces different test names in Python2 and Python3. [Issue #24888](https://github.com/web-platform-tests/wpt/issues/24888) was raised for this and discussed. The outcome is to ignore any differences because there are only a small number of changes caused and arguably the test names created in Python 3 are more meaningful. 39 | * Different test names in some `webaudio` tests. We noticed `cors-check.https.html`produced different test names for both Chrome and Firefox, also `no-cors.https.html` for Firefox. This is due to the way the tests were written, which in some way is a bit strange while creating test names. 40 | * Python file handler local-imports issue, aka [issue #25678](https://github.com/web-platform-tests/wpt/issues/25678. An [RFC has been filed](https://github.com/web-platform-tests/rfcs/pull/68) along with [a fix](https://github.com/web-platform-tests/wpt/pull/26328) for this. 41 | * Newly introduced `.py` test files or codes. It is noted that before we move to the stage of "Py3-only", there is a risk that newly introduced `.py` codes might not be Python 3 compatible. We have fixed issues found in the tracking sheets but it is very likely that this kind of issues will appear again during the transition period. 42 | 43 | Looking into every different result of the WPT test suite between Python 2 and Python 3, we believe that we have each known Python 3 issue either fixed or at least explained. 44 | 45 | ## Rollback and success criteria 46 | 47 | Admins (smcgruer & Hexcles) will be on call on #testing at irc.w3.org on the flag day. In case of unexpected massive failures (e.g. a large number of tests fail to run or produce invalid results), they will revert the following PRs (overriding any checks if needed): 48 | * [[Taskcluster] Switch wpt run on CI to Python 3 only](https://github.com/web-platform-tests/wpt/pull/26252) 49 | * [[Azure Pipelines] Use Python 3.8.x for Safari test-suite runs](https://github.com/web-platform-tests/wpt/pull/25044) 50 | * [[Azure Pipelines] Use Python 3.8.x for Edge test-suite runs](https://github.com/web-platform-tests/wpt/pull/24952) 51 | 52 | The switchover will be considered complete and successful when we have a full set of runs from all supported configurations. 53 | -------------------------------------------------------------------------------- /rfcs/additional_cypto_cbor_libs.md: -------------------------------------------------------------------------------- 1 | # RFC 208: additional crypto and CBOR libraries 2 | 3 | ## Summary 4 | 5 | Several new web APIs require additional cryptography and CBOR libraries to 6 | properly test, for example the [Protected Audience Additional Bids feature](https://github.com/WICG/turtledove/blob/main/FLEDGE.md#623-additional-bid-keys ) 7 | uses Ed25519 signatures, the [Protected Audience Key-Value services](https://github.com/WICG/turtledove/blob/main/FLEDGE_Key_Value_Server_API.md#query-api-version-2) and 8 | [Bidding and Auction services](https://github.com/WICG/turtledove/blob/main/FLEDGE_browser_bidding_and_auction_API.md) use CBOR data encoding and HPKE encryption. 9 | These additional libraries are to support cryptography and the CBOR 10 | protocols not otherwise supported in JavaScript or Python, namely Ed25519, 11 | HPKE, and CBOR. There are open source libraries commonly available that 12 | implement these protocols and have compatible licenses. This RFC proposes 13 | adding such libraries to the tools/ directory (for the Python library) and 14 | to a [/third_party/ directory (for the JavaScript libraries)](https://github.com/web-platform-tests/rfcs/issues/46#issuecomment-587707539) so 15 | that web-platform-tests may exercise and verify proper compatible 16 | implementations of these new web APIs. 17 | 18 | ## Details 19 | 20 | ### We're proposing adding these Python libraries to the tools/ directory: 21 | 22 | These are all self-contained (no dependencies) pure-python libraries. 23 | 24 | ##### An HPKE Python implementation: 25 | https://github.com/dajiaji/pyhpke 26 | 27 | This HPKE library is intended to be used by test code running on wptserve 28 | that may receive an HPKE encrypted message and have to decrypt it and encrypt 29 | a response. This library is MIT licensed. 30 | 31 | ##### A CBOR Python implementation: 32 | https://github.com/agronholm/cbor2 33 | 34 | This CBOR library is intended to be used by test code running on wptserve that 35 | may receive an CBOR encoded message and have to decode it and encode a response. 36 | This library is MIT licensed. 37 | 38 | ### We're proposing adding these JavaScript libraries to a third_party/ subdirectory under our spec directory as per [this advice](https://github.com/web-platform-tests/rfcs/issues/46#issuecomment-587707539): 39 | 40 | These HPKE and CBOR libraries are used by test code to decrypt and decode data 41 | coming from JavaScript APIs to verify their contents, and used by test code to 42 | encode and encrypt response data. These libraries and all of their included 43 | dependencies are MIT licensed. 44 | 45 | ##### An HPKE JavaScript implementation: 46 | https://github.com/dajiaji/hpke-js 47 | 48 | This library has some dependencies, so we're proposing building/transpiling into a single hpke.js file. 49 | This library is MIT licensed. 50 | 51 | ##### A CBOR JavaScript implementation: 52 | https://github.com/paroga/cbor-js/blob/master/cbor.js 53 | 54 | This library is MIT licensed. 55 | 56 | ##### An Ed25519 JavaScript implementation: 57 | https://github.com/paulmillr/noble-ed25519 58 | 59 | This Ed25519 library is intended to be used by JavaScript test code 60 | that may receive an Ed25519 private key and message to sign that message, or 61 | a public key and signature to verify that signature. This library is MIT licensed. 62 | 63 | ## Risks 64 | 65 | Users of these libraries may need to update them from time to time if new 66 | functionality or fixes are required. This is likely not a big risk. 67 | 68 | The HPKE library, once built/transpiled into one JavaScript file, may be slightly harder to debug, but we'll disable minification when building. 69 | -------------------------------------------------------------------------------- /rfcs/ahem_webfont.md: -------------------------------------------------------------------------------- 1 | # RFC 22: Load Ahem as a Webfont always 2 | 3 | ## Summary 4 | Make all tests load the Ahem font as a webfont instead of expecting Ahem to be 5 | installed as a system font. 6 | 7 | ## Details 8 | Ahem is used in a range of reftests because it has glyphs of a very precise size 9 | and shape. WPT currently assumes that Ahem is installed as a system font, and 10 | provides a mechanism to install fonts where not available. 11 | 12 | However, some environments (such as CI systems) may not have this font installed 13 | and it may not be feasible to install it (eg: on Chromium CI). 14 | 15 | The proposal is to change this assumption and require all tests to load Ahem as 16 | a webfont. Here is a proposed PR for doing this and updating all existing tests: 17 | https://github.com/web-platform-tests/wpt/pull/16951 18 | 19 | ## Risks 20 | The main risk is inability to test behaviour specific to having this font as a 21 | system font, although no such tests seem to exist today. 22 | 23 | Another is that test runtime may be slower due to the network latency of loading 24 | the font, but this is expected to be small (if detectable at all) due to 25 | caching. 26 | -------------------------------------------------------------------------------- /rfcs/any.js-globals.md: -------------------------------------------------------------------------------- 1 | # RFC 52: Remove default and negated global support in \*.any.js 2 | 3 | ## Summary 4 | 5 | https://web-platform-tests.org/writing-tests/testharness.html#multi-global-tests has a confusing syntax for specifying globals. The default set of globals is always included implicitly, e.g., `global=window` is equivalent to `global=window,dedicatedworker` and `global=jsshell` is equivalent to `global=window,dedicatedworker,jsshell`. 6 | 7 | This RFC will make these changes: 8 | 9 | 1. Remove `default` (though keep `window,dedicatedworker` as the default if `// META: global=` is not present). 10 | 2. Remove `!` (negation). Instead test developers are expected to either rely on the default or explicitly list the globals they want. 11 | 12 | A test supposed to run in dedicated and shared workers would use `dedicatedworker,sharedworker` and not `!window,sharedworker`, `!window,worker,!serviceworker`, etc. 13 | 14 | ## Details 15 | 16 | 1. Update all `*.any.js` tests that use `// META: global=` to list all the relevant globals explicitly when possible. (Some necessarily rely on `!default` or `!`.) 17 | 2. Atomically: 18 | 1. Remove support for `default` and `!`. 19 | 1. Update all remaining `*.any.js` tests that use `!` to list all the relevant globals explicitly. 20 | 21 | https://github.com/web-platform-tests/wpt/issues/23111 tracks this work. 22 | 23 | ## Risks 24 | 25 | Downstream users could have reimplemented some of the infrastructure logic and end up running tests in more globals than they should (by always including the default globals). 26 | -------------------------------------------------------------------------------- /rfcs/assert_precondition.md: -------------------------------------------------------------------------------- 1 | # RFC #16: 'PRECONDITION_FAILED' subtest result (status) 2 | 3 | ## Summary 4 | 5 | Allow another distinct subtest result, different from `FAIL` or `MISSING` results, 6 | by asserting a precondition requirement for a subtest, where the precondition 7 | needs to be met for the test to actually produce any meaningful result. 8 | 9 | ## Details 10 | 11 | ### Background 12 | 13 | Some specs, such as [WebCryptoAPI](https://www.w3.org/TR/WebCryptoAPI/#scope-algorithms), 14 | focus on the discovery of the set of a user agent's implementations, but do not mandate 15 | any *particular* algorithm is implemented. As a result, it is perfectly spec compliant to 16 | omit any (or, even **all**!) implementations. 17 | 18 | With respect to testing, there are two options for results when enumerating the 19 | set of implementations: 20 | 21 | - Consider a lack of implementation a `FAIL` 22 | - Omit results for known non-implementations (`MISSING` on wpt.fyi) 23 | 24 | The current behaviour of [some WebCryptoAPI tests](https://wpt.fyi/results/WebCryptoAPI/encrypt_decrypt/aes_ctr.https.worker.html) actually has a mixture - some tests 25 | are missing, some are "failing". 26 | 27 | ### Proposal 28 | 29 | Add subtest result of `PRECONDITION_FAILED` - a result for subtests which 30 | assert an unmet precondition, and thus do not run the remainder of the test. 31 | 32 | Add an `assert_precondition(condition, description)` function helper, which will conclude 33 | with a `PRECONDITION_FAILED` result when `condition` is not truthy. 34 | 35 | > __assert_precondition(condition, description)__ 36 | > 37 | > Concludes the test with `PRECONDITION_FAILED` status if `condition` is not truthy. 38 | > Used for skipping subtests will not produce a meaningful result without the condition, 39 | > e.g. optional features that are not implemented by the user agent. 40 | 41 | #### Example Usage 42 | 43 | promise_test(function(test) { 44 | return subtle.generateKey(algorithm, extractable, usages) 45 | .then( 46 | function(result) { ... }, 47 | function(err) { 48 | var supported = !isUnsupported(err); // "Unsupported" case determined ad-hoc 49 | assert_precondition(supported, algorithm + ' not implemented'); 50 | assert_unreached("Threw an unexpected error: " + err.toString()); 51 | } 52 | }); 53 | } 54 | 55 | ## Advantages 56 | 57 | For spec-compliant implementation omission, it would allow distinction from other 58 | outcome implications. 59 | 60 | - `FAIL` implies an incorrect implementation 61 | - `MISSING` implies a test was not run at all 62 | - `PRECONDITION_FAILED` implies a test was not applicable 63 | 64 | It also allows for more expressive tests, allowing authors to be explicit about their 65 | expectations. 66 | 67 | ## Disadvantages 68 | 69 | - An extra outcome to consider 70 | - Will involve viral changes to logging, and log consumers 71 | - Possible metrics skewing from the point that tests change to use this status 72 | -------------------------------------------------------------------------------- /rfcs/assert_precondition_rename.md: -------------------------------------------------------------------------------- 1 | # RFC 48: Split 'assert\_precondition' into optional and non-optional variants 2 | 3 | ## Summary 4 | 5 | Split the `assert_precondition` function into `assert_implements_optional` and 6 | `assert_implements`. The `assert_implements_optional` function will behave as 7 | `assert_precondition` does today; a failed assert will record a 8 | 'PRECONDITION\_FAILED' status. The `assert_implements` function behaves 9 | similarly, but records a 'FAIL' status if the assert fails. 10 | 11 | Reuse the 'PRECONDITION\_FAILED' status rather than renaming it due to the 12 | technical difficulties of getting mozlog updated. 13 | 14 | ## Details 15 | 16 | ### Background 17 | 18 | The `assert_precondition` function and associated status 19 | ('PRECONDITION\_FAILED') were added in [RFC #16](assert_precondition.md). 20 | However there has been confusion over whether the function is meant for spec 21 | features that are explicitly marked OPTIONAL (e.g. its usage in 22 | `html/semantics/embedded-content/the-video-element/resize-during-playback.html`), 23 | or whether it is meant as an early-out when a particular spec or feature is not 24 | implemented by a browser (e.g. its usage in `portals/`). 25 | 26 | The difference matters here due to the 'PRECONDITION\_FAILED' status, which has 27 | been interpreted by some users as a 'non-failing' status - which only makes 28 | sense if the failure is for truly OPTIONAL behavior. 29 | 30 | ### Proposal 31 | 32 | Rename the `assert_precondition` function to `assert_implements_optional`, with 33 | the same 'PRECONDITION\_FAILED' status produced, and add a new function 34 | `assert_implements` that produces a 'FAIL' status if the input is not 35 | [truthy](https://developer.mozilla.org/en-US/docs/Glossary/Truthy). That is: 36 | 37 | > __assert_implements(condition, description)__ 38 | > 39 | > Concludes the test with a `FAIL` status if `condition` is not 40 | > [truthy](https://developer.mozilla.org/en-US/docs/Glossary/Truthy). 41 | > Used to avoid running unnecessary code or subtests for a feature that is not 42 | > implemented by the running browser. 43 | 44 | > __assert_implements_optional(condition, description)__ 45 | > 46 | > Concludes the test with a `PRECONDITION_FAILED` status if `condition` is not truthy. 47 | > Used for skipping tests or subtests for OPTIONAL features that that would otherwise 48 | > fail due to lack of support for the feature. 49 | 50 | The 'PRECONDITION\_FAILED' status is not being renamed, as changing the logging 51 | infrastructure (mozlog, etc) is logistically difficult. 52 | 53 | ## Risks 54 | 55 | * As with any change in testharness.js, downstream embedders may have to spend 56 | effort to adapt to the deprecation and rename. That said, there are currently 57 | zero uses of `assert_precondition` in downstream Chromium tests. 58 | * As it is not being renamed, the 'PRECONDITION\_FAILED' status may confuse 59 | consumers of WPT results. 60 | -------------------------------------------------------------------------------- /rfcs/assert_throws.md: -------------------------------------------------------------------------------- 1 | # RFC 29: `testharness.js`: Replace `assert_throws` and `promise_rejects` with more fine-grained APIs 2 | 3 | ## Summary 4 | 5 | The expected value passed to `assert_throws` and `promise_rejects` APIs can be one of several different kinds of values. 6 | This makes it harder to correctly compare the actual thrown exception / rejection value. 7 | Remove these APIs, and replace them with more focused APIs. 8 | 9 | ## Details 10 | 11 | The `assert_throws` and `promise_rejects` APIs verify the value that is thrown or passed as a rejection value. 12 | They support being called with three kinds of expected values: 13 | 14 | - a string or number representing a `DOMException` (e.g., `"INVALID_STATE_ERR"`, `"InvalidStateError"`, or `DOMException.INVALID_STATE_ERR`); 15 | - an ECMAScript-style `Error` object (e.g., `new TypeError()`); 16 | - a plain object (in practice, the exact same object should be thrown or used as the rejection value). 17 | 18 | The last two are handled the same way: only the "name" properties are compared between the expected and the actual value. 19 | Because of the conflation of the various cases, the existing APIs don't get any of them quite right, and this causes some browser bugs to be missed. 20 | For example, the following passes: 21 | 22 | ```js 23 | assert_throws(new TypeError(), () => { throw new DOMException("TypeError") }); 24 | ``` 25 | 26 | Six new APIs are proposed to handle each of the cases separately: 27 | 28 | - `promise_rejects_dom` and `assert_throws_dom`, called with a string representing a `DOMException` (as above); 29 | - `promise_rejects_js` and `assert_throws_js`, called with the Error constructor (e.g. `TypeError`); 30 | - `assert_throws_exactly` and `promise_rejects_exactly`, called with the expected value (which no longer needs to be an object) and using `Object.is`. 31 | 32 | The new APIs will be added first, in order to allow a well-organized transition. 33 | Once this is done and browser vendors have updated their copies of `testharness.js`, we can start working on a (mostly automated) conversion of the tests in wpt. 34 | The old API will only be removed once all callers have been removed from wpt and browser vendors have had a chance to update any callers in proprietary tests. 35 | 36 | In wpt, there are approximately 4200 calls to `assert_throws` (of which approximately 1200 in the generated canvas tests) and approximately 1400 to `promise_rejects`. 37 | 38 | ## Risks 39 | 40 | Updates to testharness.js haven't always been made together with the test updates in all browser engine repos. 41 | If this is still the case, the testharness.js changes have to be made first and synced downstream before any test changes are made. 42 | 43 | ## References 44 | 45 | This was discussed in [wpt issue #11726](https://github.com/web-platform-tests/wpt/issues/11726) and the implementation of the new APIs was done in [wpt PR #19054](https://github.com/web-platform-tests/wpt/pull/19054). 46 | -------------------------------------------------------------------------------- /rfcs/async_promise_test.md: -------------------------------------------------------------------------------- 1 | # RFC 75: async_promise_test() 2 | 3 | ## Summary 4 | 5 | Since `await`/`async` have been introduced into the language, many tests have 6 | switched to it. Tests are now easier to read and write. Events are happening in 7 | the same order as they are written in the test. No more callback to follow. 8 | 9 | `promise_tests()` instead of `async_tests()`/`step_func()`/... So far, so good! 10 | However this comes at a cost, `promise_tests()` are running in sequence, 11 | `async_tests()` in parallel. Running too many `promise_tests()` causes timeout. 12 | Developers are forced to split tests into many files, or switch back to 13 | `async_tests()` against their will. 14 | 15 | We would like a parallel version. This could be part of `testharness.js` or be 16 | put inside an helper script. 17 | 18 | There would be 4 kind of test. Promise or function based. Running in sequence or 19 | in parallel: 20 | 21 | | | in sequence | in parallel | 22 | | -------------| ----------: | ----------------: | 23 | | **Function** | test | async_test | 24 | | **Promise** | promise_test| async_promise_test| 25 | 26 | ## Details 27 | 28 | Some tests are defining it as: 29 | ```javascript 30 | // Test using the modern async/await primitives are easier to read/write. 31 | // However they run sequentially, contrary to async_test. This is the parallel 32 | // version, to avoid timing out. 33 | let async_promise_test = (promise, description) => { 34 | async_test(test => { 35 | promise(test) 36 | .then(() => {test.done();}) 37 | .catch(test.step_func(error => { throw error; })); 38 | }, description); 39 | }; 40 | ``` 41 | 42 | ## Risks 43 | 44 | This is a pure addition. The main risk is bad design and not being able to 45 | improve it later, if used massively. 46 | 47 | In particular, we should think more about how this will interact with other 48 | functions from `test_harness.js`, like `promise_setup()`. 49 | -------------------------------------------------------------------------------- /rfcs/asynchronous_setup.md: -------------------------------------------------------------------------------- 1 | # RFC 32: Asynchronous setup 2 | 3 | ## Summary 4 | 5 | Extend testharness.js with a new API that makes it easier for test authors to 6 | defer subtest execution until the completion of some asynchronous operation. 7 | 8 | ## Details 9 | 10 | Many testharness.js tests in WPT need to perform some asynchronous operation 11 | before subtests may be executed. For instance: 12 | 13 | - WebIDL tests need to fetch data (IDL files) 14 | - CSS tests need to wait for fonts to be ready 15 | - Service worker tests need to wait for worker registration 16 | 17 | Test authors can achieve this today (that is, without any extension to 18 | testharness.js) through three different approaches: 19 | 20 | 1. Explicitly waiting for the setup operation to complete from every subtest 21 | 2. Performing the setup operation in a dedicated `promise_test` which is 22 | defined before all others 23 | 3. Enabling the `explicit_done` option, perform the setup operation, define the 24 | tests, and finally invoke the global `done` function 25 | 26 | However, each of these approaches has subtly different semantics and each 27 | requires additional code which obscures the intent of the test and increases 28 | the probability of bugs. 29 | 30 | A new API would improve consistency of results, reduce boilerplate, and reduce 31 | the associated risk of subtle errors. 32 | 33 | Introduce a new global function named `promise_setup` which implements the same 34 | signature as the existing global function named `setup`. This function should: 35 | 36 | 1. If no function is provided: 37 | 1. Set the harness status to `ERROR` 38 | 2. Transition the harness to the `COMPLETE` phase 39 | 3. Abort this algorithm 40 | 2. Apply any harness configuration parameters provided 41 | 3. Invoke the provided function and capture the result 42 | 4. If the previous step produces an exception or if the result is not a 43 | "thenable" object: 44 | 1. Set the harness status to `ERROR` 45 | 2. Transition the harness to the `COMPLETE` phase 46 | 3. Abort this algorithm 47 | 5. Configure testharness.js to defer execution of any tests subsequently 48 | defined with `promise_test` 49 | 6. Configure the global `test` function and the global `async_test` function to 50 | perform the following steps upon invocation: 51 | 1. Set the harness status to `ERROR` 52 | 2. Transition the harness to the `COMPLETE` phase 53 | 7. Wait for the result of the provided function to settle or for the amount of 54 | time equal to the internal harness timeout to pass (whichever comes first) 55 | 8. If the result of the provided function has not settled: 56 | 1. Set the harness status to `TIMEOUT` 57 | 2. Transition the harness to the `COMPLETE` phase 58 | 3. Abort this algorithm 59 | 9. If the result of the provided function is rejected: 60 | 1. Set the harness status to `ERROR` 61 | 2. Transition the harness to the `COMPLETE` phase 62 | 3. Abort this algorithm 63 | 10. Begin the serial execution of any tests which were defined with 64 | `promise_test` while this algorithm waited 65 | 66 | These steps are intended to approximate the corresponding synchronous behavior 67 | of the `setup` function. 68 | 69 | ## Risks 70 | 71 | Increasing the size of the API makes the harness more difficult to learn, and 72 | this may discourage new contributors. 73 | 74 | ## Alternatives considered 75 | 76 | Overload `setup` - extend the existing `setup` function to implement the 77 | algorithm described above only if the provided function returns a "thenable" 78 | object (and to preserve its existing behavior otherwise). This style of API is 79 | difficult to discover and error prone. 80 | -------------------------------------------------------------------------------- /rfcs/bigint_support.md: -------------------------------------------------------------------------------- 1 | # RFC 222: Support BigInts in numerical assertion functions and output 2 | 3 | ## Summary 4 | 5 | Allow BigInt values (e.g. `1234n`) in `assert_less_than()` and friends. 6 | 7 | 8 | ## Details 9 | 10 | https://github.com/web-platform-tests/wpt/issues/51331 11 | 12 | Background: BigInts are a relatively new fundamental numeric type in JavaScript that allow for arbitrary size integers. They are commonly used for 64-bit signed and unsigned integers (e.g. in conjunction with `BigInt64Array` and `BigUint64Array`) but are not limited to that range. Literals are written with an `n` suffix (e.g. `let my_num = 1234n;`, and the `BigInt()` function can be used to "cast" another value. They interoperate transparently with regular JavaScript Numbers in many cases, but are a distinct type: `typeof 1234n === 'bigint'`. 13 | 14 | The following assertion functions will be changed to support passing BigInts as the expected and actual inputs: 15 | 16 | * `assert_less_than()` 17 | * `assert_greater_than()` 18 | * `assert_less_than_equal()` 19 | * `assert_greater_than_equal()` 20 | * `assert_between_exclusive()` 21 | * `assert_between_inclusive()` 22 | 23 | Specifically: 24 | * The functions are updated to allow either Number or BigInts to be passed, instead of failing if anything other than a Number is passed. 25 | * For the `between` functions, the upper and lower bounds must be the same type. The test itself is considered at fault if the types don't match. 26 | * The assertions fail if the type of the actual value does not match the type of the expected value. 27 | 28 | Assertion functions for "approximate" comparisons (`assert_approx_equals()` and `assert_array_approx_equals()`) that take an epsilon value will *not* be modified by this RFC. 29 | 30 | In addition, the internal `format_value()` used when formatting messages such as assertion failures will be updated to output BigInt values with the `n` suffix, e.g. `1234n`. Currently it switches on the `typeof` the value and falls through to the default case, which outputs `bigint "1234"`. 31 | 32 | WIP implementation PR: https://github.com/web-platform-tests/wpt/pull/51919 33 | 34 | 35 | ## Risks 36 | 37 | Any change to `testharness.js` can affect error stacks. This could cause a mismatch in "expectation" files maintained by implementations and failures in their CIs, requiring baseline updates. This is probably unavoidable maintenance burden for such tests. Here is one example in a prototype update, in the Chromium repository: https://chromium-review.googlesource.com/c/chromium/src/+/6351358/6/third_party/blink/web_tests/external/wpt/html/infrastructure/safe-passing-of-structured-data/structured-cloning-error-stack-optional.sub.window-expected.txt 38 | 39 | The change to `format_value()` affects the output of any existing WPTs that output BigInts. This could cause a mismatch in "expectation" files maintained by implementations and failures in their CIs, requiring baseline updates. From inspection, there are no obvious cases. 40 | 41 | Requiring the types of the expected and actual values to match could potentially confuse test authors, as JavaScript allows writing `123n > 456`. 42 | -------------------------------------------------------------------------------- /rfcs/check_layout_wait_for_fonts.md: -------------------------------------------------------------------------------- 1 | # RFC 213: Wait for `document.fonts.ready` before running `checkLayout()` tests 2 | 3 | ## Summary 4 | 5 | Change [`checkLayout()`][check-layout-th] to wait for 6 | [`document.fonts.ready`][fonts-ready] to fulfill before running tests. 7 | 8 | [check-layout-th]: https://github.com/web-platform-tests/wpt/blob/9952f6fa233ad9e067b2ff28ab32832cc6d8fd66/resources/check-layout-th.js#L211 9 | [fonts-ready]: https://developer.mozilla.org/en-US/docs/Web/API/FontFaceSet/ready 10 | 11 | ## Details 12 | 13 | ### Motivation 14 | 15 | https://github.com/web-platform-tests/rfcs/pull/22 removed the requirement that 16 | [Ahem] is a system font. 17 | Since then, tests must load Ahem as a web font before use. 18 | 19 | testharness.js runs [synchronous `test()` functions][test] as soon as they're 20 | registered, which can be before Ahem and other necessary web fonts are loaded. 21 | To avoid testing against a layout with a default font in this scenario, [many 22 | tests] wait for `document.fonts.ready` to fulfill first. 23 | However, test authors can still forget to use this boilerplate ([example 1], 24 | [example 2]), especially since a cached or system-installed font can mask the 25 | underlying flakiness. 26 | 27 | [Ahem]: https://web-platform-tests.org/writing-tests/ahem.html 28 | [test]: https://web-platform-tests.org/writing-tests/testharness-api.html#test 29 | [many tests]: https://github.com/search?q=repo%3Aweb-platform-tests%2Fwpt+%2Fdocument.fonts.ready.*%28checkLayout%7Ctest%29%2F&type=code 30 | [example 1]: https://crrev.com/c/5943572 31 | [example 2]: https://crrev.com/c/5889804 32 | 33 | ### Proposed Changes 34 | 35 | * [`check-layout-th.js`][check-layout-th] is a helper script for generating 36 | layout-related `test()`s, which should generally run against a stable final 37 | layout. 38 | Therefore, in `check-layout-th.js`, change `setup(...)` to 39 | `promise_setup(() => document.fonts.ready, ...)` and `test(...)` to 40 | `promise_test(async ...)` accordingly. 41 | The `promise_test()`s [wait for `promise_setup()`][promise-setup], but will still run 42 | sequentially in the order in which they were registered. 43 | * Clean up obsolete [`document.fonts.ready.then(() => checkLayout(...))`][many 44 | tests] and similar ad hoc patterns. 45 | * To reflect WPT's minimum requirements, stop running tests in 46 | web-platform-tests/wpt Taskcluster/Azure with 47 | [`--install-fonts`][install-fonts]. 48 | Without system fonts, stability checks will be more likely to block tests 49 | that don't wait for web fonts. 50 | 51 | [promise-setup]: https://web-platform-tests.org/writing-tests/testharness-api.html#promise_setup 52 | [install-fonts]: https://github.com/web-platform-tests/wpt/blob/38c69461a2178d1c9fa80a948e393612e0220c95/tools/wptrunner/wptrunner/wptcommandline.py#L273 53 | 54 | ### Alternatives Considered 55 | 56 | Improving `checkLayout()` alone might not suffice, since [tests that use 57 | `test()` directly][example 2] might still run with incorrect fonts. 58 | However, delaying `test()` functions to run after `document.fonts.ready` 59 | fulfills likely breaks [~100s of tests] that [assume the existing eager 60 | scheduling behavior]. 61 | 62 | [~100s of tests]: https://chromium-review.googlesource.com/c/chromium/src/+/5980648?checksPatchset=2&tab=checks 63 | [assume the existing eager scheduling behavior]: https://github.com/web-platform-tests/wpt/blob/f26694cede6d5602f01e7f7762876ec43cf74951/css/css-counter-styles/counter-style-at-rule/support/counter-style-testcommon.js#L20-L44 64 | 65 | ## Risks 66 | 67 | https://crrev.com/c/5979634/1 tests a prototype of this RFC in Chromium CI: 68 | * Fewer than 10 tests register additional `test()`s after `checkLayout()`, 69 | which is not allowed if `checkLayout()` registers `promise_test()`s instead. 70 | This is trivially fixed by applying the same `test(...)` to 71 | `promise_test(async ...)` transformation. 72 | * Other regressions/flakiness due to brittle `checkLayout()` tests relying on 73 | the existing scheduling behavior seem unlikely. 74 | -------------------------------------------------------------------------------- /rfcs/code_of_conduct.md: -------------------------------------------------------------------------------- 1 | # RFC #54: Code of Conduct 2 | 3 | ## Summary 4 | 5 | Introduce a code of conduct which clearly defines the acceptable behavior for 6 | all participants in the WPT project and includes contact information for a 7 | group of individuals who can be trusted to enforce the policy. 8 | 9 | Publishing a code of conduct is a way "to be overt in our openness, welcoming 10 | all people to contribute, and pledging in return to value them as whole human 11 | beings and to foster an atmosphere of kindness, cooperation, and 12 | understanding." (via 13 | [contributor-covenant.org](https://www.contributor-covenant.org/)) 14 | 15 | ## Details 16 | 17 | This RFC consists of two parts: ratifying the content in the [proposed 18 | `CODE_OF_CONDUCT.md`](https://github.com/web-platform-tests/wpt/pull/23762) 19 | file, and setting down guidelines for the moderation team (which exist within 20 | this RFC). Accepting this RFC into WPT means accepting both of these. 21 | 22 | Note that the content below are explicitly guidelines for, not requirements on, 23 | the moderation team. That said, consistent violation of these guidelines would 24 | either be a reason to amend this RFC to correct the guidelines, or a reason for 25 | the WPT Core Team to intercede in moderation. 26 | 27 | ### The Moderation Team 28 | 29 | The members of the moderation team will be listed in the `CODE_OF_CONDUCT.md` 30 | file. There is no specified size or make-up of the team, but we aim to have a 31 | moderation team which is able to - in most cases - keep itself accountable. 32 | This implies having multiple independent members. 33 | 34 | ### Handling Incidents/Complaints 35 | 36 | The actual actions taken when handling a violation are documented in the 37 | 'Moderation' section of the [proposed Code of 38 | Conduct](https://github.com/web-platform-tests/wpt/pull/23762), however this 39 | section lays out some related guidelines. 40 | 41 | When handling an incident or complaint, moderators should: 42 | 43 | * Respond as promptly as possible to reports of violations. 44 | * When enforcing a moderation decision, notify the person who has been moderated explaining: what rule was violated, what action has been taken, and what to expect next. 45 | * Recuse themselves or be excused from handling an incident if they have any conflict of interests. 46 | * Keep each other accountable. 47 | 48 | ## Risks 49 | 50 | The burden of moderation is not to be taken lightly. If a volunteer is unable 51 | to perform this service, then they will undermine the trust of the people the 52 | code is intended to help. We need to be certain that everyone who volunteers is 53 | prepared to enforce. 54 | -------------------------------------------------------------------------------- /rfcs/conditional_requirements.md: -------------------------------------------------------------------------------- 1 | # Conditional requirements support in wptrunner 2 | 3 | ## Summary 4 | 5 | Add a new field `conditional_requirements` in commands.json so that conditional 6 | requirements can be specified for `wpt` subcommands. 7 | 8 | ## Details 9 | 10 | ### Context 11 | 12 | `wpt` uses commands.json to set up subcommands (see `load_commands()` in 13 | tools/wpt/wpt.py). A subcommand can specify 14 | [requirements.txt](https://pip.pypa.io/en/stable/user_guide/#requirements-files) 15 | to install its dependencies in a virtualenv environment. These dependencies are 16 | unconditionally installed before running the subcommand. 17 | 18 | There is a demand to install additional dependencies conditionally. For example, 19 | supporting conditional requirements can be helpful in the following situation: 20 | 21 | * A subcommand supports a command line flag to enable a specific feature. 22 | * The feature needs third party libraries. 23 | * These libraries also have dependencies. 24 | * The feature is experimental and some browser testing infrastructures may not 25 | want to introduce such dependencies yet. 26 | 27 | A specific example of the above situation is 28 | [WebTransport over HTTP/3 support](https://github.com/web-platform-tests/rfcs/blob/master/rfcs/webtransport_h3_test_server.md). 29 | The WebTransport over HTTP/3 server requires 30 | [aioquic](https://aioquic.readthedocs.io/en/latest/) and it also has several 31 | dependencies. 32 | 33 | ```sh 34 | # This doesn’t require `aioquic` library 35 | $ ./wpt run 36 | # This requires `aioquic` library 37 | $ ./wpt run --enable-webtransport-h3 38 | ``` 39 | 40 | This RFC proposes the `conditional_requirements` field in commands.json to 41 | support conditional requirements like the above example. 42 | 43 | ### commands.json addition 44 | 45 | A value of `conditional_requirements` would look like the following: 46 | 47 | ```json 48 | { 49 | "conditional_requirements": { 50 | "commandline_flag": { 51 | "enable_webtransport_h3": [ 52 | "../webtransport/requirements.txt" 53 | ] 54 | } 55 | } 56 | } 57 | ``` 58 | 59 | A `conditional_requirements` contains key value pairs. This RFC only defines 60 | `commandline_flag` as a key. `commandline_flag` is examined when the 61 | corresponding command line flag is provided. In the above example, `wpt` 62 | installs requirements in `../webtransport/requirements.txt` only when the 63 | `--enable-webtransport-h3` command line flag is provided for the subcommand. 64 | 65 | ### Alternatives considerered 66 | 67 | Conditional requirements support introduces complexity to some extent. 68 | Installing all potential dependencies unconditionally could be an alternative 69 | approach. The upside of the alternative approach is that we keep `wptrunner` as 70 | simple as possible. The downside is that installing unnecessary dependencies can 71 | also be a source of the maintenance burden. 72 | 73 | Conditional requirements support gives us a way to take an incremental approach 74 | to introduce dependencies suitable for each testing infrastructure. 75 | 76 | ### Follow ups 77 | 78 | This proposal could in theory generalize it to allow supporting the wptrunner 79 | use case for installing per-product requirements. It might look like: 80 | 81 | ```json 82 | { 83 | "conditional_requirements": { 84 | "product": { 85 | "chrome": [ 86 | "requirements_chrome.txt" 87 | ] 88 | } 89 | } 90 | } 91 | ``` 92 | 93 | ## Risk 94 | 95 | Adding more fields in `commands.json` makes `wpt` complicated to execute 96 | subcommands and may lead to increasing maintenance costs. We improve 97 | documentation and describe how `commands.json` is used. 98 | -------------------------------------------------------------------------------- /rfcs/consume_user_activation.md: -------------------------------------------------------------------------------- 1 | # RFC 128: Consume user activation 2 | 3 | Editor: Marcos Caceres, Apple Inc. 4 | 5 | ## Summary 6 | 7 | Tests in WPT are currently relying on various indirect means to "[consume user activation](https://html.spec.whatwg.org/#consume-user-activation)" [HTML]. 8 | Some of these means are non-standard in as far as they are not specified to consume user activation, meaning that the tests are inshrining non-standard behavior for conformance purposes. 9 | 10 | The workarounds being used are as follows, with several drawbacks: 11 | 12 | * Using Fullscreen API: it takes significant time to put an element into fullscreen. 13 | * Using window.open() and window.close() - non-standard, not mobile friendly. 14 | * Using Payment Request API - not widely implemented (e.g., not exposed in Gecko). 15 | 16 | We also looked at a range of other possible APIs that consume user activation, and found [none to be suitable](https://github.com/web-platform-tests/wpt/issues/36727#issuecomment-1296349964). 17 | 18 | ## Details 19 | 20 | We would like to propose the addition of an async function `test_driver.consume_user_activation()` method that returns a `Promse` (representing if the activation was consumed or not). For example: 21 | 22 | ``` 23 | const consume = await test_driver.consume_user_activation(); 24 | ``` 25 | 26 | The `consume_user_activation()` method can take a `Window` context (which defaults to the current Window object). Allowing consumption to happen at a particular window, if required. 27 | 28 | The `consume_user_activation()` method would be implemented via the proposed addition of “[Consume user activation of Window](https://github.com/w3c/webdriver/pull/1695)” to the Web Driver specification. 29 | 30 | This prposal has several advantages: 31 | * it's fast - no opening windows or waiting for fullscreen to enter/exit. 32 | * it's lightweight - it doesn't create new browsing contexts. 33 | * it's mobile friendly - as above. 34 | * it's build for purpose - no more using other APIs to indirectly achieve the desired outcome. 35 | * It's really simple - it complements `.bless()` and other user activation functionality already available. 36 | 37 | [An implementation](https://github.com/WebKit/WebKit/pull/6539) is available in WebKit. 38 | 39 | ## Risks 40 | 41 | None known. 42 | -------------------------------------------------------------------------------- /rfcs/crash_test.md: -------------------------------------------------------------------------------- 1 | # RFC 33: Crash Tests 2 | 3 | ## Summary 4 | 5 | Add a test type in which the test document is simply loaded. The test 6 | passes if this occurs without encountering a fatal error e.g. a crash. 7 | 8 | ## Details 9 | 10 | Crash Tests are static tests typically used for cases where browsers 11 | are known to have a bug such as a crash or a leak. They differ from 12 | other tests such as reftests and testharness tests in that there is no 13 | pass condition other than the page loading successfully. In vendor 14 | infrastructure these tests may also be run with additional 15 | instrumentation e.g. address sanitizing, making the pass condition 16 | more stringent. The lack of an explicit pass condition makes it easier 17 | to write these tests than tests where a detailed understanding of the 18 | correct behaviour is required, making them suitable for autogeneration 19 | e.g. as the output of a fuzzing tool. 20 | 21 | The infrastructure changes are as follows: 22 | 23 | * A new test type `crash` recognised by the manifest and assigned to 24 | any test file with a name containing `-crash` immediately before 25 | the extension or in a subdirectory called `crashtests`. 26 | 27 | * A new executor type with the following behaviour: 28 | - Load the test URL 29 | - If there is no `class=test-wait` on the root element, mark the test 30 | as passed when the load is complete, all fonts have finished 31 | loading, and the rendering is stable. 32 | - If `class=test-wait` appears on the root element, mark the test as 33 | complete once the previous conditions are met and that class has been 34 | removed. 35 | 36 | ## Risks 37 | 38 | Adding this test type may encourage test authors to write crashtests 39 | for cases where a more useful cross-browser test would have a real 40 | pass condition. 41 | 42 | Crashes and other issues caught by these tests may not be sufficiently 43 | common between browsers to make sharing these tests a good use of 44 | resources. However there's data showing cases where a tests that crash 45 | one browser also crash others. Therefore this concern seems less 46 | serious than it may at first appear. 47 | -------------------------------------------------------------------------------- /rfcs/crash_test_navigate.md: -------------------------------------------------------------------------------- 1 | # RFC 193: Navigate after Crash Tests 2 | 3 | ## Summary 4 | 5 | Navigate to about:blank after successfully loading the test URL in 6 | crash tests. 7 | 8 | ## Details 9 | 10 | Add a step to the end of crash tests where the browsing context that 11 | loaded the test is navigated to about:blank. 12 | 13 | This is because we have seen cases where the crash the test is trying 14 | to detect doesn't happen until a subsequent navigation. That kind of 15 | crash is hard to handle in web-platform-tests: if the test happens to 16 | be the last one in a group it may be missed entirely, and because we 17 | want to make it possible to import tests without blocking on fixing 18 | the bugs they reveal, it's important to be able to annotate a crash as 19 | related to a specific test so the expectation metadata can be set 20 | correctly; if the actual crash happens at the start of the following 21 | test this doesn't work because we've already recorded an incorrect 22 | PASS status for the previous test. It's also helpful for humans to see 23 | the test that's actually failing when debugging the problem. 24 | 25 | ## Risks 26 | 27 | This will decrease the performance of crash tests. However since 28 | they're few in number and generally fast compared to other test types, 29 | this seems like an acceptable tradeoff. 30 | 31 | Since about:blank is a special URL it's possible that navigating to 32 | that is triggering different codepaths to navigating to an actual 33 | server-provided resource. We could fix this by navigating to a blank 34 | page on the server, but this would likely have even worse performance 35 | implications. We should consider this option if we find there are 36 | cases not caught by this approach that would be caught when navigating 37 | to a http URL. 38 | 39 | This won't cover all possible cases of delayed crashes; in general 40 | that's an impossible problem to solve. However it is an incremental 41 | improvement over the status quo. 42 | -------------------------------------------------------------------------------- /rfcs/create-new-test-editor-repo.md: -------------------------------------------------------------------------------- 1 | # RFC #24 - Add the Editor repo 2 | 3 | ## Summary 4 | This repo allows developers to create and edit tests to be consumed by the WPT test infrastructure. 5 | 6 | ## Details 7 | Originally, this repo was built by Microsoft to help facilitate debugging and authoring of tests for web-platform-tests. 8 | 9 | It's been living under the MicrosoftEdge repo up until now, but we believe the community will benefit from this tool. 10 | 11 | By moving it to the Web-Platform-Tests github org, the people who are directly working with WPT will have access and context. 12 | 13 | The repo would live [here](https://github.com/web-platform-tests/editor). 14 | 15 | ## Risks 16 | This repo was created by an ex-Microsoft employee. Microsoft's current expertise in this repo is limited, so there is some risk regarding maintenance. 17 | 18 | Tab Atkins and John Jansen have agreed to dedicate some time to maintain it. We can work as a team should we discover there is more work necessary than 19 | what the two of them can handle. 20 | -------------------------------------------------------------------------------- /rfcs/css_build_system.md: -------------------------------------------------------------------------------- 1 | # RFC 136: Remove the CSS build system 2 | 3 | ## Summary 4 | 5 | Remove CSS build system in `css/tools/` and the path lint rules for `css/` that support that build system. 6 | 7 | ## Details 8 | 9 | http://test.csswg.org/suites/ hasn't been updated in some time, which led to a [discussion](https://github.com/w3c/csswg-drafts/issues/6896) in the CSS Working Group about how to fix it. Ultimately, the CSS Working Group [resolved](https://github.com/w3c/csswg-drafts/issues/6896#issuecomment-1499457107) to remove the now unused CSS build system, and this RFC implements that decision. 10 | 11 | These lint rules will be removed: 12 | 13 | - CSS-COLLIDING-TEST-NAME ("The filename ... in the ... testsuite is shared by: ...") 14 | - CSS-COLLIDING-REF-NAME ("The filename ... is shared by: ...") 15 | - CSS-COLLIDING-SUPPORT-NAME ("The filename ... is shared by: ...") 16 | - SUPPORT-WRONG-DIR ("Support file not in support directory") 17 | 18 | The MISSING-LINK ("Testcase file must have a link to a spec") rule which also applies only to `css/` will _not_ be removed, as it isn't only used by the build system, but is also a matter of style and preference. This may be revisited later. 19 | 20 | This RFC will be implemented by https://github.com/web-platform-tests/wpt/pull/38976. 21 | 22 | ## Risks 23 | 24 | Reviving http://test.csswg.org/suites/ becomes harder with this removal, so if that system has important functionality, it will require more work to meet those needs again. However, the web-platform-tests project remains open to contributions to support missing use cases, should the need arise. 25 | -------------------------------------------------------------------------------- /rfcs/disable_chrome_stability.md: -------------------------------------------------------------------------------- 1 | # RFC 131: Disable wpt-chrome-dev-stability runs 2 | 3 | ## Summary 4 | 5 | The wpt-chrome-dev-stability job on WPT fails quite frequently and 6 | it's impacting developer productivity. The job should be changed to 7 | non-blocking, while keeping a record of potential flakiness 8 | introductions that could be useful in further investigations. 9 | 10 | ## Details 11 | 12 | Stability jobs are intended to help us ensure that new and modified 13 | tests aren't flaky. In particular it's designed to ensure that the 14 | test author is aware when they have written a test that is flaky in a 15 | different browser. This is because authors writing a test in a 16 | specific browser engine might be unaware that they are depending on 17 | details of that implementation that make the test unreliable in other 18 | engines. Leaving this to other developers to fix is expensive, because 19 | they have to spend time becoming familiar with details of the test 20 | which are already understood by the original author. In addition, when 21 | there are many unstable web-platfrom-tests, it undermines confidence 22 | in the overall suite and, where such instability is automatically 23 | handled in CI systems (e.g. by marking a test as [PASS, FAIL] in 24 | metadata), reduces the ability of the tests to catch regressions. 25 | 26 | Currently, stability checks for the corresponding browser are skipped 27 | when [merging export PRs from their own 28 | repositories](https://github.com/web-platform-tests/wpt/issues/29737) 29 | (i.e. Chromium exports don't run Chrome stability checks, and 30 | likewise for Firefox). This is because we assume that any real 31 | stability issue is likely to have been handled in the source 32 | repository. It's common for such changesets to include code changes 33 | to the browser itself that fix intermittents, and in the source 34 | repository the tests will run together with the corresponding browser 35 | changes. However on GitHub we are using the latest development 36 | release of the browser, which is unlikely to contain code fixes at 37 | the time of export. This makes the stability checks on these PRs 38 | highly prone to misleading failures. 39 | 40 | The main problem with the stability checks as currently implemented is 41 | that although the test author is in the best place to understand the 42 | behaviour of the test, they may not understand how to reproduce the 43 | failure in another browser, or may be unable to fix the problem 44 | (e.g. if it's a browser bug rather than a test bug). 45 | 46 | Faced with these tradeoffs, the Chromium developers believe the 47 | project is better served by allowing tests which are unstable in 48 | Chrome to land in web-platform-tests and using the tooling they have 49 | available in the Chromium CI system to investigate the 50 | flakiness. Therefore the job will be marked as non-blocking, so we are 51 | able to see where it fails and assess whether the correct tradeoff has 52 | been made. 53 | 54 | Marking a job as non-blocking will require changes in the taskcluster 55 | configuration so that the job does not block the sink task. 56 | 57 | Disabling the job entirely would also reduce the CI load. This can be 58 | considered once the impact of the change is well understood. 59 | 60 | ## Risks 61 | 62 | New intermittent failures specific to Chromium could be introduced 63 | upstream and only noticed when a WPT import to Chromium CI 64 | happens. Chromium developers believe the tradeoff is worthwhile, as 65 | the Chromium project has robust tooling to investigate and handle such 66 | cases. 67 | -------------------------------------------------------------------------------- /rfcs/docker_github_registry.md: -------------------------------------------------------------------------------- 1 | # RFC 198: Move the container image from Docker Hub to GitHub Registry 2 | 3 | ## Summary 4 | 5 | This RFC proposes moving the container image from Docker Hub to GitHub Registry. 6 | 7 | ## Details 8 | 9 | This RFC addresses the issue 10 | [#28903](https://github.com/web-platform-tests/wpt/issues/28903). Docker Hub 11 | organization allows for a limited number of members and only a few members can 12 | publish a new Docker image that sometimes results in the Docker image that is 13 | used in taskcluster being outdated as documented in [this PR](https://github.com/web-platform-tests/wpt/pull/44576#issue-2133724788). 14 | 15 | The following are the benefits of using the GitHub registry for hosting: 16 | 17 | - It is [hosted on GitHub](https://github.com/web-platform-tests/wpt/packages) together with the wpt repository. 18 | - Publishing is possible [via GitHub Actions](https://github.com/web-platform-tests/wpt/blob/master/.github/workflows/docker.yml): 19 | 20 | - The workflow can be manually invoked, 21 | - or it can be further automated to deploy any changes that land. 22 | 23 | - We can take the opportunity to change the versioning scheme to use integer numbers. 24 | 25 | The following PR implements changes to use the image from GitHub registry 26 | instead of the Docker Hub one 27 | https://github.com/web-platform-tests/wpt/pull/46279 including modifications to 28 | tools/docker/frontend.py. 29 | 30 | This RFC also proposes to change the version numbers to be integers starting 31 | with `1`. 32 | 33 | ### Permissions 34 | 35 | The following permissions shall apply to publishing of the image and Dockerfile changes: 36 | 37 | * Registry permissions (**this part needs to be configured by the project admin on [this page](https://github.com/orgs/web-platform-tests/packages/container/wpt/settings)**): 38 | * Default inheritance of permissions from the repository should be disabled and the access needs to be provided to: 39 | * The [admin](https://github.com/orgs/web-platform-tests/teams/admins) team 40 | * The [core](https://github.com/orgs/web-platform-tests/teams/wpt-core-team) team 41 | * the github workflows ([docs](https://docs.github.com/en/packages/learn-github-packages/about-permissions-for-github-packages#maintaining-access-to-packages-in-github-actions-workflows)) 42 | 43 | * `Dockerfile` and `.taskcluster.yml` changes 44 | * Use CODEOWNERS to require reviews for changes in these files from the admins and core teams. 45 | 46 | ### Download time 47 | 48 | We measured the download time for Docker Hub and Github from a TaskCluster job 49 | and didn't see any regression (both jobs took approx ~48 seconds). 50 | 51 | ## Risks 52 | 53 | Users of the current image hosted at Docker Hub would need to update the image 54 | name from `webplatformtests/wpt:0.57` to `ghcr.io/web-platform-tests/wpt:1` (or 55 | whatever is the latest version). The GitHub Registry image is publicly available 56 | like the Docker Hub one but there could be some friction with the update 57 | depending on the user's setup. -------------------------------------------------------------------------------- /rfcs/expected_fail_message_tag_for_subtests.md: -------------------------------------------------------------------------------- 1 | # RFC 165: Support expected-fail-message in test metadata for subtests 2 | 3 | ## Summary 4 | 5 | Add "expected-fail-message" in test metadata so that we can track how 6 | a subtest fails. 7 | 8 | ## Details 9 | 10 | In Chromium developers are using [baselines](https://source.chromium.org/chromium/chromium/src/+/main:third_party/blink/web_tests/external/wpt/editing/event-expected.txt) to track expected output message 11 | for testharness tests. While the output message for a pass subtest is 12 | generally not interesting, the output message for a failed subtest tracks 13 | how a subtest fails, and a change in how a test fails could be unintended 14 | and suggestive of a problem. Chromium developers want to keep such capability 15 | if we want to use Wptrunner to run WPTs in Chromium. 16 | 17 | The suggestion is to support "expected-fail-message" tag for subtests. When 18 | present, it takes effect on expected FAIL or expected PRECONDITION_FAILED 19 | results. The test runner should check if the actual output message matches 20 | with the "expected-fail-message". When a mismatch is found, the test runner 21 | should turn the results to unexpected FAIL or unexpected PRECONDITION_FAILED 22 | respectively, otherwise the results continue to be the expected one. 23 | 24 | Wdspec tests can also have subtests. "expected-fail-message" should work the 25 | same way for an expected FAIL Wdspec subtest. Other than that, "expected-fail-message" 26 | should not have impact on any other scenarios. 27 | 28 | One example metadata file will look like below: 29 | ``` 30 | [event.html] 31 | expected: OK 32 | [Changing selection from handler: input event] 33 | expected: FAIL 34 | 35 | [Command createLink, value "": input event] 36 | expected: [FAIL, PRECONDITION_FAILED, PASS] 37 | expected-fail-message: "number of input events fired expectei 1 but got 0" 38 | ``` 39 | 40 | ### Implementations 41 | 42 | The current implementation for test metadata parser is flexible enough to 43 | understand any additional tag. The changes needed is 1) wpttest.Test to have one 44 | additional member function to return "expected-fail-message", 45 | 2)testrunner.TestRunnerManager.test_ended to have additional logic to turn 46 | expected FAIL or PRECONDITION_FAILED to unexpected failures per the rule 47 | discribed above. 48 | 49 | How to add "expected-fail-message" to test metadata is out of scope of this RFC. 50 | Usually this tag can be added manually by developers after carefully review the 51 | test results and implementations. 52 | 53 | ### Alternatives Considered 54 | 55 | Various alternatives have been considered, e.g. writing a new test to assert the 56 | behavior a baseline wants to assert, or duplicate the test in question as a 57 | legacy web test so that baseline will continue work. Those all would be a 58 | maintanance headache, and are rejected consequently. 59 | 60 | ## Risks 61 | 62 | "expected-fail-message" will only take effect when it presents in the metadata 63 | file. When not presented, the test will behave exactly the same way as before. 64 | It would be at developers' own discretion to use this feature or not. There is a 65 | chance this could be abused, but as the metadata file sits in each vendor's 66 | repository, the impact should be limited. 67 | 68 | The existence of this feature should not be used as an implication that subtest 69 | output messages are expected to be stable. When a subtest output message is found 70 | to be not stable, "expected-fail-message" should simply not be used, or if a 71 | vendor still want to assert on the output message, they can choose to revise 72 | the test without sacrifice the usefulness of the test. The original test author 73 | bears no responsibility of this. 74 | 75 | This feature will not have any impact to the results presented at wpt.fyi, as 76 | this only changes if a FAIL is expected or not, but not the actual result 77 | itself. 78 | -------------------------------------------------------------------------------- /rfcs/fetch_json_et_al.md: -------------------------------------------------------------------------------- 1 | # RFC 177: `fetch_json` and other `ShadowRealm` facilities 2 | 3 | ## Summary 4 | 5 | This follows up [RFC 107][rfc-107]. 6 | [`ShadowRealm`s][proposal] are a new sandboxing primitive, currently in 7 | the process of being integrated into relevant Web specifications. 8 | `ShadowRealm` environments resemble vanilla JS shell environments, but 9 | do have some Web interfaces available, such as `URL`. 10 | 11 | I've submitted a [pull request][pr] with some changes to the 12 | test harness. 13 | The goal is to more easily allow existing WPT test files to run in 14 | `ShadowRealm` contexts, without writing separate duplicated tests for 15 | `ShadowRealm`. 16 | 17 | The most notable change is a `fetch_json` function to allow importing 18 | test data from JSON files across the sandboxing boundary. 19 | 20 | ## Details 21 | 22 | The [pull request][pr] comprises four commits, which I'll explain in 23 | detail in this section. 24 | 25 | ### Document self.GLOBAL.isShadowRealm() 26 | 27 | This commit is just an improvement to the documentation. 28 | 29 | ### Add fetch_json() test harness API 30 | 31 | A common pattern for importing additional test data from a JSON file 32 | looks like this: 33 | 34 | ```js 35 | promise_test(() => fetch("resources/mydata.json").then(res => res.json()).then(run_tests), "Loading data…"); 36 | ``` 37 | 38 | `ShadowRealm` scopes do not have the `fetch()` API available. 39 | Additionally, the sandboxing only allows primitive or callable values 40 | to be passed into or out of the sandbox, not other objects. 41 | Therefore, this approach isn't feasible for `ShadowRealm` scopes. 42 | 43 | The proposed `fetch_json()`, in non-`ShadowRealm` scopes, is just a 44 | shorthand for the above: call `fetch()`, and then call the returned 45 | `Response` object's `json()` method. 46 | 47 | Inside `ShadowRealm` scopes, it is overwritten by a function that calls 48 | the host realm's `fetch()`, passes the response in string form across 49 | the sandbox boundary, and then inside the sandbox turns it into an 50 | object using `JSON.parse()`. 51 | 52 | The previous pattern described above will continue to work in 53 | non-`ShadowRealm` scopes, but if a test is to be executed inside a 54 | `ShadowRealm` scope, it will need to change to a pattern like the 55 | following: 56 | 57 | ```js 58 | promise_test(() => fetch_json("resources/mydata.json").then(run_tests), "Loading data…"); 59 | ``` 60 | 61 | ### Expose location.search in ShadowRealm scopes 62 | 63 | Browser-specifc APIs such as `location` are not available in 64 | `ShadowRealm` scopes. 65 | However, the [mechanism for variant tests][variant] relies on 66 | `location.search`. 67 | 68 | This proposed change would expose a fake `location.search`, with the 69 | same value as the host realm's `location.search`, inside `ShadowRealm` 70 | scopes. 71 | This would allow test variants to be executed in `ShadowRealm` scopes. 72 | 73 | ### Use fake setTimeout in ShadowRealm scopes 74 | 75 | `ShadowRealm` scopes also do not have timer-related APIs such as 76 | `setTimeout` and friends. 77 | However, several facilities from `testharness.js`, namely 78 | [`Test.step_timeout`][step_timeout], [`Test.step_wait`][step_wait], and 79 | [`Test.step_wait_func`][step_wait_func], rely on the global `setTimeout` 80 | being present. 81 | 82 | This proposed change adds a fallback version of `setTimeout` which is 83 | used by the test harness when the global one is not present, i.e. in 84 | `ShadowRealm` scopes. 85 | The fallback is not exposed directly to test code. 86 | 87 | ## Risks 88 | 89 | The documentation commit carries no risk. 90 | 91 | The `fetch_json()` API doesn't change how any existing APIs work. 92 | Existing tests would have to opt-in to it by replacing their use of 93 | `fetch()` to import JSON test data, with `fetch_json()`. 94 | A natural time to make this change is when enabling a test to run in a 95 | `ShadowRealm` global scope; otherwise the test won't pass. 96 | Therefore, I think the potential risk is low. 97 | 98 | Of the four proposed changes, the fake `location.search` has the biggest 99 | potential risk. 100 | It's possible that future `ShadowRealm`-scope tests might depend on the 101 | `location` object not being present. 102 | This seems acceptable for the time being. 103 | But if that changes, the variant mechanism in files such as 104 | `common/subset-tests.js` and `common/subset-tests-by-key.js` would need 105 | to be rewritten to indicate which variant to run using a different 106 | mechanism than the URL search parameters. 107 | 108 | The fake `setTimeout` should not affect any existing tests. 109 | In non-`ShadowRealm` scopes, the behaviour should remain the same, 110 | because `setTimeout` is available. 111 | In `ShadowRealm` scopes, use of test harness APIs such as `step_timeout` 112 | would previously fail due to the missing `setTimeout`; now they can be 113 | used. 114 | Therefore, I think the potential risk is low. 115 | 116 | [rfc-107]: https://github.com/web-platform-tests/rfcs/blob/master/rfcs/shadowrealm-global.md 117 | [proposal]: https://github.com/tc39/proposal-shadowrealm 118 | [pr]: https://github.com/web-platform-tests/wpt/pull/43639 119 | [variant]: https://web-platform-tests.org/writing-tests/testharness.html#variants 120 | [step_timeout]: https://web-platform-tests.org/writing-tests/testharness-api.html#Test.step_timeout 121 | [step_wait]: https://web-platform-tests.org/writing-tests/testharness-api.html#Test.step_wait 122 | [step_wait_func]: https://web-platform-tests.org/writing-tests/testharness-api.html#Test.step_wait_func 123 | -------------------------------------------------------------------------------- /rfcs/get_signal.md: -------------------------------------------------------------------------------- 1 | # RFC 115: Add a new test method that returns AbortSignal 2 | 3 | ## Summary 4 | 5 | Add a new test method called `get_signal` to provide AbortSignal that aborts 6 | when the test finishes. 7 | 8 | This method will throw if `AbortController` is not available. 9 | 10 | ## Details 11 | 12 | Modern web APIs have adopted AbortSignal as a standardized cleanup method. Providing AbortSignal can save some lines to ease writing tests. 13 | 14 | ### Existing way to add and cleanup an event listener 15 | 16 | ```js 17 | promise_test(t => { 18 | const events = ["mousedown", "mouseup", "click"]; 19 | 20 | const gotEvents = []; 21 | const listenerFn = t.step_func(event => { 22 | gotEvents.push(event.type); 23 | }); 24 | for (const event of events) { 25 | target.addEventListener(event, listenerFn, { once: true }); 26 | } 27 | t.add_cleanup(() => { 28 | for (const event of events) { 29 | target.removeEventListener(event, listenerFn) 30 | } 31 | }); 32 | 33 | await test_driver.click(target); 34 | 35 | assert_array_equals(gotEvents, events); 36 | }); 37 | ``` 38 | 39 | ### With `AbortSignal` 40 | 41 | ```js 42 | promise_test(t => { 43 | const controller = new AbortController(); 44 | const { signal } = controller; 45 | t.add_cleanup(() => controller.abort()); 46 | 47 | const events = ["mousedown", "mouseup", "click"]; 48 | 49 | const gotEvents = []; 50 | for (const event of events) { 51 | target.addEventListener(event, t.step_func(event => { 52 | gotEvents.push(event.type); 53 | }), { once: true, signal }); 54 | } 55 | 56 | await test_driver.click(target); 57 | 58 | assert_array_equals(gotEvents, events); 59 | }); 60 | ``` 61 | 62 | ### With `t.get_signal()` 63 | 64 | ```js 65 | promise_test(t => { 66 | const { signal } = t.get_signal(); 67 | 68 | const events = ["mousedown", "mouseup", "click"]; 69 | 70 | const gotEvents = []; 71 | for (const event of events) { 72 | target.addEventListener(event, t.step_func(event => { 73 | gotEvents.push(event.type); 74 | }), { once: true, signal }); 75 | } 76 | 77 | await test_driver.click(target); 78 | 79 | assert_array_equals(gotEvents, events); 80 | }); 81 | ``` 82 | 83 | ## Risks 84 | 85 | Some people may end up using `t.get_signal().onabort` to add cleanup functions. 86 | This shouldn't be problematic if it's synchronous, but async cleanup functions 87 | should use the existing `t.add_cleanup()` instead to actually wait for them to 88 | finish. 89 | -------------------------------------------------------------------------------- /rfcs/global-window-module.md: -------------------------------------------------------------------------------- 1 | # RFC 133: Add support for `window-module` as JavaScript global 2 | 3 | ## Summary 4 | 5 | Add a new global called `window-module` so that other resources can be imported 6 | as modules on tests running on `window`. 7 | 8 | ## Details 9 | 10 | `tools/manifest/sourcefile.py`'s `_any_variants` will get `"window-module"`, 11 | and `tools/serve/serve.py` will get `WindowModuleHandler` that loads the script 12 | as a module. 13 | 14 | ## Risks 15 | 16 | Since module scripts can have top level await, some people may try awaiting any 17 | promise to setup the precondition before starting tests, instead of using 18 | `promise_setup()`. But such risk already exists since one can write a full HTML 19 | to achieve the same and there are other module globals e.g. 20 | `dedicatedworker-module`. 21 | -------------------------------------------------------------------------------- /rfcs/hide_test_state.md: -------------------------------------------------------------------------------- 1 | # RFC 58: Add a new test property to control displaying test state updates 2 | 3 | ## Summary 4 | 5 | Add a new test property called `hide_test_state` to control displaying 6 | test state updates while tests are running. 7 | 8 | This property will default to `false`, and can be set to `true` via the `setup()` function. 9 | 10 | ## Details 11 | 12 | In testharness there are a few usage of `show_status()` function, 13 | which outputs the test state to the documents during test 14 | runs. Some tests are sensitive to the content of the documents, 15 | so the test state output may interfere the test results. 16 | 17 | We are adding a new setup property to prevent outputting the 18 | test state. 19 | 20 | ## Risks 21 | 22 | No risks as this is an opt-in property. 23 | -------------------------------------------------------------------------------- /rfcs/http2.md: -------------------------------------------------------------------------------- 1 | # RFC 38: Enable http/2 server 2 | 3 | ## Summary 4 | 5 | Enable the http/2 server by default, configured to serve on port `9000`. Add a filename flag `.h2.` that enables loading top-level tests in http/2 mode. 6 | 7 | ## Details 8 | 9 | There has been a HTTP/2 server implementation in the tree for some time, but it was previously disabled due to problems running the tests on Travis. Now that we are no longer using Travis, it makes sense to enable the server. 10 | 11 | By default this server is configured to run on port 9000. 12 | 13 | In addition, actually using the server for testing is easier if one can directly load top-level tests on the server. For the https server we do this with a filename flag `.https.`. By analogy, the HTTP/2 server will work the same way with a `.h2.` flag. 14 | 15 | ## Risks 16 | 17 | Running an extra server is always some risk since it may cause instability. 18 | 19 | Vendor CI systems may not allow the use of port `9000` for some reason. 20 | 21 | This adds an extra requirement for OpenSSL 1.0.2+ and Python 2.7.15+ 22 | -------------------------------------------------------------------------------- /rfcs/https_port_8444.md: -------------------------------------------------------------------------------- 1 | # RFC 57: Add another HTTPS port, 8444 2 | 3 | ## Summary 4 | 5 | Add another https port, by default serving on `8444`. 6 | 7 | ## Details 8 | 9 | We currently have two HTTP ports, on `8000` and `8001`, and one HTTPS port on `8443`. In order to test features like [origin isolation](https://github.com/WICG/origin-isolation), we need another HTTPS port. 10 | 11 | A new HTTPS port will be added on `8444`. 12 | 13 | ## Risks 14 | 15 | Vendor CI systems may not allow the use of port `8444` for some reason. 16 | -------------------------------------------------------------------------------- /rfcs/integrate_wave_test_runner.md: -------------------------------------------------------------------------------- 1 | # Integration of the Web Media API Test Suite 2018 into WPT 2 | 3 | ## Summary 4 | 5 | This RFC discusses the integration of the Web Media API Test Suite 2018 which is based on Web-platform-tests. 6 | 7 | ## Details 8 | 9 | The [Web Media API Test Suite 2018 (WMATS2018)](https://github.com/cta-wave/WMAS) is a test suite for the [Web Media API Snapshot 2018](https://www.w3.org/2018/12/webmediaapi.html) specification. The test suite and specification are being developed as part of 10 | the [CTA WAVE Project](http://cta.tech/WAVE). This project is forked from [W3C Web Platform Tests](https://github.com/web-platform-tests/wpt) and is customized 11 | to automate test runs on web browsers for embedded devices and appliances suchs as TV sets, set-top boxes, consoles, etc. A hosted version is available at: https://webapitests2018.ctawave.org 12 | This RFC is created as a recommendation to [this WPT issues](https://github.com/web-platform-tests/wpt/issues/16214) where the initial discussion to integrate the Web Media API Test Suite into WPT is started. Please refer to the comments in this issue to get more insights in the discussion. 13 | 14 | ## Risks 15 | 16 | The new test runner incoporates components that run on the client (DUT and Companion device) and on the server. The server components are written in JavaScript and run in Node.js, which could be a risk for WPT since a new depencencie (Node.js) is introduced. If this is considered as an issue, it remains the option to port the JavaScript code to Python, but this is associated with additional effort. 17 | 18 | ## UPDATE 19 | The implementation of Node.js was ported to Python ([CTA](https://cta.tech/) contracted [Fraunhofer FOKUS](https://www.fokus.fraunhofer.de/fame) for this purpose) and the [WPT Pull Request](https://github.com/web-platform-tests/wpt/pull/21323) was created, reviewed and merged. To minimize the burden on WPT tooling, all steps were taken to isolate the runner from the rest of the WPT stack. Integration tests were also added. The [CTA WAVE project](https://github.com/cta-wave/) will maintain the new test runner in the future and may contract third parties for this purpose. John Riviello (@JohnRiv), chair of the CTA WAVE HTML5 API Task Force, is the contact person for further questions. 20 | -------------------------------------------------------------------------------- /rfcs/interop_charter.md: -------------------------------------------------------------------------------- 1 | # RFC 120: Interop Team Charter 2 | 3 | ## Summary 4 | 5 | An Interop Team is created as part of the web-platform-tests project, defined by a charter [in the team repo](https://github.com/web-platform-tests/interop/pull/102). The team is tasked with maintaining [Interop 2022](./interop_2022.md), [Interop 2023](./interop_2023.md) and future such efforts. 6 | 7 | ## Details 8 | 9 | The [team charter](https://github.com/web-platform-tests/interop/pull/102) is made official by this RFC and substantial changes can only be made through the web-platform-tests RFC process. 10 | 11 | The Interop Team can define additional public efforts/metrics beyond Interop 2022 and 2023. The process for doing that is decided by the Interop Team, may evolve over time, and is not defined by this RFC. 12 | 13 | The Interop team will take ownership of the existing Browser Specific Failures metric, and manage it according to the rules in its charter. 14 | 15 | ## Risks 16 | 17 | This charter is not time limited. Yet, there is no guarantee that there will be enough interest to define a yearly effort/metric, or that participants will be able to find consensus. The outcome would simply be that there will be no new effort/metric announced. In such a case the Interop Team and/or the WPT core team should evaluate whether it's time for an official end to the charter. 18 | -------------------------------------------------------------------------------- /rfcs/manifest-path-trie.md: -------------------------------------------------------------------------------- 1 | # RFC 40: New Manifest Format (path trie) 2 | 3 | ## Summary 4 | 5 | We propose restructuring the manifest to be a trie of path segments, rather than the status-quo of being an object with (full) paths as keys. 6 | 7 | ## Detail 8 | 9 | At the moment, the manifest is formed of three components: the url_base (a string), the items (an object of type to object pairs, then an object giving each path and the items contained therein), and the path_hash (storing all the paths—again—and the type & file hash). 10 | 11 | The proposal here is two-fold: 12 | 13 | Firstly, replace the items objects with a series of nested objects forming a trie of path segments. This allows us to make it much quicker to iterate through only specific directories, which is often enough done in a test-debug cycle, as part of redoing the implementation of include/exclude (this is future work for this RFC). 14 | 15 | Secondly, we get rid of path_hash: we migrate the hashes into the item object, thereby getting rid of the duplication of all the paths. This almost halves the size of the JSON created, which when the small-update case was previously dominated by parsing/serializing the JSON is significant. This provides a ~10% speedup running `wpt run` after modifying a single test. 16 | 17 | Additionally, we stop storing the URL if it is identical to the path; this again provides size savings but is not a critical part of this proposal. 18 | 19 | ## Risks 20 | 21 | As with any manifest format change, there is a risk that some consumers aren't paying attention to the version field and will fail loudly to cope with the new format. 22 | -------------------------------------------------------------------------------- /rfcs/manifest-working-copy.md: -------------------------------------------------------------------------------- 1 | # RFC 19: Always use the working copy for the manifest 2 | 3 | ## Summary 4 | 5 | Instead of defaulting to building a manifest using what's in the git HEAD, 6 | unconditionally use what's in the working copy. 7 | 8 | ## Details 9 | 10 | `./wpt manifest` supports two modes, the default (based on the `git` tree at 11 | HEAD) and `--work` (which uses the working copy). We supported the former mode 12 | mostly as a trivial performance optimization, though I believe it is now 13 | slower. 14 | 15 | Many people find the (default) `git` behaviour confusing (adding a new test 16 | won't be runnable till it's been committed, for example), and it's not clear it 17 | gives much in the way of benefits. 18 | 19 | We should drop support for the `git` behaviour and always use the working copy 20 | behaviour. 21 | 22 | We should, however, amend the working copy behaviour to, if possible, call `git 23 | ls-tree` (or similar) to get object IDs without having to read & hash them 24 | ourselves. 25 | 26 | ## Risks 27 | 28 | This could make manifest generation much slower for people who have large 29 | numbers of extra files in their working copy. 30 | 31 | We will generate manifest items for items ignored in non-root ignore files 32 | (https://github.com/web-platform-tests/wpt/issues/7206), which e.g. if the CSS 33 | build system has been run could be tens of thousands of extra test files. 34 | -------------------------------------------------------------------------------- /rfcs/matrix.md: -------------------------------------------------------------------------------- 1 | # RFC 84: Move web-platform-tests Chat to Matrix 2 | 3 | ## Summary 4 | 5 | Move the official location for synchronous discussions to Matrix, and 6 | retire the `#testing` channel on irc.w3.org. 7 | 8 | The proposed channel is [wpt:matrix.org](https://app.element.io/#/room/#wpt:matrix.org). 9 | 10 | ## Details 11 | 12 | ### Context 13 | 14 | Since web-platform-tests has started it has primarily used IRC as a 15 | means for (quasi) synchronous communications ("chat"). In particular 16 | the `#testing` channel on irc.w3.org has been used to answer questions 17 | from test authors and collaborate on infrastructure development and 18 | maintenance. 19 | 20 | IRC has historically been the go-to medium for web-platform related 21 | chat, with W3C, TC39 and WHATWG all using IRC channels as the primary 22 | communication mechanism for standards work, and browser engine 23 | projects including Chromium, Gecko, Servo, and WebKit having IRC as a 24 | well-supported option for synchronous communication. This made it 25 | likely that web-platform-tests participants were already using IRC in 26 | some capacity, so an IRC channel was an obvious choice for 27 | web-platform-tests. 28 | 29 | Recently however, the limitations of IRC have become more apparent: 30 | 31 | * New contributors who have not grown up with IRC find it confusing 32 | and limited compared to other contemporary chat services. For 33 | example the fact that you only receive messages when connected to 34 | the channel, and there's no built-in logging/history, is a source of 35 | confusion to people who expect to be able to ask a question, 36 | disconnect, and reconnect later to see the answer. Although it's 37 | possible to work around this with the correct clients or external 38 | logs, this requires investment in understanding the IRC ecosystem 39 | that new users by definition don't have. 40 | 41 | * The general problem with lack of moderation and difficulty 42 | controlling spam can make IRC an unwelcoming environment, especially 43 | for people in groups who frequently experience harassment 44 | online. These people may prefer to not participate in a project 45 | rather than use IRC. 46 | 47 | As a result of this, and other factors, other projects have 48 | increasingly moved away from IRC, and it is no longer the "default" 49 | choice for web-platform work. In particular: 50 | 51 | * Gecko moved to Matrix, hosted on 52 | [chat.mozilla.org](https://chat.mozilla.org). 53 | 54 | * [WHATWG](https://app.element.io/#/room/#whatwg:matrix.org) 55 | [[discussion](https://github.com/whatwg/meta/issues/210)] and 56 | [TC39](https://app.element.io/#/room/#tc39-general:matrix.org) moved 57 | to Matrix, hosted on [matrix.org](https://matrix.org). 58 | 59 | * [WebKit](https://webkit.org/getting-started/#contribute) moved to Slack. 60 | 61 | * Chromium moved to an invite-only Slack instance. 62 | 63 | * Servo moved to [Zulip](https://servo.zulipchat.com/) 64 | 65 | * W3C is still extensively using IRC, but some groups have moved to 66 | [Slack](https://w3ccommunity.slack.com/). 67 | 68 | This means that it can no longer be assumed that web-platform-tests 69 | contributors are already using IRC as part of their participation in 70 | web platform work. Along with the disadvantages of IRC compared to 71 | modern alternatives, this suggests that we should also reconsider our 72 | choice here. 73 | 74 | ### A Rather Biased Discussion of the Options 75 | 76 | On the basis that having to use an additional chat system is itself a 77 | barrier to contribution, we should look for alternatives that are 78 | already used by the wpt-adjacent projects above, rather than choosing 79 | from the large set of all existing systems. This suggests the two 80 | reasonable alternatives are Matrix and Slack. 81 | 82 | Slack's primary business is company-internal communications. This is 83 | reflected in the fact that Slack instances typically require an 84 | invite. Although it's possible to make instances that are broadly 85 | accessible, the emphasis is very much on closed communities. This 86 | provides a significant barrier to entry for new contributors, who may 87 | feel they shouldn't look for an invite until they are already known 88 | within a community. In addition the fact that Slack is a centralised 89 | service run for profit risks the terms of use changing in an 90 | unfavourable way in the future (e.g. to require payment). 91 | 92 | Matrix is an open protocol with which it's possible to self-host a 93 | server, and use a custom client. However, compared to IRC, there's a 94 | clear default client (Element), which runs in a web browser and offers 95 | a polished user experience. Whilst the feature-set of Matrix/Element 96 | does not fully match that of Slack, it exceeds what's possible over 97 | IRC, offering per-channel history, limited rich text in messages, link 98 | previews, reaction emoji, etc. It also offers extensive moderation 99 | tools which suggests it will address some of the concerns around 100 | safety on IRC. Compared to Slack the focus is more on open 101 | communities, and specific invites are not required to join public 102 | channels. 103 | 104 | ### Proposal 105 | 106 | We set up a matrix channel, 107 | [wpt:matrix.org](https://app.element.io/#/room/#wpt:matrix.org), 108 | hosted on matrix.org, and point to this as the official channel for 109 | web-platform-tests chat, fully replacing IRC. 110 | 111 | This matches TC39 and WHATWG. Although self-hosting would provide some 112 | benefits in terms of not depending on third-party infrastructure, the 113 | work required is not worthwhile in the short term. The federated 114 | nature of Matrix should make it possible to move if hosting on 115 | matrix.org ever becomes problematic. 116 | 117 | ### Admins 118 | 119 | We should have multiple channel admins to ensure the channel can 120 | continue operating smoothly in the absence of any 121 | individual. Initially I propose the following, drawn from members of 122 | the core team who are regularly on IRC (using GitHub usernames): 123 | 124 | * @jgraham 125 | * @foolip 126 | * @gsnedders 127 | * @sideshowbarker 128 | 129 | It's unclear if there is a good reason to additionally have moderators. 130 | 131 | ### Documentation 132 | 133 | All documentation pointing to the IRC channel will be updated to point 134 | at the matrix channel instead. 135 | 136 | ### Risks 137 | 138 | * Bots which are present on IRC may not be available for Matrix. In 139 | particular we are using bitbot to notify about GitHub issues and 140 | PRs in the IRC channel. 141 | 142 | The use of bitbot is already somewhat controversial, with some 143 | participants disliking the frequent notifications. So it may not be 144 | a net loss to simply turn the bot off and require people to use 145 | another method to keep up with GitHub notifications. Alternatively 146 | we could keep bitbot running on IRC and allow people to use the 147 | Matrix/IRC bridge to see those messages, whilst otherwise not using 148 | the IRC channel. Given the popularity of GitHub, it is also likely 149 | we can find a Matrix bot to replace bitbot, which could then run in 150 | a separate wpt-notifications channel. 151 | 152 | * Changing communication mechanisms may fragment the community. 153 | 154 | This seems low risk; informally there was little dissent to the 155 | idea of moving to Matrix, and with other projects already making 156 | similar moves it seems more likely that contributors will 157 | participate on Matrix than on IRC. This would match the experience 158 | of Mozilla where the matrix channels quickly proved more active 159 | than IRC had ever been. 160 | -------------------------------------------------------------------------------- /rfcs/minimize_restore_window.md: -------------------------------------------------------------------------------- 1 | # RFC 100: Add testdriver.js support for window minimize/restore (available in WebDriver and Marionette) 2 | 3 | ## Summary 4 | 5 | Add testdriver.js support `minimize` and `restore`, which wrap the for the [WebDriver `minimize` command](https://w3c.github.io/webdriver/#minimize-window) and [`setWindowRect`](https://w3c.github.io/webdriver/#set-window-rect). 6 | 7 | ## Details 8 | 9 | The [Page Visibility](https://www.w3.org/TR/page-visibility-2/) spec adds web-exposed insight into 10 | whether the browser is "in the background". This can have different meanings on different platforms. 11 | However, in WebDriver one scenario of this is available through `minimize`. 12 | 13 | By allowing the test to minimize/restore the window, we can test how pages using the exposed APIs 14 | interact with that behavior. 15 | 16 | ## Risks 17 | 18 | Tests that use this API might not give consistent results when running in headless mode. 19 | This is due to the nature of the `page visibility` API and APIs that require the browser to be 20 | in a particular state with regards to the OS in order to succeed. 21 | 22 | Also, some platforms (mobile)? may not support `minimize`, in which case tests using this feature 23 | should fail early with a precondition. -------------------------------------------------------------------------------- /rfcs/origin_policy.md: -------------------------------------------------------------------------------- 1 | # RFC #44: Testing origin policy 2 | 3 | Note: This was reverted in [RFC #113](origin_policy_revert.md). 4 | 5 | ## Summary 6 | 7 | [Origin policy](https://wicg.github.io/origin-policy/) is a proposed web platform feature that applies origin-wide settings. Writing web platform tests for origin policy is complicated, because: 8 | 9 | - The origin policy is a per-origin resource, located at `/.well-known/origin-policy`. To test different origin policy manifests, a test needs different responses to be served from that resource. 10 | - Origin policy settings apply origin-wide, which can cause interference with other tests running on that origin. For example, a CSP that disables image loading would interfere with any image-loading tests running on that origin. 11 | 12 | This proposal introduces new subdomains to the web platform tests serving infrastructure which are used exclusively for origin policy tests, and also serves `/.well-known/origin-policy` with the Python script handler instead of the static file handler, to allow varying responses. 13 | 14 | ## Details 15 | 16 | An implementation of this proposal can be seen in [web-platform-tests/wpt#21705](https://github.com/web-platform-tests/wpt/pull/21705). 17 | 18 | The added subdomains are of the form `opX` for `X` from 1 through 99. We are hopeful that 99 subdomains will be enough for writing all the necessary origin policy tests; if this ends up being too few, we can do a future RFC to increase the number. 19 | 20 | The added subdomains do not contribute to the "product" subdomains (of the form `www.www1`, etc.) which were introduced in [web-platform-tests/wpt#13272](https://github.com/web-platform-tests/wpt/pull/13272). 21 | 22 | The use of the Python script handler for `/.well-known/origin-policy` allows test writers to drop a Python script in that location which serves up different origin policies depending on the subdomain (e.g. by examining `request.url_parts.hostname`). 23 | 24 | ## Risks 25 | 26 | The main risk here is the same as with any infrastructure change, in that it requires downstream consumers to be aware of the change and pick it up. 27 | 28 | To the extent embedders are using `wptserve`, they will recieve the Python script handler change without difficulty. The subdomain configuration will be automatic as long as consumers are either using `wpt make-hosts-file`, or are mapping all subdomains of `.test` uniformly (like Chromium does with its `--host-resolver-rules`). 29 | -------------------------------------------------------------------------------- /rfcs/origin_policy_revert.md: -------------------------------------------------------------------------------- 1 | # RFC #113: Remove Origin Policy infrastructure 2 | 3 | ## Summary 4 | 5 | Support for testing [Origin Policy](https://wicg.github.io/origin-policy/) was introduced in [RFC #44](origin_policy.md) + [wpt PR #21705](https://github.com/web-platform-tests/wpt/pull/21705). Revert this code, removing the 99 op* subdomains, for example op4.web-platform.test. 6 | 7 | ## Details 8 | 9 | Origin Policy is no longer being pursued. The tests using this infrastructure were removed in [wpt PR #33875](https://github.com/web-platform-tests/wpt/pull/33875). 10 | 11 | ## Risks 12 | 13 | There could be tests using the subdomains for something other than testing Origin Policy. To catch any such cases, all instances of "op" followed by a digit will be inspected, and full runs of wpt with the infrastructure removed will be compared to the current state. 14 | -------------------------------------------------------------------------------- /rfcs/page-size.md: -------------------------------------------------------------------------------- 1 | # RFC 173 - `@page` Rules in Print Tests 2 | 3 | [RFC 41](https://github.com/web-platform-tests/rfcs/blob/master/rfcs/print_test.md) added support for print reftests. It specified a fixed page size of "5 inches by 3 inches with 0.5 inch margins on all sides". 4 | 5 | CSS allows modifying the size and margins of paginated media using an [`@page` rule](https://developer.mozilla.org/en-US/docs/Web/CSS/@page). Per the letter of the previous RFC these rules would be ignored for wpt tests, making the rule itself impossible to test. 6 | 7 | To enable testing different paper sizes and margins, this RFC amends the print reftest support, so that where an `@page` rule is specififed in a test or reference file, it overrides the default paper size when generating the paginated rendering of the document. 8 | -------------------------------------------------------------------------------- /rfcs/polyfill_testing.md: -------------------------------------------------------------------------------- 1 | # RFC 111: Polyfill Testing 2 | 3 | ## Summary 4 | 5 | Add a mechanism for loading one or more polyfill implementations of a web 6 | platform feature and testing their correctness on existing web platform tests. 7 | 8 | ## Details 9 | 10 | Polyfills are typically best effort implementations of browser features. It's 11 | not uncommon for them to be outdated, broken or just not to implement all parts 12 | of the specification they are polyfilling. Additionally, polyfills may only work 13 | with certain browsers or with certain libraries. This proposal allows testing 14 | polyfills using the same test suite we test browser compliance with. Being able 15 | to know which tests the polyfill passes or doesn't pass is useful information 16 | when developers are determining whether they can rely on it, just as they would 17 | use this information to know whether they could rely on the feature in the 18 | browser. 19 | 20 | In order to support testing a polyfill, a runtime argument is added which 21 | when used specifies a single local Javascript file to be injected into 22 | responses. The contents of the file will be added as a classic script (to ensure 23 | the polyfill runs before any tests) automatically to responses with a 24 | `text/html` MIME type after the `doctype` and `html` and `head` opening tags 25 | (if they exist) but before any other tags in the document. 26 | 27 | In order to reduce visible effects on running tests, the script is inlined (to 28 | prevent an external resource load which would show up in the resource timing 29 | API) and the script will remove itself from the DOM to ensure that tests which 30 | assume the shape of the DOM is unchanged will still function correctly (e.g. 31 | `document.getElementsByTagName('script')[0]` will still return the first script 32 | in the original test). 33 | 34 | E.g. when running `wpt run --inject-script=script.js [browser] [test]` or 35 | `wpt serve --inject-script=script.js` and loading the following HTML page: 36 | ```html 37 | 38 |
39 | 42 | ``` 43 | 44 | The response will be modified to the following: 45 | ```html 46 | 47 | 52 |
53 | 56 | ``` 57 | 58 | This transparent injection of the polyfill script allows the injected script to 59 | run before any tests, and on all loaded HTML documents including reftests which 60 | often don't include any script resources without requiring any modification to 61 | existing test files. It also supports testing in a local browser as `wpt serve` 62 | responses are similarly modified. 63 | 64 | ## Risks 65 | 66 | There are a few risks or downsides with the approach that may lead to false 67 | test failures or failure to inject the polyfill. 68 | 69 | * While it should not be possible to observe the script removal in the main 70 | frame, it may be possible to observe it with a MutationObserver on a loading 71 | subframe. 72 | * Line numbers in failure messages will not match their original source as they 73 | will include injected content. We could mitigate this in the future by 74 | stripping newlines and injecting the entire polyfill into a single 75 | pre-existing line. 76 | * The polyfill is not injected on pages served from python response handlers 77 | which write directly to the output response stream. This could be supported 78 | with future modifications. 79 | 80 | ## Alternatives considered 81 | 82 | ### Adding a canonical resource which includes the polyfill 83 | 84 | 2. Add the polyfill into testharnessreport.js. 85 | 3. Add the polyfill into a new /resources/polyfill.js which could also be 86 | included by reftests. 87 | 88 | We could transparently insert the polyfill into a canonical resource which is 89 | included by most tests, for example `testharnessreport.js` or 90 | `/resources/polyfill.js`. Given that these external resources are expected to 91 | change it should not cause test failures due to the resource. 92 | 93 | Advantages: 94 | * Page structure and loaded resources are not modified. 95 | 96 | Concerns: 97 | * All tests which don't already include one of these resources (e.g. reftests) 98 | would need to be modified to include them. This would represent an ongoing 99 | maintenance burden to ensure that new tests add these resources and failure to 100 | do so wouldn't be caught by existing tests. Likely this would result in a 101 | mixture of tests which support running with polyfills and tests which don't. 102 | * If we later adopt a mechanism which doesn't require explicitly including the 103 | polyfill resources these additional artifacts may take some time to remove. 104 | 105 | ### Bootstrap scripts feature 106 | 107 | Polyfill resources could be injected using the [bootstrap scripts feature]( 108 | https://github.com/w3c/webdriver-bidi/issues/65). This would allow the polyfill 109 | to be transparently injected into tests. 110 | 111 | Advantages: 112 | * Polyfill is transparently inserted across all existing tests (including reftests). 113 | * Served files would match original test files. 114 | * As no modification is required, can easily switch mechanisms later. 115 | 116 | Concerns: 117 | * Developers would not be able to debug tests served from running `wpt serve` by 118 | simply loading them in the browser. 119 | * This mechanism is not available yet. 120 | 121 | ### Extension injection 122 | 123 | Similar to the bootstrap scripts feature, we could use an installed extension to 124 | inject the polyfill. As most polyfills would require installing functionality 125 | into the main script isolate, this would likely require injecting a script tag 126 | into the main document. 127 | 128 | Advantages: 129 | * Polyfill is transparently inserted across all existing tests (including reftests). 130 | * As no modification is required, can easily switch mechanisms later. 131 | 132 | Concerns: 133 | * As we would need to insert a script tag into the document, this would have the 134 | same DOM observability concerns as the rewriting approach, with the same 135 | possible mitigations. 136 | * Developers would not be able to debug tests served from running `wpt serve` by 137 | simply loading them in the browser. They could however install the extension 138 | which injected the polyfill. 139 | * Mobile browsers have no current support for these kinds of extensions. 140 | 141 | ## Conclusion 142 | 143 | Rewriting the responses from the wpt server is a solution that is available now, 144 | supports HTML tests, and supports testing both via `wpt run` and by 145 | visiting the served pages from an external browser. While there are 146 | observability concerns they can largely be mitigated by removing the artifacts 147 | added to the page in the injected script. Since injecting a polyfill is not the 148 | default execution path, this is a pragmatic solution that will allow testing 149 | polyfills now and can be modified in the future with minimal changes. 150 | -------------------------------------------------------------------------------- /rfcs/promise.md: -------------------------------------------------------------------------------- 1 | # RFC 93: Allow testharness.js to depend on Promise being available 2 | 3 | ## Summary 4 | 5 | Currently testharness.js has a test-enforced policy that ES Promises 6 | are only required by features which specifically depend on promises 7 | (e.g. `promise_test`). Other functionality must work in the absence of 8 | promises. 9 | 10 | This RFC removes that restriction, allowing ES Promises to be used 11 | throughout the testharness.js codebase. 12 | 13 | ## Details 14 | 15 | Historically ES Promises were not widely supported across browser 16 | engines. Native promises became available in 2014, and even some years 17 | later browsers that were known to make use of web-platform-tests were 18 | running JavaScript engine versions without Promise support. In 19 | particular Servo was making heavy use of web-platform-tests but didn't 20 | support promises until late 21 | [2016](https://github.com/servo/servo/issues/4282), whilst Internet 22 | Explorer has never had Promise support. 23 | 24 | Due to the Servo requirements in particular, the testharness.js tests 25 | were written to run in a mode with the [global promise constructor 26 | removed](https://github.com/web-platform-tests/wpt/commit/08995880734011eed359c4e79420db29626016ff#diff-d33a74992c12a398b4876be7ea865df0387f779f740be2ff79e792f7d4db5badR12-R14). This 27 | ensures that features which don't explicitly depend on Promise don't 28 | accidentally rely on it, allowing most tests to run in 29 | implementations without Promises. 30 | 31 | Today the number of browsers which might conceivably run wpt but lack 32 | Promise support is very low. Many new DOM APIs require promises, so 33 | the only implementations which lack Promise support are those such as 34 | IE 11 which are no longer receiving feature updates. This makes it 35 | safe to rely on promises across the entire testharness codebase. It 36 | opens up the possibility of refactorings to make the code simpler, and 37 | removes a point of confusion for new contributors who are unaware of 38 | the unusual policy of avoiding promises. 39 | 40 | The proposal is that we allow promises throughout testharness as a 41 | matter of policy, and remove the variants feature from the 42 | testharness.js tests, along with all usage of it. Since this feature 43 | is currently only used for with and without promise variants, we can 44 | simplify test authoring by not having the additional complexity. 45 | 46 | ## Risks 47 | 48 | * Some unusual older browsers might still be running wpt. For example 49 | we know that the WAVE server is used for testing TV-based browsers, 50 | which could conceivably be so outdated as to not support 51 | promises. We don't have much insight into how this is used so could 52 | conceivably result in "dark matter" breakage. 53 | 54 | * Removing the variants feature from testharness.js tests entirely 55 | might be premature if we find we need similar functionality in the 56 | future. However the fact that we've currently only used it for this 57 | one use case, and the fact that it adds noticeable complexity to test 58 | authoring and to the harness suggests we should consider re-adding 59 | the feature if we find an additional use case. 60 | -------------------------------------------------------------------------------- /rfcs/proxy_for_connection_testing.md: -------------------------------------------------------------------------------- 1 | # RFC 112: Add a Proxy Automatic Config file (PAC) to test connection timing/hints 2 | 3 | ## Summary 4 | 5 | Use a proxy to create artificial connection delays and use fresh domain names for tests. 6 | 7 | ## Details 8 | 9 | The [Resource Timing](https://w3c.github.io/resource-timing/) spec contains information about 10 | connection timing (time spent in DNS resolution, creating a connection, handshake etc). 11 | 12 | So far it has been difficult/impossible to test this reliably in WPT, as connections are usually 13 | pooled and it's difficult to assert that a connection made from a test to one of the predefined 14 | WPT domains is fresh. 15 | 16 | The same goes for [preconnect](https://html.spec.whatwg.org/#link-type-preconnect) - testing that 17 | preconnecting has an effect would require creating artificial connection delays and check that a 18 | preconnect eliminates those delays. 19 | 20 | Other use cases are listed [here](web-platform-tests/wpt#13465). In essence, specs that rely on 21 | caching based on [network partition keys](https://fetch.spec.whatwg.org/#network-partition-keys) 22 | can be tested more reliably if an arbitrary fresh [registrable domain](https://url.spec.whatwg.org/#host-registrable-domain) is created on the fly and can simulate a situation where a cold network partition is accessed. Note that tests that require ad-hoc registrable domains will be non-secure, as the certificate cannot be generated on the fly. 23 | 24 | In addition, this can help test behavior on default ports, as PAC files can reroute fetches to 25 | given ports. 26 | 27 | Suggesting to generate the delays using a [Proxy Automatic Config file](https://developer.mozilla.org/en-US/docs/Web/HTTP/Proxy_servers_and_tunneling/Proxy_Auto-Configuration_PAC_file) that would route arbitrary hosts to the web platform tests server and would generate artificial resolution delays. 28 | 29 | The URL for the PAC file would be declared in the testheader , e.g. 30 | ``, which in some cases would mean that tests with a PAC 31 | file would have to run in a separate browser session. 32 | 33 | 34 | ## Risks 35 | 36 | * Relies on a non-standard feature (PAC). However, that feature is very old and widely supported. 37 | * Introducing delays inside PAC can theoretically cause browsers to ignore it, need to pay attention to this. 38 | * Testing timing information is sensitive to raciness, but that's the nature of the feature and 39 | not directly related to proxies. 40 | * More a limitation than a risk - by generating ad-hoc registrable domains we won't be able to use the certificate, which would limit that kind of test to non-secure only. 41 | -------------------------------------------------------------------------------- /rfcs/pullpanda_analytics.md: -------------------------------------------------------------------------------- 1 | # RFC 35: Add third-party GitHub App: Pull Panda 2 | 3 | ## Summary 4 | 5 | At TPAC 2019 we set out some [priorities for 2020](https://docs.google.com/document/d/1gie7LFb6cAUfabY86MYuWM7m7ux_FaKhDkLdpz0zWkg/edit), 6 | which include increasing PR review velocity and reduce the number of stale PRs. 7 | 8 | In order to track progress over time, 9 | install the [Pull Panda](https://pullpanda.com/) GitHub App 10 | for the web-platform-tests organization (all repositories). 11 | 12 | Pull Panda can show [analytics](https://pullpanda.com/analytics) for the past 24 weeks for, among other things, 13 | PR merge time and open PRs, 14 | which are relevant for tracking the above-mentioned priorities. 15 | 16 | No setup required, only give it read permission to the organization. 17 | The analytics should be [visible to all organization members](https://docs.pullpanda.com/products/pull-analytics/user-access). 18 | This app is owned by GitHub. 19 | 20 | ## Details 21 | 22 | Pull Panda Analytics has these graphs and explanations: 23 | 24 | * Review turnaround 25 | * Reviewer workload: Shows the number of PRs currently assigned to reviewers. 26 | * Open PRs: This is the number of pull requests open right now and at the start of each week in the past. 27 | * PR merge time 28 | - Breakdown of merge time: Merge time is the time it takes for a pull request to go from opened to merged. 29 | - Average merge time: This is the average (mean) pull request merge time by week. Merge time is the time it takes for a pull request to go from opened to merged. 30 | * PR throughput 31 | - PRs merged: This shows the number of pull requests opened and merged by week. 32 | - PR status by week opened: This is a cohort analysis showing the status of pull requests grouped by the week they were opened. 33 | * PR size 34 | - PR size is caclulated as the sum of lines added and removed. For example, if a pull request has 5 lines added and 5 lines removed, its size is 10 LOC. 35 | - Success rate (based on a target of 500 LOC (custimizable)): This shows the percentage of pull requests that met the target. PR size is calculated as the sum of lines added and removed. 36 | * Code churn 37 | - Additions and deletions: This shows the number of lines added and deleted in merged pull requests. 38 | - Contributors: This shows the number of users who authored merged pull requests. 39 | 40 | Pull Panda can be installed on the whole organization or select repositories. 41 | It only requests read access for analytics. 42 | 43 | Pull Panda was [acquired by GitHub](https://pullpanda.com/github) in June, 2019. 44 | It is free to use for GitHub orgs. 45 | 46 | ## Risks 47 | 48 | 24 weeks (\~6 months) is not enough history to see trends on a longer time scale. 49 | There is [an open issue](https://github.com/pullreminders/backlog/issues/94) about this. 50 | 51 | Trust. This risk is severely mitigated by the granted permission being read-only 52 | and that the app is owned by GitHub. 53 | 54 | ## Alternatives considered 55 | 56 | https://github.com/cncf/devstats supports showing graphs for various interesting things, in particular 57 | PR Time to Approve and Merge ([example](https://k8s.devstats.cncf.io/d/44/pr-time-to-approve-and-merge?orgId=1)), 58 | Open PR Age by Repository Group ([example](https://k8s.devstats.cncf.io/d/25/open-pr-age-by-repository-group?orgId=1)) and 59 | Open issues/PRs by milestone and repository ([example](https://k8s.devstats.cncf.io/d/22/open-issues-prs-by-milestone-and-repository?orgId=1)). 60 | 61 | This was pretty involved in its setup. 62 | Initial attempts to configure locally on macOS were unsuccessful. 63 | Either a project can be added to devstats.cncf.io, 64 | or something can be set up with Travis CI, 65 | but these options have not been further investigated as both seem non-trivial. 66 | -------------------------------------------------------------------------------- /rfcs/py_3.md: -------------------------------------------------------------------------------- 1 | # RFC 69: Switch web-platform-tests to Python 3 2 | 3 | ## Summary 4 | 5 | This RFC sets out the overall plan and timetable for the Python 6 | 2->Python 3 transition. It is anticipated that further details on the 7 | transition will be set out in other RFCs as appropriate. 8 | 9 | ## Details 10 | 11 | 1. **Week of 23rd November 2020** - Python 3 becomes default on GitHub CI 12 | 13 | Details: https://github.com/web-platform-tests/rfcs/pull/65/ 14 | 15 | Prerequisites: 16 | 17 | * All existing Python code updated to work with Python 3 18 | * No regressions in test behaviour or harness functionality in Python 3 19 | * A --py2 flag is added to `wpt` to opt-in to running in Python 2 20 | 21 | After this time unittests will still be run as Python 2 and Python 22 | 3 but full test runs and PR test runs will be Python 3 only. 23 | 24 | Handler code will be expected to continue working in Python 2.7. 25 | 26 | 27 | 1. **January 1st 2021** - Vendors are expected to have completed their transition to Python 3 28 | 29 | After the upstream transition to Python 3, it is assumed that 30 | vendors will wait a few days to ensure that the change sticks and 31 | then arrange to update their CI systems to match upstream. 32 | 33 | While upstream is mainly running Python 3 but vendors are still 34 | running Python 2 they may submit changes that only work with 35 | Python 2, requiring post-hoc fix-up. Therefore in this period 36 | careful monitoring of Python changes will be needed. 37 | 38 | After this date `wpt` will run with Python 3 by default and `wpt 39 | --py2` will be needed to get the old behaviour. The `--py3` flag 40 | will be a no-op. 41 | 42 | 1. **February 1st 2021** - Python 2 no longer run on CI 43 | 44 | After January 1st it will be assumed that everyone has converged 45 | on Python 3 and Python changes will not be monitored for 2.7 46 | compatibility. However unittests will continue running on 2.7 47 | until February, in case there is an unexpected need to revert the 48 | changes. In this period we will not intentionally accept 49 | changes that regress Python 2 (e.g. removing six usage). 50 | 51 | After February 1st all testing on Python 2 will be disabled and we 52 | will accept patches to remove Python 2 support. 53 | 54 | The Python version flags will be removed from `wpt`. 55 | -------------------------------------------------------------------------------- /rfcs/python-file-handler-local-imports.md: -------------------------------------------------------------------------------- 1 | # RFC 68: Make local imports in Python file handlers root-relative 2 | 3 | ## Summary 4 | 5 | Require local imports in Python file handlers to be specified relative to the 6 | `docroot` of wptserve, e.g.: 7 | 8 | ``` 9 | html/ 10 | └── dom/ 11 | ├── handler.py 12 | └── helper.py 13 | ``` 14 | 15 | Would require `handler.py` to import `helper.py` via: 16 | 17 | ```python 18 | from html.dom import helper 19 | ``` 20 | 21 | ## Background 22 | 23 | [Python file handlers] are Python files that `wptserve` will execute in 24 | response to requests matching a corresponding URL. Currently Python file 25 | handlers are able to import local helper scripts from the same directory they 26 | are in, e.g.: 27 | 28 | ``` 29 | html/ 30 | └── dom/ 31 | ├── handler.py 32 | └── helper.py 33 | ``` 34 | 35 | Here, `handler.py` can import `helper.py` via: 36 | 37 | ```python 38 | import helper 39 | ``` 40 | 41 | This behavior is currently supported by [mutating sys.path and 42 | sys.modules][mutating-code] when loading a file handler. Unfortunately this 43 | mutation [does not actually work][filehandler-bug]: 44 | 45 | 1. The copying of `sys.modules` is not sufficient, so helper scripts collide by 46 | name (no matter where in the tree they are). 47 | 2. The copying of `sys.modules` can also corrupt underlying state, leading to 48 | import errors in Python 3. 49 | 3. `wptserve` is actually threaded, so we are also messing with `sys.path` and 50 | `sys.modules` in a threaded manner - this amazingly hasn't caused us pain so 51 | far in Python 2 but is definitely dangerous. 52 | 53 | An effort was made to [fix these problems][filehandler-fix-1] whilst keeping 54 | the same import syntax, but required brittle changes relying on deep Python 55 | internals, so instead this RFC proposes changing the local-import syntax. 56 | 57 | ## Details 58 | 59 | The core changes that this RFC captures are to: 60 | 61 | 1. Remove the adding of the local directory for a given Python file handler to 62 | `sys.path`, 63 | 2. Remove the attempted mutation of `sys.modules`, and 64 | 3. Add the wptserve `docroot` (usually the WPT root directory) to `sys.path`. 65 | 66 | This has a number of knock-on effects. Most prominently, test authors will now 67 | have to reference local helper files by their full path: 68 | 69 | ```python 70 | from html.dom import helper 71 | ``` 72 | 73 | If the helper script is in a directory that contains a hyphen (e.g. 74 | `css/css-animations/helper.py`), test authors will have to use 75 | `importlib.import_module`: 76 | 77 | ```python 78 | import importlib 79 | helper = importlib.import_module("css.css-animations.helper") 80 | ``` 81 | 82 | Until we reach Python3-only (and thus can rely on [PEP 420]), test authors will 83 | also have to make sure there is a path of `__init__.py` files from the WPT root 84 | to the helper file, e.g.: 85 | 86 | ``` 87 | __init__.py 88 | html/ 89 | ├── __init__.py 90 | └── dom/ 91 | ├── __init__.py 92 | ├── handler.py 93 | └── helper.py 94 | ``` 95 | 96 | ### Knock-on impact of root `__init__.py` on unittests 97 | 98 | As noted above, until we get to Python3-only this RFC requires a root 99 | `__init__.py` file to be added to WPT. This causes some problems for the WPT 100 | unittests, which previously relied on [pytest behavior] to 'magically' add the 101 | root WPT directory to `sys.path` and allow them to do e.g. 102 | 103 | ```python 104 | from tools.ci.tc import decision 105 | ``` 106 | 107 | With this RFC, these unittests all need to change to explicitly add the root to 108 | the path, e.g.: 109 | 110 | ```python 111 | here = os.path.dirname(__file__) 112 | root = os.path.abspath(os.path.join(here, "..", "..", "..", "..")) 113 | sys.path.insert(0, root) 114 | 115 | from tools.ci.tc import decision 116 | ``` 117 | 118 | Once we have PEP 420 available, these can all be cleaned up and restored to 119 | their previous behavior. 120 | 121 | ## Risks 122 | 123 | * The new syntax is undeniably harder to use than the previous syntax, and so 124 | may annoy test authors or push them away from using python file handlers. 125 | * In particular we have a lot of directories in WPT that use hyphens 126 | (approximately 60% of unique directory paths in WPT contain a hyphen). 127 | * Albeit note that this only applies if one is using a local helper file, 128 | which few file handlers actually do. 129 | 130 | [Python file handlers]: https://web-platform-tests.org/writing-tests/python-handlers/index.html 131 | [mutating-code]: https://github.com/web-platform-tests/wpt/blob/09dd5ef1d47633394f6e368ef62fd4f665cf18a6/tools/wptserve/wptserve/handlers.py#L290 132 | [filehandler-bug]: https://github.com/web-platform-tests/wpt/issues/25678 133 | [filehandler-fix-1]: https://github.com/web-platform-tests/wpt/pull/26111 134 | [PEP 420]: https://www.python.org/dev/peps/pep-0420/ 135 | [pytest behavior]: https://docs.pytest.org/en/stable/pythonpath.html#prepend-and-append-import-modes-scenarios 136 | -------------------------------------------------------------------------------- /rfcs/python_37.md: -------------------------------------------------------------------------------- 1 | # RFC 134: Bump minimum supported Python to 3.7 2 | 3 | ## Summary 4 | 5 | Increase the minimum supported Python version to 3.7. 6 | 7 | ## Details 8 | 9 | Currently we support Python 3.6+. However 3.6 was EOL in December 10 | 2021, and the Python ecosystem has generally moved away from offering 11 | 3.6 support. Practically this means that we have been unable to update 12 | many dependencies which depend on 3.7+. 13 | 14 | Previously the blocker for updates was the downstream integration with 15 | Gecko's CI which still used Python 3.6. This has recently been updated 16 | to use [3.7](https://bugzilla.mozilla.org/show_bug.cgi?id=1734402), 17 | which unblocks updating to that version. 18 | 19 | Practically this requires several manual changes: 20 | 21 | * Change the CI jobs running under Python 3.6 to instead use 3.7. 22 | * Update any outdated vendored dependencies that are being held back 23 | due to lack of 3.6 support. 24 | * Retrigger CI runs for dependabot PRs. Hopefully this allow many 25 | dependencies to be updated, although there may be some for which 26 | the latest versions require 3.8+. In these cases we should manually 27 | try to update to the latest 3.7 compatible release. 28 | 29 | 30 | ## Risks 31 | 32 | 3.7 itself is near the end of its support period, so there is some 33 | risk that we end up in a similar situation again soon, with libraries 34 | requiring 3.8+. However given the requirement to support vendors, and 35 | the fact that Gecko depends on 3.7, bumping the minimum version 36 | further is not possible at this time. 37 | 38 | Some unknown external users may depend on 3.6 support. This is 39 | hopefully reasonably unlikely as they would both need to be using wpt 40 | in a large system where Python updates are challenging, and also not 41 | be participating in the RFC process. However if this does happen we 42 | will need to work with the external user to determine a viable path 43 | forward. 44 | -------------------------------------------------------------------------------- /rfcs/python_38.md: -------------------------------------------------------------------------------- 1 | # RFC 189: Bump minimum supported Python to 3.8 2 | 3 | ## Summary 4 | 5 | Increase the minimum supported Python version to 3.8. 6 | 7 | ## Details 8 | 9 | Currently we support Python 3.7+. However 3.7 was EOL in June 2023, 10 | and the Python ecosystem has generally moved away from offering 11 | 3.7 support. Practically this means that we have been unable to update 12 | many dependencies which depend on 3.8+. 13 | 14 | Previously the blocker for updates was the downstream integration with 15 | Gecko's CI which still used Python 3.7. This has recently been updated 16 | to use [3.8](https://bugzilla.mozilla.org/show_bug.cgi?id=1843209), 17 | which unblocks updating to that version. 18 | 19 | Practically this requires several manual changes: 20 | 21 | * Change the CI jobs running under Python 3.7 to instead use 3.8. 22 | * Update any outdated vendored dependencies that are being held back 23 | due to lack of 3.7 support. 24 | * Retrigger CI runs for dependabot PRs. Hopefully this allow many 25 | dependencies to be updated, although there may be some for which 26 | the latest versions require 3.9+. In these cases we should manually 27 | try to update to the latest 3.8 compatible release. 28 | 29 | ## Risks 30 | 31 | 3.8 itself is near the end of its support period (October 2024), so there 32 | is some risk that we end up in a similar situation again soon, with libraries 33 | requiring 3.9+. However given the requirement to support vendors, and 34 | the fact that Gecko depends on 3.8, bumping the minimum version 35 | further is not possible at this time. 36 | 37 | Some unknown external users may depend on 3.7 support. This is 38 | hopefully reasonably unlikely as they would both need to be using wpt 39 | in a large system where Python updates are challenging, and also not 40 | be participating in the RFC process. However if this does happen we 41 | will need to work with the external user to determine a viable path 42 | forward. 43 | -------------------------------------------------------------------------------- /rfcs/quic.md: -------------------------------------------------------------------------------- 1 | # RFC 42: Add an optional QUIC server to wpt 2 | 3 | ## Summary 4 | 5 | Add an *optional*, *off-by-default* [QUIC](https://quicwg.org/) server to wpt, 6 | implemented with [aioquic](https://github.com/aiortc/aioquic) in Python 3. This 7 | server is used to test web APIs that rely on the QUIC protocol (e.g. 8 | [WebTransport](https://wicg.github.io/web-transport/)). If the server is not 9 | enabled, `wptrunner` will skip the tests that rely on QUIC. 10 | 11 | QUIC custom handlers use a new filename flag `.quic` (`.quic.py`). And a meta 12 | tag is included by tests that require the QUIC server for easy filtering. 13 | 14 | This RFC focuses on the high-level architecture and dependency management to 15 | enable prototyping the QUIC server. The server APIs are not yet finalized, but 16 | we commit to filing another RFC to discuss and finalize the API design once it 17 | is ready. 18 | 19 | See the [original issue][original-issue] for more background. 20 | 21 | ## Details 22 | 23 | ### Requirements 24 | 25 | 1. The QUIC server requires decent cryptography support; pure Python isn't 26 | sufficient. 27 | 2. The server needs to be scriptable, and the custom handlers should live in 28 | the wpt repo next to test files, so that they can be updated together. 29 | 3. An async programming model is preferred. 30 | 31 | ### Architecture 32 | 33 | The QUIC server has a standalone CLI, and is managed by `wpt serve` as a 34 | subprocess. Configurations are stored in `config.json` (same as `wptserve`), and 35 | `wpt serve` is responsible for passing the relevant configurations (e.g. port 36 | numbers) to the QUIC server via command line flags. `wpt serve` handles the 37 | error gracefully if the QUIC server cannot be started. 38 | 39 | The QUIC server is written in Python 3 and makes no attempt to support Python 2. 40 | Its entrypoint will have a shebang that explicitly requires `python3`. An 41 | implication is that the QUIC server will be relatively self contained, without 42 | dependencies on other modules in wpt that are currently Python 2-only. 43 | 44 | Code for the QUIC server itself will live in `tools/quic`. 45 | 46 | ### User interface 47 | 48 | For **test writers**: the QUIC server will support Python custom handlers 49 | similar to `wptserve`. The APIs are not yet finalized and will likely be 50 | different. To more easily distinguish the two, we add a filename flag to QUIC 51 | custom handlers (`.quic.py`). These handlers will be ignored by `wptserve`. 52 | Tests that require the QUIC server need to be marked with a meta tag 53 | (tentatively `` and `// META: quic=true`). This 54 | information will be extracted by `wpt manifest` into `MANIFEST.json`, so that 55 | `wptrunner` can skip the tests that require the QUIC server easily. A lint rule 56 | will be added to require this meta tag if `.quic.py` appears in the file, even 57 | though it cannot prevent all accidental dependency of the QUIC server (e.g. 58 | through a resource script). 59 | 60 | For **test runners**: the goal is to not introduce any observable changes to 61 | users not yet concerned with QUIC or WebTransport. All existing tests will 62 | continue to run without any infra changes, and all new QUIC-dependent tests will 63 | be skipped by default. To enable the QUIC server, users need to pass a flag to 64 | `wpt run` (or `wpt serve`). `wpt` front-end will then attempt to download the 65 | the dependencies (wheels) from PyPI into `_venv`; maintainers of infrastructures 66 | where PyPI cannot be used need to manage dependencies themselves (see the next 67 | section). 68 | 69 | ### Notable dependencies 70 | 71 | * [sys] Python 3 (>=3.6) 72 | * [py] aioquic 73 | * [py] cryptography 74 | * [sys] openssl (>=1.0.1, already required by [HTTP/2](http2.md)) 75 | * [py] pylsqpack 76 | 77 | All of the Python dependencies above have native extensions, which requires 78 | users to use a platform supported by the prebuilt wheels from PyPI or build and 79 | distribute the wheels themselves (e.g. in Chromium through 80 | [cipd](https://chromium.googlesource.com/chromium/src/+/master/docs/cipd.md)). 81 | 82 | The source code of all dependencies will also be vendored into 83 | `tools/third_party` to allow building from source if needed. 84 | 85 | All Python dependencies are BSD-licensed. 86 | 87 | ### Alternatives considered 88 | 89 | We have also [considered][original-issue] writing the QUIC server in other 90 | languages (e.g. Go) and publishing a statically linked binary that would be 91 | downloaded by `wpt` similar to how WebDriver binaries are installed. This would 92 | incur a smaller overhead on the wpt repo in terms of new dependencies, but would 93 | make writing [scriptable custom handlers](#requirements) harder -- we would need 94 | to build an interface between the server and the handler scripts (which will 95 | likely be in a different language, like Python). 96 | 97 | Another alternative that avoids mixing Python 2 and 3 is to migrate the whole 98 | `wptserve` to Python 3, but that work has a much bigger scope. 99 | 100 | ## Risks 101 | 102 | The change required in `wpt` is limited, and the QUIC server itself is 103 | standalone, so the maintenance burden of the code base will be low. The 104 | Ecosystem Infra team at Google will be responsible for maintaining this 105 | integration point. 106 | 107 | A new meta tag means a new field in the manifest JSON (only needed for tests 108 | that use the QUIC server), which should be backward compatible. 109 | 110 | The biggest concern moving forward is the management of the binary dependencies. 111 | Whenever a dependency is updated, we will need to notify all vendors who opt 112 | into the QUIC server so that they can prepare the new wheels in their infras. 113 | 114 | [original-issue]: https://github.com/web-platform-tests/wpt/issues/19114 115 | -------------------------------------------------------------------------------- /rfcs/reftest_simplification.md: -------------------------------------------------------------------------------- 1 | # RFC #15: Reftest graph simplification 2 | 3 | ## Summary 4 | 5 | Simplify what we allow within the reftest graph. 6 | 7 | ## Details 8 | 9 | ### Background 10 | 11 | Currently, reftests form a graph which can quickly end up with a large number of pass conditions. 12 | 13 | In rough principle, we load everything with ``/`` into a directed graph (with URLs as vertices, `match`/`mismatch` as edges), and then make every source node (i.e., a node with no incoming edges) into a "test". 14 | 15 | From each test, we enumerate all possible walks from the source node with either no repeating nodes and ending at a sink node (i.e., a node with no outgoing edges), or ending at the first repeated node. Each walk becomes a sequence of ANDs, and at least one walk must be satisfied for the test to pass. 16 | 17 | In reality, we have very little complexity within the graph, as of [a12f37b6a0](https://github.com/web-platform-tests/wpt/tree/a12f37b6a04bdbde0902b0fc36b6a46f4782bd06): we have 15905 tests, and only 253 are more complex than a simple two page comparison. 18 | 19 | Of those 253: 20 | * 118 just have alternates at the top level (the basic OR case), 21 | * 132 are just trails (the basic AND case), 22 | * leaving us with 3 more complex cases (which mix AND and OR). 23 | 24 | However, notably our definition of how we define the pass condition differs from what the [CSS WG had documented for their testsuite](https://wiki.csswg.org/test/format#reference-links) originally defined (in 2011), which the WPT behaviour was originally meant to match. 25 | 26 | This behaviour is that if there is at least one `` at least one must pass, and all `` must pass. 27 | 28 | Of the 118 tests we detect as OR under our current logic, but [85 of these](https://gist.github.com/gsnedders/2ee57070569e177d973a6736f7d278bb#file-only_or_mixed_eq_cond-txt) have a combination of both match and mismatch, which is likely a bug under our current logic, and looking at a mostly random subset of these, it appears that the intention was the CSS WG documented behaviour. 29 | 30 | There are also [7 tests](https://gist.github.com/gsnedders/2ee57070569e177d973a6736f7d278bb#file-only_or_mismatch-txt) that have multiple mismatches; again it appears like the intention was the CSS WG documented behaviour. In both categories, all of these are within `/css`, and most of these tests predate WPT's manifest generation, so were just following the documented CSS WG behaviour. 31 | 32 | ### Proposal 33 | 34 | This RFC contains a number of proposals, given below: 35 | 36 | It is proposed that we get rid of the references-of-references (i.e., the current AND case) as this is the root of a large amount of implementation complexity and the current behaviour of only source nodes being considered tests has confused a number of users. It also requires us to process every file in the repository to update the manifest before we can run a single reftest, which would be desirable to avoid. 37 | 38 | In general, we shall make every file with at least one ``/`` into a test; this will introduce new tests for all current references which currently chain onto further references. 39 | 40 | Further, it is proposed that we change the combinatorial behaviour to the one originally defined by the CSS WG: if there is at least one ``, at least one must pass, and all `` must pass. 41 | 42 | ### Potential future additions 43 | 44 | Some people have raised concerns about the ability to continue to do more complex AND/OR combinations, we could in the future re-introduce support for this through some explicit grouping (into AND) via some attribute on the `link` elements. 45 | 46 | The risk of punting this to be future work is that it makes some tests impossible to write in web-platform-tests in the meantime. However, it has been eight years since the CSS WG defined their format and four years since we started supporting arbitrary reftest chains and it appears nobody has written such a test. Either it turns out this functionality is unneeded or nobody knows about it. 47 | 48 | ## Advantages 49 | 50 | * Avoids needing to create a whole graph to find/execute reftests, 51 | * Avoids tests disappearing when they are referenced elsewhere in the graph. 52 | * Makes it simpler to explain failure to users. 53 | 54 | ## Disadvantages 55 | 56 | * Loses the chains of failure when a reference renders incorrectly. 57 | -------------------------------------------------------------------------------- /rfcs/reftest_variants.md: -------------------------------------------------------------------------------- 1 | # RFC 149: Support reftest variants 2 | 3 | ## Summary 4 | 5 | Currently, there is no support for [variants] for reftests. This RFC proposes 6 | that we should support reftest [variants]. 7 | 8 | ## Details 9 | 10 | When writing reftests for anything that may have an argument the writer of the 11 | test is limmited to either creating one large test that contains several 12 | subtests within it, or creating several largely similar tests with minor 13 | differences. As an example, consider the `` element and the test created 14 | in the [reftest tutorial]. To test both `dir="rtl"` and `dir="ltr"` for the 15 | `` element, two separate tests or one less descriptive test with both 16 | variants must be created. 17 | 18 | The proposal is to change the reftest manifest to allow for reftests to specify 19 | variants via a `` tag, as currently described in 20 | the [variants] section of the documentation. 21 | 22 | ### Example 23 | 24 | Again consider the [reftest tutorial] example of the `` element. With 25 | reftest variants, the test could look like the following: 26 | 27 | ``` 28 | 29 | 30 | 31 | BDO element 32 | 33 | 34 | 35 | 36 | 37 | 38 |

Test passes if displayed correctly.

39 | SAW 40 | 50 | 51 | ``` 52 | 53 | With reftest variants, the test reference could look like the following: 54 | 55 | ``` 56 | 57 | 58 | 59 | BDO element reference 60 | 61 |

Test passes if displayed correctly.

62 |

63 | 87 | 88 | ``` 89 | 90 | ## Risks 91 | 92 | This change could encourage reftest authors to create more tests and test 93 | references that rely on [reftest-wait] and are generated by a script. It may be 94 | harder to evaluate the expected behavior of a test when viewing the reference 95 | for such tests. 96 | 97 | [reftest tutorial]: https://web-platform-tests.org/writing-tests/reftest-tutorial.html#writing-the-test-file 98 | [variants]: https://web-platform-tests.org/writing-tests/testharness.html#variants 99 | [reftest-wait]: https://web-platform-tests.org/writing-tests/reftests.html#controlling-when-comparison-occurs 100 | -------------------------------------------------------------------------------- /rfcs/remove_ie_edge_legacy.md: -------------------------------------------------------------------------------- 1 | # RFC 179: Remove support for Internet Explorer and Microsoft Edge Legacy 2 | 3 | ### Summary 4 | 5 | Remove support for running WPT tests in Internet Explorer and Microsoft Edge 6 | Legacy. 7 | 8 | ### Details 9 | 10 | Support for the Microsoft Edge Legacy desktop application ended on March 9, 11 | 2021[1], and support for the Internet Explorer desktop application ended on 12 | June 15, 2022[2]. 13 | 14 | The proposal is to remove the `edge`, `edge_webdriver`, and `ie` options from 15 | the product parameter to `wpt run`, along with all related code. 16 | 17 | After removal, the remaining supported products would be: 18 | 19 | ``` 20 | [--product {android_weblayer,android_webview,chrome,chrome_android,chrome_ios, 21 | chromium,content_shell,edgechromium,firefox,firefox_android,safari,sauce,servo, 22 | servodriver,opera,webkit,webkitgtk_minibrowser,wktr,epiphany,ladybird}] 23 | ``` 24 | 25 | [1] https://blogs.windows.com/msedgedev/2021/03/09/microsoft-edge-legacy-end-of-support/ 26 | 27 | [2] https://blogs.windows.com/windowsexperience/2022/06/15/internet-explorer-11-has-retired-and-is-officially-out-of-support-what-you-need-to-know/ 28 | 29 | 30 | ### Risks 31 | 32 | * Someone running WPT tests against these browsers would now be broken. -------------------------------------------------------------------------------- /rfcs/remove_stub.md: -------------------------------------------------------------------------------- 1 | # RFC 27: Remove support for stub-* files in the manifest 2 | 3 | ## Summary 4 | Any files named stub-* are currently treated as "stub" in the manifest. Remove support for this. 5 | 6 | ## Details 7 | Files named stub-* exist only in service-workers/, added in [Initial SW stub addition](https://github.com/web-platform-tests/wpt/commit/5ac8b780247e02235b3d8d344428886b20e5b8b0) and since then not modified in any meaningful way. 8 | 9 | The important effect in the manifest code is that stub-* aren't treated as tests, effectively ignoring them. 10 | 11 | The stub-* files will be removed first, followed eventually by the manifest changes. 12 | 13 | ## Risks 14 | 15 | A MANIFEST.json file now lists the stub-* files under `manifest.items.stub`, where `manifest` is the result of parsing MANIFEST.json. The combination of an existing manifest and the new manifest code has the following risks. 16 | 17 | ### stub-* files treated as tests 18 | 19 | The files will be removed first, and that change allowed to propagate to all downstream users of WPT, before the manifest code change is made. 20 | 21 | ### Failure to do an incremental update 22 | 23 | It will be confirmed that any existing `manifest.items.stub` object is dropped when doing an incremental update. Keeping it would likely be harmless, but an incremental update should always produce the same result as a full update. 24 | -------------------------------------------------------------------------------- /rfcs/rename_lint_whitelist.md: -------------------------------------------------------------------------------- 1 | # RFC 51: Rename lint.whitelist to lint.ignore 2 | 3 | ## Summary 4 | 5 | Rename lint.whitelist to lint.ignore, to accurately reflect the purpose of the 6 | list and avoid using 'white' as a synomnym for 'good'. See the [WHATWG style 7 | guide](https://whatwg.org/style-guide#tone) note and the similar [Google 8 | documentation style 9 | guide](https://developers.google.com/style/word-list#blacklist). 10 | 11 | ## Details 12 | 13 | Aside from renaming the file, `tools/lint/lint.py` will need minor 14 | modifications to look for the correct default file, and the 15 | [web-platform-tests.org](https://web-platform-tests.org) documentation will 16 | need updating. 17 | 18 | ## Risks 19 | 20 | Known risks: 21 | 22 | 1. Downstream consumers or infrastructure may rely on the name of our lint 23 | file. Neither Gecko and Chromium appear to rely on it, and so the risk here 24 | is low. 25 | 1. There may be some mental overhead in people adapting to the new name. 26 | -------------------------------------------------------------------------------- /rfcs/retry_unexpected.md: -------------------------------------------------------------------------------- 1 | # RFC 119: Allow wptrunner to retry tests with unexpected results 2 | 3 | ## Summary 4 | 5 | Add a `--retry-unexpected=` option to wptrunner to retry each test with 6 | unexpected initial results up to `n` times or until it runs as expected. 7 | `n` defaults to zero. 8 | 9 | ## Details 10 | 11 | ### Rationale 12 | 13 | Retries are intended for CI systems that should be robust to false positives 14 | that unnecessarily block developers: 15 | * Transient infrastructure issues (e.g., CPU utilization spike causing 16 | timeouts). 17 | * Incomplete expectation data checked into the trunk of a browser vendor's 18 | source control system. 19 | The CI check might not surface statuses that should have been added. 20 | 21 | ### Proposal 22 | 23 | To gather initial results, wptrunner will run tests according to `--repeat` 24 | and `--rerun` as it does currently. 25 | When passed `--retry-unexpected`, wptrunner will retry tests that gave an 26 | unexpected test or subtest status in every repeat iteration. 27 | In a retry, a test takes any expected status as its final result and is not 28 | retried in subsequent iterations. 29 | The retry loop can halt early if all tests have been exonerated. 30 | wptrunner can return a successful exit code if retries were enabled and all 31 | tests eventually ran as expected. 32 | 33 | Because the purpose of retries is to separate flakiness from consistently 34 | unexpected results, there is no need to retry tests that ran as expected in at 35 | least one repeat iteration. 36 | 37 | A retry iteration works as a repeat iteration does (e.g., emits 38 | `suite_{start,end}` log events), except the `rerun` option is forced to 1. 39 | Requiring restarts eliminates bad state carried over as a possible cause of 40 | failure. 41 | 42 | ### Alternatives 43 | 44 | * Using `--repeat` alone and only failing the CI check when a test runs 45 | unexpectedly consistently. 46 | Given that almost all tests should run as expected the first time, this scheme 47 | unnecessarily burdens the CI infrastructure. 48 | * Implement retries in vendor CI systems. 49 | Browser vendors will not benefit from a shared implementation native to 50 | wptrunner. 51 | This also adds complexity downstream for merging test results from multiple 52 | runs. 53 | * A `--no-repeat-expected` that modifies `--repeat` to skip tests that run 54 | expectedly. 55 | Retries have different semantics, including changing when wptrunner exits with 56 | a successful status code. 57 | 58 | ## Risks 59 | 60 | Minimal, since this is an opt-in feature. 61 | Existing behavior will not change. 62 | 63 | ## Appendix 64 | 65 | [Prototype](https://github.com/web-platform-tests/wpt/compare/master...jonathan-j-lee:wpt:no-repeat-expected) 66 | -------------------------------------------------------------------------------- /rfcs/rfc183_remove_unused_dump_test_results_function_in_testharnessreport_js.md: -------------------------------------------------------------------------------- 1 | # RFC 183: Remove experimental `dump_test_results` function in 2 | 3 | ## Summary 4 | That function is still called but the element it creates is never accessed. The created element contains stringified JSON of test result data. 5 | PR [^1] proposes to remove the function. It was added in 2015 with the comment "This is experimental until we have a better way to integrate with saucelabs". 6 | PR [^2] reduces the performance of the generating the test results' summary table which is consumed by `dump_test_results`. 7 | If that function is never used, degraded performance doesn't matter. 8 | 9 | ## Details 10 | See [^3] and [^4]. 11 | 12 | ## Risks 13 | None known as the element generated from the function seems unused in WebKit, Gecko and Chromium; see [^3]. 14 | It could be used in other downstream-projects, though. 15 | 16 | [^1]: https://github.com/web-platform-tests/wpt/pull/45003 17 | [^2]: https://github.com/web-platform-tests/wpt/pull/45002 18 | [^3]: https://github.com/web-platform-tests/wpt/pull/45003#issue-2176117406 19 | [^4]: https://github.com/web-platform-tests/wpt/issues/44352#issuecomment-1985862862 20 | -------------------------------------------------------------------------------- /rfcs/rfc_as_pr.md: -------------------------------------------------------------------------------- 1 | # RFC 9 2 | 3 | ## Summary 4 | 5 | Use PRs rather than issues for the RFC process. 6 | 7 | ## Details 8 | 9 | Each RFC should be a markdown-formatted document in the rfcs/ subdirectory. Creating the PR will signal the the RFC is ready for review and merging it will indicate that it's accepted. Conversely closing it will be used to indicate the RFC is rejected. 10 | The RFC document should get a title that is its PR number (added after submission) and consists of a summary of the proposed changes, full details of the proposal and a discussion of pros and cons. 11 | 12 | ## Advantages 13 | 14 | * PRs involve an actual document that can be revised after change and still has the version control expected of a git repo. Although this is possible with comments, it's much less expected that a user would have to examine the history of a comment to understand the discussion, and the tooling for exploring that history is weaker. 15 | * Commiting the RFC produces a record of decisions that are made. 16 | * Allowing people to clone the repo and use existing RFCs as a template helps ensure a consistent structure. 17 | * Comments on specific spec can be made inline on that text. 18 | 19 | ## Disadvantages 20 | * The overhead of creating a PR is slightly higher than for creating an issue (although both can be done fully in the GitHub UI). 21 | * Seeing the rendered PR requires extra work in the UI. 22 | * No template is available in the UI for the initial file content analogous to the new issue template. 23 | -------------------------------------------------------------------------------- /rfcs/rfc_exemption_testdriver_method.md: -------------------------------------------------------------------------------- 1 | # RFC 127: Add RFC Exemption for testdriver.js methods similar to WebDriver endpoints 2 | 3 | ## Summary 4 | 5 | To expedite changes and reduce developer friction, the web-platform-tests 6 | community should consider allowing a developer to bypass the RFC process if the 7 | developer is only extending testdriver.js with a method similar to an existing 8 | WebDriver endpoint. 9 | 10 | ## Details 11 | 12 | During the 2022-10-04 [meeting](https://github.com/web-platform-tests/wpt-notes/blob/master/minutes/2022-10-04.md), 13 | attendees discussed RFC [#121](https://github.com/web-platform-tests/rfcs/pull/121) 14 | and similar changes to testdriver.js. Attendees decided that RFC 121 could land 15 | automatically if the WebDriver changes landed first. 16 | 17 | Scaling the learnings from that, this RFC proposes that the README should add a 18 | RFC exemption for `extending testdriver.js with a method that closely matches a WebDriver endpoint`. 19 | 20 | To make the reviewer aware of the type of change, PR author should add the 21 | `testdriver.js` label to the PR. 22 | 23 | ## Risks 24 | 25 | Low/minimal risks: 26 | - A developer trying to extend testdriver.js with a method that does not match 27 | an existing WebDriver endpoint. 28 | - Mitigation: 29 | - The PR review process can catch this if the WebDriver endpoint already 30 | exists. -------------------------------------------------------------------------------- /rfcs/shadowrealm-global.md: -------------------------------------------------------------------------------- 1 | # RFC 107: Add support for Shadow Realms as Javascript global 2 | 3 | ## Summary 4 | 5 | [Shadow realms](https://github.com/tc39/proposal-shadowrealm) are a new 6 | sandboxing primitive, currently a stage 3 proposal in TC39 and in the process of 7 | being integrated into relevant Web specifications; part of this extends some 8 | pure/non-rendering/purely computational Web interfaces to shadow realms (which 9 | otherwise do not have browser APIs and more closely resemble a vanilla JS shell 10 | environment.) 11 | 12 | I think it makes sense to extend WPT tests for these interfaces to 13 | also run in shadow realm contexts, so we can be assured they still work there, 14 | therefore ... 15 | 16 | This RFC proposes adding a new valid global for `.any.js` tests, `shadowrealm` 17 | that runs the test harness in a shadow realm under a regular window environment. 18 | 19 | ## Details 20 | 21 | To accomplish this, I add a new `ShadowRealmTestEnvironment` to `testharness.js` 22 | that reuses much of the existing machinery to run tests remotely in e.g. 23 | dedicated workers. There are however, two main differences: 24 | 25 | 1. Shadow realms are not DOM contexts and so have no onload or similar event for 26 | us to use to actually begin test aggregation; instead, I add a new hook for 27 | the incubating realm to directly call to signal that all desired tests have 28 | been loaded. 29 | 30 | 2. The actual message port used to communicate test results back to a 31 | RemoteContext requires JSON-serialization of the mesasges (primitive values 32 | and callables may cross between a shadow- and incubating realm, but not 33 | arbitrary objects): it happens that this works correctly given the current 34 | encodings of `Test`, harness state, etc. seem to be JSON-round-trippable, but 35 | this would become a hard requirement from this point forward. 36 | 37 | A proposed patch is available at https://github.com/web-platform-tests/wpt/pull/33162 38 | 39 | ## Risks 40 | 41 | There's two concerns I have with the current proposed implementation: 42 | 43 | 1. It's unclear (to me) given the current state of the way Shadow Realms are 44 | integrated into the HTML spec how we might be able to deal with unhandled 45 | promise rejections. It happens, for now, that they will be handed by the 46 | incubating Window's `unhandledrejection` handler, which is installed by the 47 | parent test harness, and prints to the console. This seems acceptable for the 48 | time being, but may need to change as the spec is clarified. 49 | 50 | 2. Detecting whether the test harness is being run in a shadow realm is not 51 | straightforward--Shadow realm global objects are, by design, bare-bones. 52 | Their prototype is `Object.prototype` and contain only a handful of Web 53 | interfaces; the patch above implements a crude test by checking for the 54 | presence of `AbortController` if `globalThis` has been determined not to be a 55 | window or worker; if we do not require compatibility with e.g. nodejs, this 56 | seems fine, if, perhaps, fragile. A mechanism to pass information about the 57 | environment into the testharness early in the lifecycle would be needed to 58 | avoid this. 59 | -------------------------------------------------------------------------------- /rfcs/simplify_license.md: -------------------------------------------------------------------------------- 1 | ## RFC #18: Simplify License 2 | 3 | ### Summary 4 | 5 | Reduce WPT's license to the 3-Clause BSD License and update the copyright 6 | holder to "web-platform-tests contributors." 7 | 8 | ### Details 9 | 10 | WPT is currently dual-licensed under [the W3C Test Suite 11 | License](#w3c-test-suite-license) and [the W3C 3-clause BSD 12 | License](#w3c-3-clause-bsd-license). Both licenses name "W3C" as the copyright 13 | holder. 14 | 15 | In [WPT issue gh-11009](https://github.com/web-platform-tests/wpt/issues/11009) 16 | Philippe Le Hegaret (W3C) led a discussion about how this dual license is no 17 | longer appropriate for the web-platform-tests project. Wendy Seltzer (W3C) 18 | proposed changes to the Patents and Standards Interest Group (PSIG) and 19 | reported: 20 | 21 | > Per conversation with [Philip Jägenstedt (Google)] I'm filing two pull 22 | > requests against license.md, one dropping from the dual license to the W3C 23 | > 3-clause BSD; the other replacing with the straight 3-clause BSD (which 24 | > differs only in not having the named copyright holder W3C). PSIG has given 25 | > non-objection to the W3C 3-clause BSD. If that would still cause legal 26 | > hiccups, then we can probably persuade them of the vanilla 3-clause. 27 | 28 | Because [Philippe Le Hegaret and Philip Jägenstedt have made it known that 29 | naming "W3C" as the copyright holder is 30 | undesirable](https://github.com/web-platform-tests/wpt/pull/11191), we choose 31 | to adopt [the 3-Clause BSD 32 | License](https://opensource.org/licenses/BSD-3-Clause) which lists the 33 | copyright holder as "web-platform-tests contributors." The complete license 34 | text is as follows: 35 | 36 | > # The 3-Clause BSD License 37 | > 38 | > Copyright 2019 web-platform-tests contributors 39 | > 40 | > Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 41 | > 42 | > 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 43 | > 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. 44 | > 3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. 45 | > 46 | > THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 47 | 48 | ### Risks 49 | 50 | Pending the endorsement from the W3C Patents and Standards Interest Group, 51 | there appears to be no risk associated with this change. 52 | -------------------------------------------------------------------------------- /rfcs/test_arrow_name.md: -------------------------------------------------------------------------------- 1 | # RFC 64: Use simple arrow funcs as the default test name 2 | 3 | ## Summary 4 | 5 | For `test(() => assert_true(true))`, use `assert_true(true)` as the default test name rather than `document.title` 6 | 7 | ## Details 8 | 9 | At the moment, we practically require all tests to have a name be explicitly given. This makes testharness.js tests much more verbose than WebKit/Blink's js-test.js. For example, currently in WebKit we have: 10 | 11 | ```javascript 12 | const shadowHost = document.createElement("div"); 13 | const shadowRoot = shadowHost.attachShadow({mode: 'closed'}); 14 | const shadowRootChild = shadowRoot.appendChild(document.createElement('div')); 15 | 16 | shouldBeFalse('shadowHost.isConnected'); 17 | shouldBeFalse('shadowRoot.isConnected'); 18 | shouldBeFalse('shadowRootChild.isConnected'); 19 | shouldBeFalse('document.contains(shadowHost)'); 20 | shouldBeFalse('document.contains(shadowRoot)'); 21 | shouldBeFalse('document.contains(shadowRootChild)'); 22 | ``` 23 | 24 | In WPT, the shortest way to write this is: 25 | 26 | ```javascript 27 | test(() => assert_false(shadowHost.isConnected), 'shadowHost.isConnected'); 28 | test(() => assert_false(shadowRoot.isConnected), 'shadowRoot.isConnected'); 29 | test(() => assert_false(shadowRootChild.isConnected), 'shadowRootChild.isConnected'); 30 | test(() => assert_false(document.contains(shadowHost)), 'document.contains(shadowHost)'); 31 | test(() => assert_false(document.contains(shadowRoot)), 'document.contains(shadowRoot)'); 32 | test(() => assert_false(document.contains(shadowRootChild)), 'document.contains(shadowRootChild)'); 33 | ``` 34 | 35 | This is comparatively rather verbose, and has been cited as a reason to not adopt testharness.js tests. 36 | 37 | This proposal would make the default test names `assert_false(shadowHost.isConnected)`, etc., rather than whatever the `title` element contains. This would only apply when the test function is an arrow function with no arguments where the body contains no new lines, as determined from `Function.prototype.toString`. As a result, the following becomes possible: 38 | 39 | ```javascript 40 | test(() => assert_false(shadowHost.isConnected)); 41 | test(() => assert_false(shadowRoot.isConnected)); 42 | test(() => assert_false(shadowRootChild.isConnected)); 43 | test(() => assert_false(document.contains(shadowHost))); 44 | test(() => assert_false(document.contains(shadowRoot))); 45 | test(() => assert_false(document.contains(shadowRootChild))); 46 | ``` 47 | 48 | An [implementation](https://github.com/web-platform-tests/wpt/pull/25853) has already been done. 49 | 50 | ## Risks 51 | 52 | Some existing tests relying on `document.title` for their name could have their name change. However, we have few tests that currently match the above. Running the above implementation against Firefox didn't show any test name changes. 53 | 54 | If `Function.prototype.toString` is not sufficiently interoperable then we could end up with different test names across different browsers. However, it is believed that in simple cases, such as those likely to occur on a single line, the risk is minimal. 55 | -------------------------------------------------------------------------------- /rfcs/test_rendered_event.md: -------------------------------------------------------------------------------- 1 | # RFC 34: `reftests`: Fire event when painting is done 2 | 3 | ## Summary 4 | 5 | Fire a `TestRendered` event for reftests when a `reftest-wait` class is present on the root and layout, fonts and painting are complete. This corresponds to the point at which the screenshot would be taken in the absence of `reftest-wait`. 6 | 7 | ## Details 8 | 9 | Reftests which want to check for behaviour in the face of dynamic changes need to know when those changes can be made with a complete initial state. This corresponds to the time that a screenshot would be taken in the case that the `reftest-wait` attribute is not present. Firing an event at this time allows the test to schedule dynamic changes and then remove the `reftest-wait` attribute to trigger a screenshot without concern that the changes are batched into the intial render. 10 | 11 | The detailed steps for running a reftest with this addition are as follows: 12 | 13 | * Wait for the document to complete loading, all fonts to load, and for pending paints to be flushed. 14 | * If the document root element does not contain a class named `reftest-wait`, screenshot the viewport and abort these steps 15 | * Fire an event at the document root with the name `TestRendered` and the bubbles attribute set to true. 16 | * Wait for the `reftest-wait` class to be removed from the document root 17 | * Wait for pending paints to flush 18 | * Screenshot the viewport 19 | 20 | This addition corresponds to the gecko `MozReftestInvalidate` event. Providing this event is expected to unblock upstreaming some more Gecko reftests. 21 | 22 | ## Example 23 | 24 | Adapted from the [gecko docs](https://searchfox.org/mozilla-central/source/layout/tools/reftest/README.txt#509) 25 | 26 | ```js 27 | 28 | function doTest() { 29 | document.body.style.border = ""; 30 | document.documentElement.removeAttribute('class'); 31 | } 32 | document.addEventListener("TestRendered", doTest, false); 33 | ``` 34 | 35 | ## Risks 36 | 37 | No expected risks except those introduced by any feature e.g. complexity. 38 | 39 | The event itself may not be enough to test invalidation codepaths in browsers; browser-specific approaches may be required to do that. However it's a prerequisite for that work. 40 | -------------------------------------------------------------------------------- /rfcs/test_scoring_change.md: -------------------------------------------------------------------------------- 1 | # RFC: Change results aggregation on wpt.fyi and visualization on Interop-20** views 2 | 3 | ## Summary 4 | 5 | The aggregate scoring that is displayed in the results view of wpt.fyi is 6 | suboptimal in some aspects. This proposal will describe a change to the way 7 | that test results are aggregated, as well as a change to how those results are 8 | displayed for Interop 2022 scoring purposes. 9 | 10 | ## Details 11 | There are a number of aspects to this change. Each subsection will describe 12 | these aspects in detail. 13 | ### 1. Remove harness status from subtest aggregation 14 | 15 | When interpreting the results of test suite runs, a status is present that 16 | represents either the overall test's status or the "harness status", which 17 | represents whether any harness errors were present for `testharness.js` test 18 | runs. The current results aggregation method counts this harness status as a 19 | separate subtest itself, which can inflate the overall passing percentage of 20 | a test. There are many instances of a test outright failing its only subtest, 21 | but the test is weighted as 50% passing because there were no harness errors 22 | [(example)](https://wpt.fyi/results/audio-output/selectAudioOutput-sans-user-activation.https.html?run_id=5642791446118400&run_id=5668568732532736&run_id=5739999172493312&run_id=5735075831349248). 23 | The solution proposed is to no longer aggregate this harness status into the 24 | subtest totals. 25 | 26 | ### 2. Display additional information if a harness error exists. 27 | 28 | Although it seems intuitive to simply disregard the harness status when 29 | calculating tests, there are instances where a harness error will occur, which 30 | causes a number of subtests to not be executed. In some circumstances, it is 31 | possible that a harness error will occur, allowing for some subtests to run 32 | and pass as expected, but some subtests to be not run and not be registered, 33 | as they are not present during aggregation. This has the possibility of hiding 34 | the scope of failure for a specific test in this scenario. The solution 35 | proposed is to mark these tests with harness errors with some additional 36 | visual notice that this a harness error occurred. This is to signify that 37 | the results should not be taken at face value and engineers should investigate 38 | the test further. 39 | 40 | ### 3. Implement optional display of Interop-20xx scores for labelled tests 41 | 42 | Currently, wpt.fyi displays a fraction of the number of subtests that pass and 43 | the number of subtests that exist for each test or every test that exists in a 44 | directory. This view is not necessarily indicative of Interop scores and it 45 | can be unclear to engineers what aspects are affecting scores easily. The 46 | proposed change is to have these Interop views better match their actual score 47 | weights for easier understanding. This can be handled by aggregating by tests 48 | rather than by subtests. 49 | 50 | Passing percentage for a single test = `passing subtests / total subtests` 51 | 52 | Passing percentage for directories containing multiple tests = 53 | `sum of passing percentages of tests in directory / total number of tests in directory` 54 | 55 | This will be implemented by adding a new query parameter, `view`. This can have 56 | two valid values: `subtest` and `interop`. `subtest` will represent the current 57 | view that exists on wpt.fyi and will be enabled by default. `interop` will 58 | display a view that more closely mirrors the Interop-20** scores. 59 | 60 | **Note**: In the interest of avoiding scored comparisons being visible outside 61 | the tests that have been agreed upon for Interop-20**, this `interop` view will 62 | only be available when viewing tests with an `Interop-20**` test label. The 63 | `interop` view will not function outside of results without this label. 64 | 65 | ## Risks 66 | * This change will cause a discrepancy in the number of subtests when comparing 67 | runs with old summary files and new summary files together, as the harness 68 | status passing has been artificially inflating some subtest numbers. This is a 69 | rework of the current process for aggregating and summarizing test results, 70 | so older test runs will still be aggregated with the harness status. 71 | -------------------------------------------------------------------------------- /rfcs/testdriver-features.md: -------------------------------------------------------------------------------- 1 | # RFC 214: Add testdriver features 2 | 3 | ## Summary 4 | 5 | This RFC is an alternative to [RFC 212](https://github.com/web-platform-tests/rfcs/pull/212) providing a more generic and extendable solution. It proposes adding a query parameter `?feature` to `testdriver.js` and adding a `testdriver_features` property to the [TestharnessTest](https://github.com/web-platform-tests/wpt/pull/49122/files#diff-8f280b64b0700ab9b4b343adabc7ff4d5ea4b1f45cb6c4bd7f50a19b21ebdb8cR169). At the moment the only supported feature will be `bidi`, which will be used to enable the [testdriver BiDi API](https://github.com/web-platform-tests/rfcs/blob/master/rfcs/testdriver_bidi.md) for specific WPT tests. The feature `bidi`, applicable to HTML and JS files, would let test runners choose between [**WebDriverProtocol**](https://github.com/web-platform-tests/wpt/pull/49122/files#diff-c2f15328bc1ddfa8fb93c09ae41651e2bcfdad4f257932263a3e92c3f8deffceR277) and [**WebDriverBidiProtocol**](https://github.com/web-platform-tests/wpt/pull/49122/files#diff-c2f15328bc1ddfa8fb93c09ae41651e2bcfdad4f257932263a3e92c3f8deffceR279). This approach does not require all the browsers to implement WebDriver BiDi. 6 | 7 | The query parameter `feature` takes a single value, which is a string representing a feature name. Feature names are compared by case-sensitive string matching. Multiple features are represented as multiple `feature` parameters for example `testdriver.js?feature=bidi&feature=SOME_OTHER_FEATURE`. This RFC only defines the `bidi` feature; future RFCs may add additional features. 8 | 9 | An implementation may define extension features. These features must have a single colon ":" character. The part before the colon is the prefix; this is typically the same for all extensions specific to a given implementation and should be unique for a given implementation. For example `testdriver.js?feature=VENDOR_PREFIX:VENDOR_SPECIFIC_FEATURE`. 10 | 11 | This approach is preferable over the [RFC 212](https://github.com/web-platform-tests/rfcs/pull/212), as it is more ergonomic and supports future extensions ergonomically. The list of supported features can be extended when required. 12 | 13 | [Prototype](https://github.com/web-platform-tests/wpt/pull/49122). 14 | 15 | ## Background 16 | 17 | [RFC 185](https://github.com/web-platform-tests/rfcs/blob/master/rfcs/testdriver_bidi.md) proposes adding BiDi support to `testdriver.js`, which involves using [**WebDriverProtocol**](https://github.com/web-platform-tests/wpt/blob/f54cd920024da0857fdc5b036f16a7d1fd8792fd/tools/wptrunner/wptrunner/executors/executorwebdriver.py#L587) instead of [**WebDriverBidiProtocol**](https://github.com/web-platform-tests/wpt/blob/f54cd920024da0857fdc5b036f16a7d1fd8792fd/tools/wptrunner/wptrunner/executors/executorwebdriver.py#L679) for test runners using the WebDriver protocol. However, this change could cause regressions in tests that don't use the testdriver BiDi API due to unintended side effects of the transport change. This RFC proposes a solution for the issue. 18 | 19 | ## Details 20 | 21 | ### Examples 22 | 23 | [html test](https://github.com/web-platform-tests/wpt/blob/117958ef4317d8ed16e9ea6ae63da30262f3b875/infrastructure/webdriver/bidi/subscription.html#L6) 24 | ```javascript 25 | ... 26 | 27 | ... 28 | ``` 29 | [javascript test](https://github.com/web-platform-tests/wpt/blob/117958ef4317d8ed16e9ea6ae63da30262f3b875/infrastructure/webdriver/bidi/subscription.window.js#L2) 30 | ```javascript 31 | ... 32 | // META: script=/resources/testdriver.js?feature=bidi 33 | ... 34 | ``` 35 | 36 | ### Required changes 37 | 38 | #### Configure testdriver.js based on the `?feature` flag 39 | 40 | The `testdriver.js` checks if the `?feature=bidi` query parameter was included using [document.currentScript.src](https://developer.mozilla.org/en-US/docs/Web/API/Document/currentScript) like in the [prototype](https://github.com/web-platform-tests/wpt/pull/49122/files#diff-1fe2b624679a3150e5c86f84682c5901b715dad750096a524e8cb23939e5590fR16). If `bidi` was not enabled, a [descriptive error will be thrown](https://github.com/web-platform-tests/wpt/pull/49122/files#diff-1fe2b624679a3150e5c86f84682c5901b715dad750096a524e8cb23939e5590fR22) when testdriver Webdriver BiDi API is used. The same check should be applied for any new features. 41 | 42 | #### Add `testdriver_features` to TestharnessTest 43 | 44 | This RFC proposes adding `testdriver_features` to [`TestharnessTest`](https://github.com/web-platform-tests/wpt/pull/49122/files#diff-8f280b64b0700ab9b4b343adabc7ff4d5ea4b1f45cb6c4bd7f50a19b21ebdb8cR169) so that executors can get the list of features declared by a test. The list is populated by parsing the `feature` query param from the `testdriver.js` imports used by a test. Currently, only the `bidi` feature is supported. 45 | 46 | #### (implementation-specific) Set up WebDriver BiDi in the test executor if needed 47 | 48 | Even though the `TestExecutor` needs information about whether to activate BiDi or not at `connect`, which happens before the runner receives the test to run, the configuration can be done at the point [the browser is configured](https://github.com/web-platform-tests/wpt/pull/49122/files#diff-0df24b5b583c460182e687f7dc7a6a79dd2cd3389bc4a96f48483f60fceb51f7R275). This data can be used later by the specific executor implementation to [decide which protocol to use](https://github.com/web-platform-tests/wpt/pull/49122/files#diff-0df24b5b583c460182e687f7dc7a6a79dd2cd3389bc4a96f48483f60fceb51f7R275). 49 | 50 | ## Risks 51 | 52 | ### Having `testdriver_features` in the `Test` is not sufficient 53 | 54 | `TestExecutor` needs information about whether to activate BiDi or not at `connect`, which happens before the runner receives the test metadata. This can be addressed by passing the data to implementation-specific `TestharnessExecutor` via the implementation-specific `BrowserSettings` and like in the [example](https://github.com/web-platform-tests/wpt/pull/49122/files#diff-0df24b5b583c460182e687f7dc7a6a79dd2cd3389bc4a96f48483f60fceb51f7R269). 55 | 56 | ### `testdriver.js` is included in a subresource 57 | 58 | The use of `testdriver.js` within a subresource presents a challenge. Both the main resource and the subresource must specify the same features, which is hard to verify due to the lack of a complete picture of subresource usage. The most problematic scenario would be if the subresource enabled bidi APIs without the top-level test opting in (e.g., in tests against implementations where the flag is ineffective in wptrunner). Currently, this is preventable because all testdriver commands go through the top-level script, but once BiDi can be used for wptrunner itself, it could become a problem for other features. 59 | 60 | ## Alternatives considered 61 | 62 | List of other alternatives is in the [RFC 212](https://github.com/web-platform-tests/rfcs/pull/212). 63 | 64 | ### Always provide `bidi` API in testdriver 65 | 66 | In this alternative, the testdriver does not raise an error if the bidi feature was not enabled.This creates a risk of potential false-negative test results, if the test author forgets to add the `feature=bidi` and check the test only on the browser implementation which supports WebDriver BiDi by-default, regardless of the feature. 67 | -------------------------------------------------------------------------------- /rfcs/testdriver_cookie_methods.md: -------------------------------------------------------------------------------- 1 | # RFC 74: Add testdriver.js support for WebDriver delete_all_cookies commands 2 | 3 | ## Summary 4 | 5 | Add testdriver.js support for the [WebDriver `delete_all_cookies` command](https://w3c.github.io/webdriver/#delete-all-cookies). 6 | 7 | ## Details 8 | 9 | There are many different helper methods in WPT for setting, getting, and deleting cookies(see [cookies/resources](https://github.com/web-platform-tests/wpt/tree/master/cookies/resources), or the cookie-related `Response` methods for some examples). Some of them work well, 10 | and others are prone to race conditions, or [don't cover all use cases](https://github.com/web-platform-tests/wpt/blob/d1aaf685cfb02e0350701550afe4146d555f2461/cookies/resources/cookie.py#L13-L17). 11 | 12 | Rather than re-write existing support code to fix these issues, this RFC aims to add support for the WebDriver `delete_all_cookies` command. An PR exists at https://github.com/web-platform-tests/wpt/pull/26723`, as it addresses a known issue of [flaky leftover state](https://github.com/web-platform-tests/wpt/issues/26707). 13 | 14 | Future RFCs may propose adding the rest of the cookie commands, once [cookie serialization is defined by WebDriver](https://github.com/w3c/webdriver/issues/1562). That will allow writing cookie tests independent of HTTP or DOM cookie APIs (i.e., `Set-Cookie`, `document.cookie`, `CookieStore`). 15 | 16 | ## Risks 17 | 18 | This increases the API surface of testdriver.js, so it's more code to maintain or reason about. In addition, the developer experience of running local cookie tests will be less convenient (requiring `wpt run` instead of just hitting refresh). -------------------------------------------------------------------------------- /rfcs/testdriver_cross_origin.md: -------------------------------------------------------------------------------- 1 | # RFC 63 2 | 3 | ## Summary 4 | Support running testdriver commands in browsing contexts other than 5 | the test browsing context, including cross origin contexts where it's 6 | possible to get a handle to the browsing context containing the test 7 | (i.e. excluding the rel=noopener case). 8 | 9 | ## Details 10 | 11 | For contexts in which it's possible to get a handle to the context 12 | containing testharness.js ("the test context"), we can support 13 | testdriver interactions by using postMessage to the top-level window 14 | with a message containing the command to run and the context to run it 15 | in. That approach works with the WebDriver-based implementation in 16 | wptrunner where WebDriver can only address a single frame at a time, 17 | and a WebDriver-driven event loop in the test context is used to 18 | invoke testdriver actions and also to report the test results. 19 | 20 | The specific proposed changes are: 21 | 22 | * Allow testdriver.js to be loaded in contexts other than the test 23 | context. 24 | 25 | * In such contexts require use of testdriver to first call the 26 | `test_driver.set_test_context()` function with a handle to the test 27 | context. 28 | 29 | * Also provide a `message_test` function as a wrapper around 30 | postMessage to the test context. 31 | 32 | * Add an optional `context` argument to test_driver commands that 33 | don't depend on a provided element, to specify the context in which 34 | they should run. 35 | 36 | The implementation details for wptrunner are as follows: 37 | 38 | * testdriver events are given a command id and a context argument. The 39 | command id is used to match the response from webdriver to the 40 | specific request. The context id is used to identify which frame or 41 | window should be used for the request. 42 | 43 | * Ideally the context id would be the wptrunner window/frame handle 44 | for the window or frame. But this is 45 | [poorly supported](https://wpt.fyi/results/webdriver/tests/execute_script/json_serialize_windowproxy.py) 46 | so instead we put a `__wptrunner_id` property on the window object, 47 | set it to a uuid, and do a depth-first search for a frame with that 48 | id in the wptrunner code when looking for the context in which to 49 | run the command. 50 | 51 | * WebDriver is switched to the relevant context for the duration of 52 | the action and switched back to the test context once the action is 53 | complete. 54 | 55 | ## Risks 56 | 57 | Adding optional arguments to functions in js is possibly confusing API 58 | design and hard to extend. It might be better to use an options object 59 | as the final argument, but this is arguably harder to use. 60 | 61 | Searching through all contexts is likely slow and could have side 62 | effects if it e.g. changes focus. It's unclear how to do better than 63 | this without fixing the underlying WebDriver issue across multiple 64 | implementations. 65 | 66 | It is likely surprising to users that this would not work in all 67 | cases, and by addressing this problem in this was we are putting off 68 | the work to address it in a way that works for any browsing context. 69 | 70 | Some of the proposed API additions are likely unnecessary if we later 71 | update the implementation to work in all contexts. 72 | -------------------------------------------------------------------------------- /rfcs/testdriver_generate_test_report.md: -------------------------------------------------------------------------------- 1 | # RFC 13 2 | 3 | ## Summary 4 | Add testdriver.js support for the `generate_test_report` WebDriver command, which will allow generic reports to be consistently generated during web platform tests. This command is specced as [part of the Reporting API spec](https://w3c.github.io/reporting/#generate-test-report-command). 5 | 6 | ## Details 7 | The real-world causes of reports (like deprecations, interventions, and crashes) can differ between different user agents. In order to test reporting consistently, we need the ability to generate reports universally across the web platform. This command provides that ability. 8 | 9 | ## Risks 10 | The WebDriver endpoint is new and may still require changes, which would then affect the testdriver.js API as well. 11 | -------------------------------------------------------------------------------- /rfcs/testdriver_get_cookie_methods.md: -------------------------------------------------------------------------------- 1 | # RFC 108: Add testdriver.js support for WebDriver "Get All Cookies" and "Get Named Cookie" commands 2 | 3 | ## Summary 4 | 5 | Add testdriver.js support for WebDriver commands that can be used to get cookies from the browser, ["Get All Cookies"](https://w3c.github.io/webdriver/#get-all-cookies) and ["Get Named Cookie"](https://w3c.github.io/webdriver/#get-named-cookie). 6 | 7 | ## Details 8 | 9 | Some tests that assert how a browser treats certain cookies require a greater 10 | level of introspection into the internal browser storage than is afforded to 11 | websites. One example of this are tests that need to know whether a cookie was 12 | saved with "Session" expiry or whether a certain expiry value was successfully 13 | parsed. Writing those tests in WPT is sometimes impossible. 14 | 15 | In other cases, writing an assertion to check the value of a cookie attribute 16 | may not be impossible, but requires a complex setup of echo servers and 17 | third-party requests that could be frustrating to test authors. 18 | 19 | It would be great if WPT had the ability to call 20 | `test_driver.get_cookies()` instead. 21 | 22 | This has a dependence on agreeing to the right strategy for [cookie serialization 23 | in WebDriver](https://github.com/w3c/webdriver/issues/1562), but there seems to 24 | be agreement on the issue that preserving raw bytes without intermediate UTF-8 25 | parsing seems like the right way to go. 26 | 27 | ## Risks 28 | 29 | As mentioned, cookie serialization isn't fully defined (though loosely agreed 30 | upon) yet. However, an initial attempt at implementation might actually 31 | significantly help inform and speed up the serialization question, so I think 32 | this is a good thing. 33 | 34 | Finally, these risks that were already surfaced in 35 | [RFC 74](https://github.com/web-platform-tests/rfcs/pull/74) apply in this case 36 | as well: 37 | This increases the API surface of testdriver.js, so it's more code to maintain or reason about. 38 | In addition, the developer experience of running local cookie tests will be less convenient (requiring `wpt run`). 39 | 40 | 41 | -------------------------------------------------------------------------------- /rfcs/testdriver_in_other_types.md: -------------------------------------------------------------------------------- 1 | # RFC 211: Support testdriver.js in other test types 2 | 3 | ## Summary 4 | 5 | Allow reftests, print-reftests, and crashtests to use `testdriver.js`, and add 6 | the necessary support to `wpt {run,manifest,lint}`. 7 | 8 | ## Motivation 9 | 10 | [`testdriver.js`][0] provides APIs that allow tests to send WebDriver-like 11 | commands from client-side JavaScript. 12 | Tests often use testdriver.js to access powerful capabilities that would 13 | normally not be automatable because the web platform does not expose them. 14 | 15 | Currently, only [testharness tests] may use testdriver.js. 16 | However, test authors [have expressed interest] in extending testdriver.js 17 | support to [reftests], WPT's main source of rendering coverage. 18 | The features they wish to test with respect to rendering are often gated on 19 | user activation or require pointer input, which can't be obtained without 20 | testdriver.js or manual interaction. 21 | The table below lists some use cases, and (if applicable) tests already written: 22 | 23 | | Tests | testdriver.js usage | Purpose | 24 | | --- | --- | --- | 25 | | [`fullscreen/`][1] | `bless()` | Provide [required transient activation][2] | 26 | | [`customizable-select/`][3] | `bless()`, `click()` | Provide transient activation [required to show `` picker in mainstream browsers is rendered 143 | in a separate popup window that won't be captured in a reftest screenshot. 144 | In those cases, testdriver support can still help drive additional 145 | interoperability and infrastructure improvements that may be required to make 146 | those features fully testable. 147 | * Opening up testdriver to the other test types may encourage authors to write 148 | unnecessarily JavaScript-heavy tests. 149 | Test reviewers should exercise their judgement in these cases. 150 | 151 | [execute-async-script]: https://github.com/web-platform-tests/wpt/blob/f3dcc205a202467a922386036791372cd3e372fd/tools/wptrunner/wptrunner/executors/executorwebdriver.py#L969 152 | [take-screenshot]: https://github.com/web-platform-tests/wpt/blob/f3dcc205a202467a922386036791372cd3e372fd/tools/wptrunner/wptrunner/executors/executorwebdriver.py#L1079 153 | -------------------------------------------------------------------------------- /rfcs/testdriver_set_rph_registration_mode.md: -------------------------------------------------------------------------------- 1 | # RFC X: Add Custom Handlers support to testdriver.js 2 | 3 | ## Summary 4 | 5 | Add testdriver.js support for [Custom Handlers](https://html.spec.whatwg.org/multipage/system-state.html#custom-handlers/), by 6 | supporting a `setRPHRegistrationMode` command ([spec 7 | PR])(https://github.com/whatwg/html/pull/8267). 8 | 9 | ## Details 10 | 11 | The proposed `setRPHRegistrationMode` command places the user agent in a mode 12 | where it will automatically accept or reject (depending on the mode) future 13 | Custom Protocol Handlers registrations. This allows full end-to-end 14 | testing of Custom Handlers API (eg, registerProtocolHandler), which normally [requires user 15 | interaction](https://html.spec.whatwg.org/multipage/system-state.html#dom-navigator-registerprotocolhandler) 16 | to confirm the user's consent to the protocol registration. 17 | 18 | Links: 19 | - [WIP spec PR](https://github.com/whatwg/html/pull/8267) 20 | - [WIP web-platform-tests implementation PR](https://github.com/web-platform-tests/wpt/pull/35792) 21 | - [WIP Chromedriver implementation](https://chromium-review.googlesource.com/c/chromium/src/+/3865270) 22 | 23 | ### Alternatives Considered 24 | 25 | I've also considered the usag of the already defined [Set Permission](https://w3c.github.io/permissions/#set-permission-command) 26 | extension command, implemented at least by Chromedriver. In order to use such command, we would need to define a 27 | new permission descriptor for the Custom Handlers feature. Such [proposal](https://github.com/whatwg/html/issues/7920) 28 | has been discussed and rejected by Chrome and Firefox implementors. 29 | 30 | ## Risks 31 | 32 | This increases the API surface of testdriver.js, so it's more code to maintain 33 | or reason about. 34 | -------------------------------------------------------------------------------- /rfcs/testdriver_set_spc_transaction_mode.md: -------------------------------------------------------------------------------- 1 | # RFC 101: Add Secure Payment Confirmation support to testdriver.js 2 | 3 | ## Summary 4 | 5 | Add testdriver.js support for [Secure Payment 6 | Confirmation](https://w3c.github.io/secure-payment-confirmation/), by 7 | supporting a `setSPCTransactionMode` command ([spec 8 | PR](https://github.com/w3c/secure-payment-confirmation/pull/151)). 9 | 10 | ## Details 11 | 12 | The proposed `setSPCTransactionMode` command places the user agent in a mode 13 | where it will automatically accept or reject (depending on the mode) future 14 | Secure Payment Confirmation authentications. This allows full end-to-end 15 | testing of Secure Payment Confirmation, which normally [requires user 16 | interaction](https://w3c.github.io/secure-payment-confirmation/#sctn-transaction-confirmation-ux) 17 | to confirm the user's consent to the transaction authentication. 18 | 19 | Links: 20 | - [WIP spec PR](https://github.com/w3c/secure-payment-confirmation/pull/151) 21 | - [WIP web-platform-tests implementation PR](https://github.com/web-platform-tests/wpt/pull/31345) 22 | - [WIP Chromedriver implementation](https://chromium-review.googlesource.com/c/chromium/src/+/3237267) 23 | 24 | ### Alternatives Considered 25 | 26 | We also prototyped an approach where the WebDriver API was to accept/reject 27 | just the currently-open SPC dialog (e.g. `webdriver.accept_spc_prompt`). This 28 | design is cleaner in terms of being idempotent, however it [led to 29 | flake](https://chromium-review.googlesource.com/c/chromium/src/+/3225868/11#message-443faa84dfec42cf930bf7bf99e47d35f784bbbc) 30 | as the web-page is not able to tell exactly when the dialog has been displayed 31 | to the user. For example, if the card art icon is slow to load, the test 32 | web-page may race ahead of the UI being displayed and try to accept/reject it 33 | before it even shows. 34 | 35 | One could imagine defining more complex semantics there ('wait a reasonable 36 | amount of time for an SPC dialog to exist, and then accept/reject it'), but 37 | given that complexity the 'mode' approach seemed easier. 38 | 39 | ## Risks 40 | 41 | This increases the API surface of testdriver.js, so it's more code to maintain 42 | or reason about. 43 | -------------------------------------------------------------------------------- /rfcs/testharness_include_late_tests.md: -------------------------------------------------------------------------------- 1 | # RFC 141: Change testharness to include late tests 2 | 3 | ## Summary 4 | 5 | Currently, "late tests" are 6 | [ignored](https://github.com/web-platform-tests/wpt/blob/c751476c04678faa79e598683f2a6e94d17cf9e0/resources/test/tests/unit/late-test.html#L10-L17). 7 | This RFC proposes that testharness should now include them. 8 | 9 | ## Details 10 | 11 | Conside the following test: 12 | 13 | ```html 14 | 21 | ``` 22 | 23 | Currently, only test 1 will be run. The issue is that the testharness 24 | immediately adds a window load handler that marks `all_loaded = true`, 25 | and that ends the tests as soon as the first result from the first test 26 | is processed. (The test runner waits for the first test because 27 | `Tests.prototype.all_done()` also waits until `this.tests.length > 0`.) 28 | There were various mitigating corner cases, such as if you started 29 | the list of tests with a `promise_test()`, that would increment a 30 | counter that kept the rest of the tests alive. Etc. 31 | 32 | The proposal is to change testharness `window.onload` handler to run 33 | with a `setTimeout(0)` so that all_loaded is only set to true after all of 34 | the tests are loaded by any `window.onload` handler. 35 | 36 | ## Risks 37 | 38 | This change will expose a few tests that should have been failing but were 39 | masked by the lack of test coverage - bugs have been filed for those. 40 | 41 | Also, several tests that were working around this via various 42 | means will be cleaned up in the implementation of this RFC. 43 | -------------------------------------------------------------------------------- /rfcs/top_level_subdomain.md: -------------------------------------------------------------------------------- 1 | # [RFC 39](https://github.com/web-platform-tests/rfcs/pull/39): Loading top-level tests on a subdomain. 2 | 3 | ## Summary 4 | 5 | Add a `.www` [file name flag](https://web-platform-tests.org/writing-tests/file-names.html) that causes a given test to be loaded from the subdomain `www.web-platform.test` rather than the apex domain `web-platform.test`. 6 | 7 | ## Details 8 | 9 | Tests are typically loaded from `http://web-platform.test/`. For some features that verify behavior within a site, it would be helpful to load the top-level document from a subdomain (e.g. `http://www.web-platform.test/`). Tests for cookies' `Domain` attribute are a good example: it's possible to write tests that verify that attribute's effects on requests from `www.web-platform.test` to `web-platform.test`, but requires creating a proxy document in a new window or frame, running tests in that context, and `postMessage()`ing results back to the main document. This adds a good deal of complexity that could be avoided if we opened the top-level document on the subdomain to begin with. 10 | 11 | We distinguish files that ought to be loaded from `https://web-platform.test/` with an `.https` [file name flag](https://web-platform-tests.org/writing-tests/file-names.html). We could support loading tests from `www.web-platform.test` by adding a `.www` flag. 12 | 13 | ## Risks 14 | 15 | * At some point, we might end up with a more-than-reasonable number of flags to consider when writing a test: `amazing-cookie-test.https.www.sub.tentative.any.js` might be a bit much...) 16 | * Naming is hard. `.subdomain` is appealing, but confusable with `.sub`. 17 | -------------------------------------------------------------------------------- /rfcs/variant_name_not_empty.md: -------------------------------------------------------------------------------- 1 | # RFC 158: Variant name should be a non-zero length string 2 | 3 | ## Summary 4 | 5 | Make a policy change to disallow zero length Variant names. 6 | 7 | ## Details 8 | 9 | A test file can have multiple variants by including meta elements. 10 | Even though zero-length variant names are allowed currently, we 11 | can not run it the same way as other variants. 12 | 13 | For example, the command below runs `websockets/binary/005.html?wss` 14 | only. 15 | ``` 16 | wpt run ... websockets/binary/005.html?wss 17 | ``` 18 | 19 | But the commands below run all three variants in the file. 20 | ``` 21 | wpt run ... websockets/binary/005.html 22 | wpt run ... websockets/binary/005.html? 23 | ``` 24 | 25 | This consequently causes problem when we do external sharding, then 26 | pass all the tests we want to run to wptrunner through `--include`. 27 | While we only intended to run the empty string variant by passing 28 | `websockets/binary/005.html`, wptrunner runs all the variants in 29 | that file, causing the non-empty variants to run twice. It is 30 | possible we shard all the variants from the same file to the same 31 | shard, this could load the different shards unevenly, and increase 32 | the perceived total run time. 33 | 34 | Proposal is to disallow zero length variant names. 35 | 36 | ### Implementations 37 | 38 | Below are the changes required to make this happen: 39 | 1. Update all the existing tests that have an empty variant name. Update 40 | the supporting code if necessary to make sure the tests continue to 41 | work. 42 | 2. Add a lint to `wpt lint` to make it an error for zero length variant 43 | names. 44 | 3. Update the document to not use zero length variant names as examples. 45 | 4. Update manifest generation code to throw an error for empty variant 46 | names. 47 | 48 | ### Alternatives Considered 49 | 50 | It is possible we update wptrunner to accept `path/to/test.html?` as the 51 | url for the empty name variants, but such changes would be hard to implement 52 | and it will be surprising to the developers. Meanwhile making people name 53 | each variant does add a little extra documentation. 54 | 55 | Alternatively we can add `/` at the beginning of a test name, to denote 56 | the parameter is an url instead of a path. This deviates further from 57 | how developers are running tests now, and it will break the command line 58 | auto-completion feature. 59 | 60 | ## Risks 61 | 62 | As this is adding restrictions to the existing system, it should not break 63 | anything. 64 | 65 | Developers should be able to run the tests the same way as before, with 66 | the added capability to run the previous empty name variant alone. 67 | -------------------------------------------------------------------------------- /rfcs/vendoring.md: -------------------------------------------------------------------------------- 1 | # RFC 218: Migrate Python vendoring to vendoring 2 | 3 | ## Summary 4 | 5 | We should use [vendoring](https://pypi.org/project/vendoring/) to re-do all of our Python vendoring, 6 | directly into `third_party` (crudely: `pip install -t tools/third_party ...`), 7 | and then move to a generated `requirements_vendor.txt`. 8 | 9 | [Implementation](https://github.com/web-platform-tests/wpt/pull/49752). 10 | 11 | 12 | ## Background 13 | 14 | We currently have a directory in WPT, called `tools/third_party`, 15 | which contains various bits of vendored code, 16 | which we copy into our tree primarily to 17 | [avoid Mozilla having to update their partial PyPI mirror](https://github.com/web-platform-tests/rfcs/issues/82#issuecomment-2462913061). 18 | 19 | This currently contains copies of a number of Python libraries as well as pdf.js, 20 | currently mostly imported via `git-subtree`. 21 | 22 | It's difficult to know what exact versions we have vendored, 23 | and it is hard to update them, 24 | and especially hard to know when we no longer need libraries we vendor 25 | (especially those we vendor only as dependencies of other libraries). 26 | 27 | The Python code is added to `sys.path` via 28 | [`tools/localpaths.py`](https://github.com/web-platform-tests/wpt/blob/59ee38f559b1bb85b9474b4e870bf70183f50414/tools/localpaths.py). 29 | 30 | 31 | ## Details 32 | 33 | Similar to the [implementation](https://github.com/web-platform-tests/wpt/pull/49752/commits), 34 | this is split into four sections: 35 | 36 | ### Move `pdf.js` 37 | 38 | `vendoring` as a tool expects the vendored directory to be fully under its control; 39 | as a result, we need to move the singular non-Python dependency out of the directory. 40 | 41 | This RFC proposes that we move `pdf.js` from `tools/third_party/pdf_js` to `tools/third_party_js/pdf_js`. 42 | 43 | 44 | ### Re-vendor everything we currently have vendored 45 | 46 | To make it easy to bisect any regressions from this change, 47 | we should start off by initially just re-vendoring everything, 48 | just now using the `vendoring` tool, 49 | listing everything in `tools/requirements_vendor.txt`. 50 | 51 | This causes one major change: 52 | it removes everything except the contents of the distribution packages, 53 | thus removing our vendored copies of the documentation, 54 | test suites, 55 | CI scripts, 56 | and everything else they happen to have in their repos. 57 | 58 | This results in everything being importable from the `tools/third_party` directory, 59 | thus `localpaths` can be simplified to only add that directory to `sys.path`. 60 | 61 | This causes us to have to make two other changes to our Python: 62 | 63 | 1. Change the path to the `tooltool` module in `tools/wpt/android.py`. 64 | 65 | 2. Explicitly list `pytest_asyncio` as a plugin in `webdriver/tests/conftest.py`. 66 | 67 | The latter of these is necessary because we no longer discover the entry-point of the vendored copy. 68 | 69 | The only other notable change is we now maintain a `tools/third_party_patches`, 70 | which contains all the patches we are applying to what we're importing. 71 | This is currently only `pywebsocket3`, with changes in 72 | [efa4a99b8d](https://github.com/web-platform-tests/wpt/commit/efa4a99b8dde1d9ab572efb9e1757e6900289bed) 73 | ([upstream issue #38](https://github.com/GoogleChromeLabs/pywebsocket3/issues/38)) and 74 | [3bd9ff9f8a](https://github.com/web-platform-tests/wpt/commit/3bd9ff9f8a175b554cc2e78f80b9d28fff73f66f) 75 | (no upstream issue). 76 | 77 | Otherwise, this change is roughly a two million line removal from WPT, 78 | because we import a lot of documentation and tests. 79 | 80 | 81 | ### Downgrade html5lib to 1.1 82 | 83 | Currently we have html5lib imported from [f4646e6](https://github.com/html5lib/html5lib-python/commit/f4646e6ed4eeb9780f67d2083d0c09c8fffbec53), 84 | which is the commit after 1.1's release, 85 | changing the version number to 1.2-dev. 86 | 87 | We should downgrade html5lib to 1.1, 88 | thus allowing us to vendor the version released on PyPI. 89 | 90 | 91 | ### Use `uv pip compile` to build the dependency tree 92 | 93 | Finally, we can move to just listing everything we actually depend on in a `requirements_vendor.in` 94 | and generate our `requirements_vendor.txt` from that. 95 | 96 | This is where risks start to come in: 97 | this removes a number of vendored libraries, 98 | because we no longer need them. 99 | 100 | 101 | ## Risks 102 | 103 | ### External code relying on paths within `tools/third_party` 104 | 105 | One could, hypothetically, have someone doing: 106 | 107 | ```python 108 | sys.path[0:0] = ["path/to/wpt/tools/third_party/pytest"] 109 | ``` 110 | 111 | and have it expected that this works to make `pytest` importable. 112 | 113 | Making these changes would break this, 114 | as `tools/third_party` is now the only path added to the module search path. 115 | 116 | 117 | ### External code relying on dependencies we no longer require 118 | 119 | By removing projects like `attrs`, 120 | which we no longer use, 121 | it could turn out that other external code was relying on the WPT vendored copies being on the `sys.path`, 122 | rather than listing them as dependencies itself. 123 | 124 | 125 | ### Lowering the barrier to vendoring new projects may make us overly reliant on vendoring 126 | 127 | As a result of making it easier to vendor projects, 128 | we could end up with people adding much more vendored code. 129 | 130 | This is problematic both insofar as we become distributors of the contents of the distribution package, 131 | with the potential legal implications of that, 132 | and makes us potentially more inclined to distribute an every larger number of packages. 133 | 134 | It also potentially decreases the motivation for Mozilla to make it simpler to update their PyPI mirror, 135 | thereby effectively perpetuating our vendoring further into the future. 136 | 137 | 138 | ## Further work 139 | 140 | The most obvious follow-up item to this is "update all of our vendored projects", 141 | and while we normally don't file RFCs for dependency updates, 142 | it is potentially higher risk when we haven't done this in years. 143 | -------------------------------------------------------------------------------- /rfcs/wait_for.md: -------------------------------------------------------------------------------- 1 | # RFC 56: `step_wait*` methods 2 | 3 | ## Summary 4 | 5 | Add `step_wait`, `step_wait_func` and `step_wait_func_done` methods to 6 | the wpt `Test` object that evaluate a condition, and resolve a 7 | promise, or call a callback, once the condition is true. 8 | 9 | ## Details 10 | 11 | Using timers in tests is problematic; depending on the speed of the 12 | system the test can either wait a long time unnecessarily or be flaky 13 | because the operation sometimes takes longer than allowed by the 14 | timeout. The timeout multiplier in conjunction with `step_timeout` 15 | can alleviate this by adjusting the timeout to the overall system 16 | speed, but authors are still incentivised to provide short timeouts in 17 | order to allow tests to run faster. 18 | 19 | In some cases it's possible to detect directly whether the thing being 20 | waited on has occured or not. In this case the performance/robustness 21 | tradeoff can be improved by having a relatively long total timeout but 22 | polling for success to allow the test to proceed as soon as possible. 23 | 24 | To encourage this pattern we introduce three new methods on `Test`: 25 | 26 | * `step_wait_func(cond, callback, description, timeout=3000, 27 | interval=100)` 28 | * `step_wait_func_done(cond, callback, description, timeout=3000, 29 | interval=100)` 30 | * `step_wait(cond, description, timeout=3000, interval=100)` 31 | 32 | In each case the function `cond` is called with no 33 | arguments, immediately and then every `interval` milliseonds until it 34 | returns a value that evaluates as `true`. At this point `callback` is called. 35 | If this does not happen in `timeout` milliseconds, an `AssertionError` is 36 | raised, using `description` as the error description. 37 | 38 | The difference between the functions is in what happens after `cond` 39 | evaluates to true: 40 | 41 | * `step_wait_func` - `callback` is called as a step on `Test`. 42 | * `step_wait_func_done` - `callback` is called as a step on `Test` if 43 | it's not `undefined` or `null`. After `callback` returns, or in any 44 | case if it's not called, the `done` function on test is called. 45 | * `step_wait` - The initial function call returns a promise, which is 46 | resolved when `cond` evaluates to true. 47 | 48 | These functions must methods on `Test` to allow any error raised to be 49 | associated with the correct subtest. 50 | 51 | The default timeout and interval were adopted from the 52 | `waitForCondition` 53 | [function](https://searchfox.org/mozilla-central/source/testing/mochitest/tests/SimpleTest/SimpleTest.js#1236) 54 | in Mozilla's mochitest, which implements a similar pattern, and has 55 | been used to support a policy of only allowing `setTimeout` with 56 | non-zero timeout in exceptional cases. 57 | 58 | ## Examples 59 | 60 | A test which waits to check that `contentDocument` on an `iframe` 61 | becomes null after some action, might intially look like: 62 | 63 | ``` 64 | t.step_timeout(() => { 65 | assert_equals(frame.contentDocument, null); 66 | t.done(); 67 | }, 2000); 68 | ``` 69 | 70 | With these APIs it can instead look like 71 | 72 | ``` 73 | t.step_wait_func_done(() => frame.contentDocument === null); 74 | ``` 75 | 76 | ## Risks 77 | 78 | * Adding API surface to testharness.js requires long-term support or 79 | is quite difficult to remove or change if the feature is deemed to 80 | be a failure. 81 | * Features like this and `step_timeout` could be useful in support 82 | files, but using testharness.js features from support files is 83 | tricky. 84 | * This RFC doesn't propose a lint to flag uses of `step_timeout` / 85 | `setTimeout`, so it's hard to educate authors about the optimal 86 | pattern. Such a lint could be proposed as a followup, but our 87 | linting infrastructure lacks support for parsing js code, so it's 88 | hard to enforce complex conditions like only allowing zero-duration 89 | timeouts. 90 | -------------------------------------------------------------------------------- /rfcs/websockets_http2.md: -------------------------------------------------------------------------------- 1 | # RFC #67: Add WebSockets over HTTP/2 support to wptserve 2 | 3 | ## Summary 4 | 5 | Enable wptserve to support WebSockets over an existing http2 connection. 6 | 7 | ## Details 8 | 9 | Allow tests to open a WebSocket over an existing http2 connection. 10 | 11 | wptserver needs to implement the [Bootstrapping WebSocket with HTTP/2 protocol](https://tools.ietf.org/html/rfc8441). 12 | 13 | It means that it needs to be modified to: 14 | - Set ENABLE_CONNECT_PROTOCOL settings to 1 15 | - Detect Extended CONNECT Method and implement the WebSocket opening handshake. 16 | - pywebsocket operates on Apache mod_python Request objects. So we need to wrap H2Response and H2Request into a Request-like object. 17 | 18 | ## Risks 19 | 20 | WebSocket tests who source **websockets/constants.js** have a wss variant. We will need to add a new variant named **http2** in order to use the http2 port when opening the WebSockets. WPT server does not support insecure HTTP/2 so we don't need to expose an insecure HTTP/2 WebSocket variant. 21 | 22 | This change needs to update hyper libraries to a more recent version. 23 | 24 | No other risks are expected except those introduced by any feature e.g. complexity. 25 | -------------------------------------------------------------------------------- /rfcs/websockets_py3.md: -------------------------------------------------------------------------------- 1 | # RFC 70: Add Python3 WebSockets dependency for WebDriver Bidi tests 2 | 3 | ## Summary 4 | 5 | Add a Py3-only [WebSockets client](https://github.com/aaugustin/websockets) library that uses `asyncio`, to enable WebDriver Bidi tests. 6 | 7 | ## Details 8 | 9 | ### Background 10 | 11 | [WebDriver Bidi](https://w3c.github.io/webdriver-bidi/) builds and extends on [WebDriver](https://w3c.github.io/webdriver/). 12 | WebDriver Bidi uses WebSockets for bidirectional communication between the remote end (acting as the server) and the local end (WPT, acting as a client). 13 | The existing pywebsocket3 library that is vendored into WPT only serves as a WebSockets server. Therefore, a new WebSockets client library dependency is needed in order to connect to WebDriver for Bidi communication. 14 | 15 | ### Proposal 16 | 17 | Add the Py3-only WebSockets client library linked in the Summary above. 18 | 19 | Additionally, add a dependency on [pytest-asyncio](https://github.com/pytest-dev/pytest-asyncio) for its convenient test marker. 20 | Note pytest-asyncio depends on later versions of pytest and pluggy so they also need to upgrade to compatible versions. 21 | 22 | ### Example usage 23 | 24 | The [patch](https://github.com/web-platform-tests/wpt/pull/26510) 25 | 1. Adds a simple test to verify the new capability that would enable websocket connection for Bidi communication. 26 | 1. Upgrades existing tools/third_party dependencies which are depended by the pytest-asyncio which provides convenient markers such as 27 | "@pytest.mark.asyncio" for writing tests. 28 | 1. Makes use of the async/await syntax. 29 | 1. Adds a BidiSession class in tools/webdriver/webdriver/client.py to be used 30 | in test fixture. 31 | 1. Adds tests to verify the behavior of BidiSession across tests. 32 | 33 | A couple issues not solved in the patch: 34 | 35 | 1. When upgrading pytest to 6.1.1 in example code, there was an error from tools/third_party/pytest/src/_pytest/assertion/rewrite.py 36 | about missing _pytest._version module. 37 | I commented out the import and hardcoded the version to 6.1.1. 38 | 1. I installed the WebSockets and pytest-asyncio dependencies locally instead of including them in tools/third_party library. 39 | 40 | ## Risks 41 | 42 | The current plan for python migration(I believe) is Py3-first by the end of the year and Py3-only Feb 2021. 43 | The patch is Py3-only and there is a gap of 3 months from now and then. 44 | Do we want to wait until then to add the dependencies? 45 | Or do we want to be able to exclude these dependencies for Py2? 46 | -------------------------------------------------------------------------------- /rfcs/webtransport_h3_cert_hash_test_server.md: -------------------------------------------------------------------------------- 1 | # RFC 216: WebTransport over HTTP/3 Test Server that uses a self-signed certificate for authentication with serverCertificateHashes 2 | 3 | ## Summary 4 | 5 | Start another instance of the [WebTransport over HTTP/3](https://datatracker.ietf.org/doc/html/draft-ietf-webtrans-http3-01) server in [RFC 85](https://github.com/web-platform-tests/rfcs/blob/master/rfcs/webtransport_h3_test_server.md) in wpt, that uses instead of the standard certificate a self-signed certificate. The hash of the certificate is passed to the test defined. The certificate is autogenerated at the startup of the wpt tests within the python code. 6 | 7 | ## Details 8 | 9 | ### Implementation 10 | 11 | It is a slight modification of the server from [RFC 85](https://github.com/web-platform-tests/rfcs/blob/master/rfcs/webtransport_h3_test_server.md) in the `tools/webtransport` directory. 12 | The implementation uses the `cryptography` lib for certificate generation, that is already used by wpt through the `aioquic` lib. 13 | 14 | ### Handlers 15 | See [RFC 85](https://github.com/web-platform-tests/rfcs/blob/master/rfcs/webtransport_h3_test_server.md) for details. 16 | 17 | ### `wptserve` integration 18 | 19 | See [RFC 85](https://github.com/web-platform-tests/rfcs/blob/master/rfcs/webtransport_h3_test_server.md) for details. 20 | 21 | A server certificate hash is autogenerated within the python code in `environment.py`. It is passed as part of the config (property `cert_hash_info`) to the tests. The tests can access the server certificate hash as `server_certificate_hash` inside the javascript templates. 22 | 23 | ### Dependencies 24 | 25 | As of writing this RFC, the only dependency are `aioquic` as in RFC 85 and `cryptography`, which is already a dependency of `aioquic`. 26 | 27 | ## Risks 28 | 29 | Risks are similar to [RFC #85](https://github.com/web-platform-tests/rfcs/blob/master/rfcs/webtransport_h3_test_server.md#risks), as it is a minimal modification of RFC 85, so that maintenance cost increase is neglible. 30 | -------------------------------------------------------------------------------- /rfcs/webtransport_h3_test_server.md: -------------------------------------------------------------------------------- 1 | # RFC 85: WebTransport over HTTP/3 Test Server 2 | 3 | ## Summary 4 | 5 | Add a [WebTransport over HTTP/3](https://datatracker.ietf.org/doc/html/draft-ietf-webtrans-http3-01) server to wpt. The server is used to test the [WebTransport Web APIs](https://w3c.github.io/webtransport/). The server requires Python 3. 6 | 7 | The server uses handlers to handle WebTransport events. Handlers are python scripts which contains callback functions to respond WebTransport events. 8 | 9 | This RFC shares the same background/motivation with [RFC 42](https://github.com/web-platform-tests/rfcs/blob/master/rfcs/quic.md) but focuses on WebTransport over HTTP/3. See RFC 42 for more background. 10 | 11 | ## Details 12 | 13 | ### Implementation 14 | 15 | The WebTransport over HTTP/3 test server is built on top of `aioquic`, which is already added under `tools/third_party`. As of version 0.9.11, `aioquic` lacks some mechanisms to implement a WebTransport server (e.g. configuring SETTINGS HTTP/3 frame) but we plan to update `aioquic` once it supports necessary mechanisms. 16 | 17 | The server will be implemented under `tools/webtransport` directory. 18 | 19 | ### Handlers 20 | 21 | A WebTransport handler is a Python script which contains some callback functions. Callback functions are called every time a WebTransport event happens. The current proposed callback functions can be found [here](https://bashi.github.io/wpt-wt-test-server-api/handler.html). The follwoing is an example handler which echos back received data. 22 | 23 | ```python 24 | from webtransport import WebTransportSession 25 | 26 | def stream_data_received(session: WebTransportSession, 27 | stream_id: int, 28 | data: bytes) -> None: 29 | session.send_stream_data(stream_id=stream_id, data=data) 30 | 31 | 32 | def datagram_received(session: WebTransportSession, 33 | data: bytes) -> None: 34 | session.send_datagram(data=data) 35 | ``` 36 | 37 | `session` is an object that represents a WebTransport over HTTP/3 session. It provides APIs to handle the session. Current proposed APIs can be found [here](https://bashi.github.io/wpt-wt-test-server-api/session.html). 38 | 39 | `session` provides methods to create unidirectional and bidirectional streams. These methods are similar to `aioquic`'s [asyncio API](https://aioquic.readthedocs.io/en/latest/asyncio.html#aioquic.asyncio.QuicConnectionProtocol.create_stream). The following handler creates a bidirectional stream and writes "PASS" to the stream when a session is established. 40 | 41 | ```python 42 | import asyncio 43 | from webtransport import WebTransportSession 44 | 45 | def session_established(session: WebTransportSession, path: str) -> None: 46 | loop = asyncio.get_event_loop() 47 | loop.create_task(notify_pass(session)) 48 | 49 | 50 | async def notify_pass(session: WebTransportSession) -> None: 51 | _reader, writer = await session.create_bidirectional_stream() 52 | writer.write(b"PASS") 53 | writer.write_eof() 54 | ``` 55 | 56 | ### Handler lookup 57 | 58 | When the server receives an [extended CONNECT method](https://datatracker.ietf.org/doc/html/draft-ietf-webtrans-http3-01#section-3.2) for WebTransport, it looks up a corresponding handler from the root directory specified by a configuration. In this RFC the root directory is referred to as `$ROOT`. 59 | 60 | The server uses `":path"` header field of an extended CONNECT request to look up a handler. For example, if `":path"` is `/webtransport/handlers/echo.py`, the server looks up `$ROOT/webtransport/handlers/echo.py`. 61 | 62 | This lookup mechanism provides flexibility to put handlers under multiple directories, but we will draw up a guideline to put handlers in a single directory for discoverability. 63 | 64 | If the server finds a corresponding handler the server creates a new WebTransport session with the handler and starts passing events to the handler. 65 | 66 | ### File name flags 67 | 68 | This RFC focues on WebTranport over HTTP/3. WebTransport over HTTP/2 is out of scope. However, in the future, we may consider adding `.h2` and `.h3` name flags to handler names to specify the underlying protocol (HTTP/2 or HTTP/3). A separate RFC will be submitted as needed. 69 | 70 | ### `wptserve` integration 71 | 72 | When the WebTransport over HTTP/3 test server runs as a part of `wptserve`, `$ROOT` will be the same as `$WPT_ROOT`. 73 | 74 | `wptserve` runs the WebTransport over HTTP/3 test server in a way similar to [pywebsocket](https://github.com/web-platform-tests/wpt/blob/246a32576020cb9c4241b7cfbc296f92d944ff6b/tools/serve/serve.py#L713): `wptserve` launches the server as a daemon and passes relevant configurations (port number, root directory etc) to the server. The daemon is monitored by `wptserve` like other daemons. 75 | 76 | ### Dependencies 77 | 78 | As of writing this RFC, the only dependency is `aioquic`. 79 | 80 | ### `tools/quic` deprecation 81 | 82 | There are no ongoing standardlization efforts which require a general QUIC server. We plan to remove `tools/quic` directory once the WebTransport over HTTP/3 test server is introduced to wpt. 83 | 84 | ## Risks 85 | 86 | Risks are similar to [RFC #42](https://github.com/web-platform-tests/rfcs/blob/master/rfcs/quic.md#risks). 87 | 88 | The WebTransport over HTTP/3 test server itself is standalone and the maintenance cost of the code will be low from the `wptserve`'s perspective. The Web Networking team and the Ecosystem Infra team at Google will be responsible for maintaining the integration point and the server. 89 | -------------------------------------------------------------------------------- /rfcs/wpt-results-analysis.md: -------------------------------------------------------------------------------- 1 | # RFC 106: Move wpt.fyi metrics generation into the WPT org 2 | 3 | ## Summary 4 | 5 | Move https://github.com/Ecosystem-Infra/wpt-results and https://github.com/Ecosystem-Infra/wpt-results-analysis into the web-platform-tests GitHub org. 6 | 7 | ## Details 8 | 9 | The following metrics on wpt.fyi are generated by wpt-results-analysis: 10 | 11 | - Browser Specific Failures 12 | - Compat 2021 13 | - Interop 2022 14 | 15 | wpt-results-analysis depends on wpt-results, a highly compressed form of all full wpt.fyi runs ever, currently ~52k stored in a ~385M Git repository. This makes it possible to load thousands (or all) runs into memory at the same time, speeding up metrics generation. 16 | 17 | ## Risks 18 | 19 | wpt-results-analysis depends on nodegit, which only works on Node.js 14, see https://github.com/nodegit/nodegit/issues/1840. 20 | 21 | The wpt.fyi frontend fetches data from https://raw.githubusercontent.com/Ecosystem-Infra/wpt-results-analysis/gh-pages/data/interop-2022/interop-2022-experimental.csv and similar URLs. Those URLs will break when the wpt-results-analysis repo is transferred, and the [HTML redirect method](https://gist.github.com/domenic/1f286d415559b56d725bee51a62c24a7) will not work. To avoid this, the data files should be hosted in a storage bucket instead of on GitHub pages, and wpt.fyi needs to start using that storage bucket before the repository is moved. 22 | 23 | ## Alternatives 24 | 25 | We could merge the metrics generation code into the wpt.fyi repository, but the wpt-results would still need to be standalone. 26 | -------------------------------------------------------------------------------- /rfcs/wpt_live.md: -------------------------------------------------------------------------------- 1 | # RFC 36: wpt.live 2 | 3 | ## Summary 4 | 5 | Maintain a publicly-accessible deployment of the web-platform-tests project, 6 | hosted at http://wpt.live. Maintain another deployment which allows project 7 | contributors to preview their submissions during the peer review process, 8 | hosted at http://wptpr.live. 9 | 10 | ## Details 11 | 12 | Bocoup and Google have collaborated on a service to provide a 13 | publicly-accessible instance of the web-platform-tests. The source code and the 14 | infrastructure configuration are maintained in the following repository (to be 15 | relocated to the web-platform-tests GitHub organization): 16 | 17 | https://github.com/bocoup/web-platform-tests.live 18 | 19 | This system involves two services, accessible from two distinct hosts. 20 | 21 | [wpt.live](http://wpt.live) runs the WPT project's "wptserve." It automatically 22 | synchronizes with the latest revision of the `master` branch of the WPT git 23 | repository. It is expected to be referenced by many highly-visible domains of 24 | the web platform (e.g. web specifications, [wpt.fyi](https://wpt.fyi), and 25 | [web-platform-tests.org](https://web-platform-tests.org)). 26 | 27 | [wptpr.live](http://wptpr.live) also runs the WPT project's "wptserve," and it 28 | includes additional logic to automatically retrieve patches submitted to the 29 | WPT repository through GitHub.com. These submissions are made available for 30 | "trusted" patches (i.e. those authored by project collaborators and those that 31 | have been explicitly "safelisted" by project collaborators) in order to 32 | facilitate peer review. The WPT GitHub project will be configured to inform 33 | contributors of the status of these previews. 34 | 35 | ## Risks 36 | 37 | A publicly-accessible service like wpt.live is a substantial responsibility for 38 | the group of people maintaining it. Because it is expected to be referenced 39 | from many independent projects, consistent availability is critical. Ensuring 40 | that availability requires a different mode of support than much of the WPT 41 | staff's existing responsibilities. Accepting and promoting a project like 42 | wpt.live therefore involves the risk of unpredictable maintenance requirements. 43 | 44 | The stability of wptserve is not guaranteed. It is derived from software whose 45 | maintainers [explicitly discourage its use in production 46 | contexts](https://docs.python.org/2/library/simplehttpserver.html). 47 | Additionally, it is designed to execute code written by WPT test authors, and 48 | these contributors may not have stability or security in mind. 49 | 50 | wpt.live has been designed to mitigate this risk via multiple automated failure 51 | recovery mechanisms. Instances use [Docker's restart 52 | policies](https://docs.docker.com/engine/reference/commandline/run/) to rapidly 53 | recover from transient runtime errors (e.g. memory leaks and stack overflows). 54 | In addition, the project has been configured to use [Google Cloud Platform's 55 | "autohealing" 56 | feature](https://cloud.google.com/compute/docs/instance-groups/#autohealing) in 57 | order to guard against more persistent problems (e.g. faulty disk state). 58 | 59 | Because the web-platform-tests project lacks a general solution for sharing 60 | secrets between its members, the overhead of ad-hoc access control is another 61 | risk of inheriting this service. Deployment and troubleshooting requires 62 | privileged access to the relevant Google Cloud Platform project. Members of 63 | Google's Ecosystem Infra team have this access. This includes a non-Google 64 | employee, demonstrating that employment in that organization is not a 65 | requirement. Although TLS certificates commonly represent another source of 66 | privileged information, the process has been automated via [Let's 67 | Encrypt](https://letsencrypt.org/). 68 | 69 | ## Alternatives considered 70 | 71 | Do nothing - the service described in this RFC is already provided by the W3C 72 | via http://w3c-test.org. WPT could avoid the risks and continue to enjoy the 73 | benefits by continuing to rely on that service. However, the service lacks 74 | automated failure recovery mechanisms, and its unreliability is well-known by 75 | both its users and its maintainers. Its status as a W3C-owned project reduces 76 | the ability of WPT members to understand and fix problems. 77 | 78 | Inherit the existing solution - To the latter point, another alternative is for 79 | WPT to inherit w3c-test.org. However, because this would not improve 80 | reliability, doing so would almost increase the maintenance effort required 81 | from WPT project members. (If this proposal is accepted, the maintainers of 82 | w3c-test.org will configure that domain to redirect to wpt.live). 83 | -------------------------------------------------------------------------------- /rfcs/wpt_pr_live_turndown.md: -------------------------------------------------------------------------------- 1 | # RFC 76: wptpr.live turndown 2 | 3 | ## Summary 4 | 5 | Turn down http://wptpr.live, which allows project contributors to preview their 6 | submissions during the peer review process, due to lack of uptake/interest and 7 | overhead of maintaining the system. 8 | 9 | This RFC does not affect http://wpt.live, which will continue to run. 10 | 11 | ## Details 12 | 13 | In [RFC 36](https://github.com/web-platform-tests/rfcs/pull/36), which landed 14 | in Nov 2019, Bocoup and Google's Ecosystem Infra team jointly proposed bringing 15 | up both a public hosted copy of the WPT repository (http://wpt.live) and a 16 | PR-preview service (http://wptpr.live). The goal was to support reviewers of 17 | test changes in WPT, by letting them preview the changes without needing to run 18 | WPT locally. 19 | 20 | Unfortunately however the system has faced a number of technical issues 21 | (historical and ongoing), as well as limited uptake from users (not least 22 | because it is difficult to locate in the GitHub PR workflow). Bocoup is also no 23 | longer involved in maintaining the project. Given this situation, we either 24 | need to invest heavily in the project (to reduce its long-term maintenance cost 25 | and improve the PR workflow integration), or make the decision to shut it down. 26 | 27 | Due to current and expected available engineer resourcing, this RFC proposes 28 | shutting down http://wptpr.live. This will remove the ability to preview 29 | in-flight pull requests in WPT without checking out the branch in a local clone 30 | of WPT. As noted in the Summary, this will not affect http://wpt.live. 31 | 32 | ## Risks 33 | 34 | The obvious risk here is an impact to the efficacy of PR reviews, if reviewers 35 | are relying on http://wptpr.live. Unfortunately the page does not have 36 | analytics, so the actual user-count is unknown, but anecdotally few users have 37 | mentioned the desire to use this service over the year it has been available. 38 | Moreso, http://wptpr.live didn't work for pull requests from forks from Feb 39 | 2020 until Dec 2020, and again anecdotally almost nobody noticed. 40 | -------------------------------------------------------------------------------- /rfcs/wptfyi_enable_view_eq_test.md: -------------------------------------------------------------------------------- 1 | ## RFC 190: Enable view=test mode in wpt.fyi 2 | 3 | ## Summary 4 | This RFC proposes enabling a new "test" view mode. This mode introduces a new method for counting total and failing tests, focusing on a strict pass/fail metric based on subtest counts. It includes frontend modifications to facilitate this change. This is to support this [feature request](https://github.com/web-platform-tests/wpt.fyi/issues/3740). 5 | 6 | ## Details 7 | 8 | ### Key Changes 9 | Currently, the feature is behind a feature flag that is set to false by default in production. 10 | * **Frontend Changes:** 11 | No frontend changes. This is will only be reachable by a query parameter that the end user must construct. 12 | 13 | ### Background about "test" View Mode Behavior 14 | * Test pass is only registered when the number of passed subtests equals the total number of subtests. 15 | * Percentages are not displayed, only raw counts at the directory level. 16 | * Individual test level display: 17 | - Shows "PASS" only when subtest passes equal total subtests. 18 | - Shows "FAIL" when `status == OK` but subtest passes do not equal total subtests. 19 | - Existing `STATUS_ABBREVIATIONS` mappings used for other statuses. 20 | - "FAIL" as a fallback for any unmapped statuses. 21 | 22 | ### How To Test The Change 23 | 24 | The feature has been enabled in staging by default so users can try it out. 25 | 26 | - Go to any page on https://staging.wpt.fyi that shows test counts 27 | - Add the view=test query parameter to the URL 28 | 29 | Examples: 30 | - https://staging.wpt.fyi/results/?label=experimental&label=master&aligned&view=test 31 | - https://staging.wpt.fyi/results/compat?label=experimental&label=master&aligned&view=test 32 | - https://staging.wpt.fyi/results/webstorage?label=experimental&label=master&aligned&view=test 33 | 34 | 35 | ### Risks 36 | 37 | 1. This new approach has different strengths and weaknesses to the long-standing default subtest view on wpt.fyi. 38 | - Mitigation: The default will remain unchanged to support current workflows and existing links. 39 | --------------------------------------------------------------------------------