├── README.md
└── spec.md


/README.md:
--------------------------------------------------------------------------------
  1 | # Layered APIs
  2 | 
  3 | A new standards effort for collaborating on high-level features.
  4 | 
  5 | ## Problem
  6 | 
  7 | The [Extensible Web Manifesto](https://extensiblewebmanifesto.org/)’s focus on low-level primitives promotes a healthy, well-layered platform that encourages innovation and experimentation in JavaScript. But focusing on low-level primitives means that developers must build most application-level components on their own, creating a high barrier to entry for new web developers.
  8 | 
  9 | This lack of built-in high-level features also bloats page load size. The average site payload is [2.5 MB and takes 19 seconds to load](https://www.doubleclickbygoogle.com/articles/mobile-speed-matters/).
 10 | 
 11 | Historically, standards bodies and implementers have been reluctant to work on higher-level APIs. In particular, introducing new capabilities via high-level APIs is dangerous, as when we get something wrong, developers are often left with no other way to access these capabilities. Shipping features also incurs an ongoing maintenance and runtime cost — every new feature pollutes the browser namespace, increases JS startup costs, and represents a new surface to introduce bugs throughout the codebase.
 12 | 
 13 | Additionally, the incentive for web developers to adopt higher-level APIs is low due to uneven browser uptake. If a feature does not add some essential new capability, but instead makes your application easier to write in newish browsers, it's rarely seen as worthwhile to go through the feature-detection dance if you have to write the fallback code anyway. Instead, developers often just use libraries built on top of the widely-supported lower-level APIs, incurring the attendant costs on all of their users.
 14 | 
 15 | ## Goal
 16 | 
 17 | Enable the creation of high-level features such that:
 18 | 
 19 | - They stay layered on top of low-level features, never getting access to new capabilities unavailable to web developers
 20 | - Runtime costs for web developers using the features scale
 21 | - Maintenance costs for standardizing and implementing the features scale
 22 | - Fallback to polyfills, for browsers that do not support the features, is easy and transparent
 23 | 
 24 | ## Solution
 25 | 
 26 | ### Part 1: the infrastructure
 27 | 
 28 | **WARNING: this syntax is under heavy discussion and probably will not survive future revisions. What is shown here is an idea that we once thought was a good one, but have since realized has many problems. See [the issue tracker](https://github.com/drufball/layered-apis/issues?q=is%3Aissue+is%3Aopen+sort%3Aupdated-desc+label%3A%22fallback+semantics%22) for open issues on it.**
 29 | 
 30 | **Until we know what a better solution would be, we leave the below to give some concrete idea of the problem we're trying to solve. But be aware we are actively looking into replacements.**
 31 | 
 32 | We propose a new syntax for accessing certain web platform features, known as _layered APIs_, by importing them from special URLs:
 33 | 
 34 | ```html
 35 | <script type="module"
 36 |         src="std:virtual-scroller|https://some-cdn.example/virtual-scroller.mjs">
 37 | </script>
 38 | 
 39 | <virtual-scroller>...</virtual-scroller>
 40 | ```
 41 | 
 42 | ```html
 43 | <script type="module">
 44 | import { storage } from "std:async-local-storage|https://other-cdn.example/async-local-storage.mjs";
 45 | 
 46 | storage.get("key").then(...);
 47 | </script>
 48 | ```
 49 | 
 50 | As shown here, this `std:x|y` URL syntax contains both an _API identifier_ (e.g. "`virtual-scroller`" or "`async-local-storage`"), and a _fallback URL_. If the browser does not support the layered API specified by the given API identifier, it instead loads the contents of the fallback URL.
 51 | 
 52 | See [this document](https://docs.google.com/document/d/1jRQjQP8DmV7RL75u_67ps3SB1sjfa1bFZmbCMfJCvrM/edit) for an exploration of alternate syntax options; the above is our tentative choice for now.
 53 | 
 54 | See [the proto-spec](./spec.md) for more details on LAPIs infrastructure.
 55 | 
 56 | ### Part 2: the standards process
 57 | 
 58 | Like all web platform features, layered APIs would go through the standards process, producing specifications for their API surface and behavior. However, they would have an important additional constraint: their specifications _must not_ use any "magic" that is inaccessible to web developers. A concrete way of stating this is that a web developer must be able to implement a given layered API's specification, purely in unprivileged JavaScript.
 59 | 
 60 | Apart from this additional requirement, layered APIs would be standardized in the same way as other APIs: incubation and explainers; transition to a standards body; TAG review; etc.
 61 | 
 62 | ## Benefits for web developers
 63 | 
 64 | ### Cheaper high-level features
 65 | 
 66 | Layered APIs will reduce the amount of script developers need to load over the network.
 67 | 
 68 | Also, because developers explicitly import the features they use, we don’t bloat the global context of the platform for everyone. You only pay the cost of a feature for features that you use. Because of the requirement to import, implementations can use a variety of implementation strategies, ranging from business-as-usual to lazily-loading the feature from their own servers on every use.
 69 | 
 70 | ### Encourage layering
 71 | 
 72 | By requiring that layered APIs not use any unexposed primitives, we are forced to identify and ship the appropriate low-level primitives needed to build the high-level feature. This gives web developers the tools they need to build their own applications and libraries. And new capabilities are never locked up inside of a higher-level API.
 73 | 
 74 | ### Built-in fallback
 75 | 
 76 | Layered APIs are instantly usable in all browsers via the built-in fallback to polyfill code. At the same time, newer browser versions that do include the feature will not be shipped unnecessary code, thus decreasing page size and JavaScript parsing time.
 77 | 
 78 | ## Benefits for standardization and implementation
 79 | 
 80 | ### Healthier platform implementation
 81 | 
 82 | Requiring that layered APIs sit on top of the platform's primitives provides a clean implementation boundary. Changing a layered API can't create thorny bugs throughout other specifications or parts of the implementation.
 83 | 
 84 | ### Decreased maintenance overhead
 85 | 
 86 | Implementers often shy away from building high-level features since they can create large, ongoing technical debt. Layered APIs can reduce this risk, as the clean separation means that much less maintenance work will be required. (Generally, maintenance would only be required if the browser purposefully breaks backward-compatibility in a lower-level features the layered API builds on, or fixes some bug upon which the layered API implementation inadvertently depended.)
 87 | 
 88 | ### Security and privacy
 89 | 
 90 | Layered APIs will have an easier time with security and privacy review, since they build on top of other APIs which have already passed security and privacy review. By definition, they are unable to do anything that web developers can't already do themselves, which sets an upper bound on the amount of harm possible.
 91 | 
 92 | ## Caveats
 93 | 
 94 | We believe that the layering restriction, and the benefits that come from it, is necessary in order for the web to responsibly ship high-level features. However, this restriction has its tradeoffs:
 95 | 
 96 | - Privacy and security sensitive features could not be implemented using this method
 97 | - Features that require new low-level primitives would be blocked on those primitives being standardized and shipped first
 98 | 
 99 | ## What makes a good layered API candidate?
100 | 
101 | When judging whether a feature is a good fit for the layered APIs effort, here are some criteria to consider:
102 | 
103 | - **Does this feature need new low-level capabilities to work successfully?** If so it, it isn't a good candidate yet; we need to fill in those gaps first.
104 | 
105 | - **Can this feature stand on its own, or does it require integration into existing APIs?** For example, adding methods or properties to existing web platform objects like `Array` or `HTMLElement` is potentially tricky, and we're not sure yet whether we should create layered APIs that, upon importing, have global side effects. For now, features that require such integration are not a good candidate for layered APIs.
106 | 
107 | - **Is this feature tricky to implement performantly or correctly?** It's better for the platform if such features can be standardized once, and implemented by browsers, instead of requiring developers to get them right every time independently. This criteria motivates the potential infinite virtual list or [tasklets](https://github.com/GoogleChromeLabs/tasklets) layered APIs.
108 | 
109 | - **Do the APIs for this feature vary wildly across the JS ecosystem, or have they mostly settled down?** Layered APIs will be less successful when they try to pick a winner that excludes popular styles or paradigms. For example, a virtual DOM layered API would likely be a poor idea at this time.
110 | 
111 | - **Does this feature involve a lot of styling choices for its UI?** If so, we're still figuring that out (see below), so the feature is probably not (yet) a good fit for the layered APIs effort.
112 | 
113 | - **Will this feature be used commonly, or rarely?** In the long term, layered APIs are a good fit for both cases. But for the initial batch of layered APIs, we'd like to focus on ones that will be used widely to show their value in terms of bringing down code size and making it easier to build web apps out of the box.
114 | 
115 | ## Styling and UI-component layered APIs
116 | 
117 | Several potentially good layered APIs, including the [infinite virtual scroller component](https://github.com/domenic/infinite-list-study-group), are UI components. Such components should generally come with minimal styling—at least as minimal, if not more, than existing standard HTML UI components. It would not be appropriate to encode a specific UI styling, like Material Design (Google) or Cupertino (Apple), into the layered APIs.
118 | 
119 | At the same time, layered APIs should be extremely styleable: authors should be able to make them fit into their pages, ideally with only CSS modifications. In the current landscape, this will require care; e.g. we cannot over-use shadow DOM, since it cannot be styled inside. In the future, [CSS shadow parts](https://tabatkins.github.io/specs/css-shadow-parts/) will greatly help with this.
120 | 
121 | That said, it’s important that built in UI components on the platform look and feel good by default. So there’s an open problem we’ll have to figure out for how UI components should be themed by default such that they can be consistent with OS-specific expectations. We can avoid dealing with this problem by starting with UI components that have no visual aspect to them (e.g. virtual list, which has no expectations, vs. new form controls, which do).
122 | 
123 | ## Trying these ideas out
124 | 
125 | We're currently prototyping the ideas in this repository in Blink, while working on two particular layered APIs. Those are:
126 | 
127 | * [`<virtual-scroller>`](https://github.com/valdrinkoshi/virtual-scroller): a HTML element that maps a provided set of JavaScript objects onto DOM nodes, and renders only the DOM nodes that are currently visible, leaving the rest "virtualized".
128 | * [Async local storage](https://github.com/domenic/async-local-storage): an asynchronous key-value store, including isolated storage areas and support for non-string values, layered on top of IndexedDB.
129 | 
130 | You can try these APIs, as well as the general LAPI infrastructure, in Chrome versions after 68.0.3420.0 (use [Chrome Canary](https://www.google.com/chrome/browser/canary.html)) with the `#enable-layered-api` flag flipped in `chrome://flags`. (Alternately, you can instead use `#enable-experimental-productivity-features` to get LAPIs plus some [feature policy](https://github.com/WICG/feature-policy/) work, or use the general `#enable-experimental-web-platform-features` flag to get all the currently-flagged web platform features.)
131 | 
132 | For demos, check out the [`<virtual-scroller>` demo repository](https://valdrinkoshi.github.io/virtual-scroller/demo/), or play with the following:
133 | 
134 | * [`<virtual-scroller>` playground](https://glitch.com/edit/#!/lowly-pantry)
135 | * [Async local storage playground](https://glitch.com/edit/#!/bead-hubcap)
136 | 


--------------------------------------------------------------------------------
/spec.md:
--------------------------------------------------------------------------------
  1 | # Layered APIs Proto-spec
  2 | 
  3 | This document contains the basics of a spec for the [layered APIs infrastructure](./README.md#part-1-the-infrastructure) discussed in the README. It is by no means finalized or formalized, but it gives implementers something to look at and possibly prototype with.
  4 | 
  5 | We expect that if this spec gains acceptance it would eventually be incorporated in the HTML Standard's script loading sections.
  6 | 
  7 | ## Introduction
  8 | 
  9 | Although layered APIs are loaded as JavaScript modules, they cannot have web-exposed source code. Their source code is an implementation detail, and depending on the browser, may not be in JavaScript.
 10 | 
 11 | This means we need to modify the module loading infrastructure slightly, as currently you are only able to load and evaluate module source code that you can also read with `fetch()`, due to the requirement for module scripts to be shared via CORS.
 12 | 
 13 | ## Layered API specifications
 14 | 
 15 | Layered API specifications define the following key concepts:
 16 | 
 17 | <dl>
 18 |   <dt id="api-identifier">API identifier
 19 |   <dd>A unique string by which this layered API will be imported
 20 | 
 21 |   <dt id="exports">Exports
 22 |   <dd>An <a href="https://infra.spec.whatwg.org/#ordered-map">ordered map</a> of strings to JavaScript objects, where the keys must be <a href="https://tc39.github.io/ecma262/#prod-IdentifierName">IdentifierName</a>s
 23 | 
 24 |   <dt id="evaluation-steps">Evaluation steps
 25 |   <dd>An optional set of steps to run when the layered API is evaluated, i.e. when it is first imported
 26 | </dl>
 27 | 
 28 | ## Layered API module records
 29 | 
 30 | We introduce a new type of [Module Record](https://tc39.github.io/ecma262/#sec-abstract-module-records), here called a Layered API Module Record.
 31 | 
 32 | We could define this in one of two ways:
 33 | 
 34 | 1. **Hand-wave**: reusing the existing Source Text Module Record (STMR) structure, and saying that implementations should create a STMR as if by parsing source text that would have the observable exports and evaluation steps as specified in the layered API's specification
 35 | 2. **Formalize**: actually create a new Module Record specification type; details below
 36 | 
 37 | We're not sure that formalization would actually be beneficial to implementers, but here we sketch out what such a module record would look like:
 38 | 
 39 | <details>
 40 | <summary>Formalized Layered API Module Record (LAMR)</summary>
 41 | 
 42 | <dl>
 43 |   <dt>[[Realm]]
 44 |   <dd>Every Realm contains a full set of LAMRs
 45 | 
 46 |   <dt>[[Environment]]
 47 |   <dd>A synthetic Module Environment created and populated with bindings corresponding to the exports defined in the layered API's specification
 48 | 
 49 |   <dt>[[Namespace]]
 50 |   <dd>No special treatment needed; the ECMAScript specification machinery will lazily fill this in for us
 51 | 
 52 |   <dt>[[HostDefined]]
 53 |   <dd>Not used
 54 | 
 55 |   <dt>GetExportedNames(exportStarSet)
 56 |   <dd>Returns the list of exported names defined in the layered API's specification. (The <var>exportStarSet</var> can be ignored in our case, as layered APIs cannot participate in module cycles.)
 57 | 
 58 |   <dt>ResolveExport(exportName, resolveSet)
 59 |   <dd>Checks if <var>exportName</var> is present in the list of exported names defined in the layered API's specification; returns { [[Module]]: this LAMR, [[BindingName]]: <var>exportName</var> } if so, or null otherwise. (The <var>resolveSet</var> can be ignored since layered APIs cannot participate in module cycles.)
 60 | 
 61 |   <dt>Instantiate()
 62 |   <dd>Sets up [[Environment]] as described above
 63 | 
 64 |   <dt>Evaluate()
 65 |   <dd>Evaluates any setup code described in the layered API's specification
 66 | </dl>
 67 | </details>
 68 | 
 69 | ## Pre-populating the module map
 70 | 
 71 | We would specify that on realm initialization, the [module map](https://html.spec.whatwg.org/multipage/webappapis.html#concept-settings-object-module-map) is pre-populated with mappings from every `std:x` URL to the corresponding layered API module record. This must include `std:blank`, which is detailed below
 72 | 
 73 | A browser could implement this by lazily populating the map when the appropriate layered API is imported; this would be observably equivalent. Note that the map is only populated; _moduleRecord_.Evaluate() is not called until `import` time.
 74 | 
 75 | ## Reserved layered APIs
 76 | 
 77 | `std:blank` is a layered API that is guaranteed to be supported if layered APIs are supported by a browser. It has an API identifier of "`blank`", an empty list of exports, and no initialization steps. Its primary use case is feature detection.
 78 | 
 79 | `std:none` denotes a layered API that is never supported; that is, layered API specifications must not use "`none`" as their API identifier. It is also useful for tests and feature detection.
 80 | 
 81 | ## Modifications to module loading
 82 | 
 83 | We make the following changes to ensure that when loading modules, we retrieve the LAMR from the module map if the corresponding LAPI is implemented, or use the fallback URL if not.
 84 | 
 85 | ### Resolve a module specifier
 86 | 
 87 | Modify [resolve a module specifier](https://html.spec.whatwg.org/multipage/webappapis.html#resolve-a-module-specifier)'s first step as follows:
 88 | 
 89 | 1. Let _parsed_ be the result of applying the [URL parser](https://url.spec.whatwg.org/#concept-url-parser) to specifier. If _parsed_ is not failure, then return the [layered API fetching URL](#user-content-layered-api-fetching-url) given _parsed_ and _script_'s [base URL](https://html.spec.whatwg.org/multipage/webappapis.html#concept-script-base-url).
 90 | 
 91 | This impacts `import` statements and `import()` calls.
 92 | 
 93 | ### `<script>`'s prepare a script
 94 | 
 95 | In [prepare a script](https://html.spec.whatwg.org/multipage/scripting.html#prepare-a-script), change step 23.6's "`module`" case to the following:
 96 | 
 97 | 1. Set _url_ to the [layered API fetching URL](#user-content-layered-api-fetching-url) given _url_ and _settings object_'s [API base URL](https://html.spec.whatwg.org/multipage/webappapis.html#api-base-url).
 98 | 1. If _url_ is failure, [queue a task](https://html.spec.whatwg.org/multipage/webappapis.html#queue-a-task) to [fire an event](https://dom.spec.whatwg.org/#concept-event-fire) named [`error`](https://html.spec.whatwg.org/multipage/indices.html#event-error) at the element, and return.
 99 | 1. [Fetch a module script graph](https://html.spec.whatwg.org/multipage/webappapis.html#fetch-a-module-script-tree) given _url_, _settings object_, "`script`", and _options_.
100 | 
101 | This impacts `<script type="module">`'s `src=""` resolution.
102 | 
103 | ### `<link rel="modulepreload">`'s obtain the resource
104 | 
105 | In the "obtain the resource" algorithm for [link type "`modulepreload`"](https://html.spec.whatwg.org/#link-type-modulepreload), move the current step 5 (which creates the _settings object_ variable) up to before the current step 4. Then, insert the following steps after the current step 4:
106 | 
107 | 1. Set _url_ to the [layered API fetching URL](#user-content-layered-api-fetching-url) given _url_ and _settings object_'s [API base URL](https://html.spec.whatwg.org/multipage/webappapis.html#api-base-url).
108 | 1. If _url_ is now failure, return.
109 | 
110 | This impacts `<link rel="modulepreload">`'s `href=""` resolution.
111 | 
112 | ### Supporting algorithm
113 | 
114 | <p id="layered-api-fetching-url">The above modifications delegate to the following algorithm for determining the <dfn>layered API fetching URL</dfn> given a URL <var>url</var> and a URL <var>baseURLForFallback</var>:
115 | 
116 | 1. If _url_'s [scheme](https://url.spec.whatwg.org/#concept-url-scheme) is not "`std`", return _url_.
117 | 1. Let _serialized_ be the result of running the [URL serializer](https://url.spec.whatwg.org/#concept-url-serializer) on _url_.
118 | 1. Remove the leading "`std:`" from _serialized_.
119 | 1. Let _identifier_ be the portion of _serialized_ before the first U+007C (|), or all of _serialized_ if no U+007C is present.
120 | 1. Let _fallback_ be the portion of _serialized_ after the first U+007C, or null if no U+007C is present.
121 | 1. If the layered API identified by _serialized_ is implemented by this user agent, return the result of [parsing](https://url.spec.whatwg.org/#concept-url-parser) the concatenation of "`std:`" with _identifier_.
122 | 1. If _fallback_ is null, return failure.
123 | 1. Return the result of [parsing](https://url.spec.whatwg.org/#concept-url-parser) _fallback_ with the base URL _baseURLForFallback_.
124 | 
125 | This operation maps URLs of the form `std:x|y` to a specific absolute URL: either `std:x`, or `y` (resolved relative to the given base).
126 | 
127 | ## Specification consequences
128 | 
129 | Here are some things to note about the above proto-spec:
130 | 
131 | ### Only impacts module loading
132 | 
133 | Because this specification only modifies module specifier resolution and module fetching, its impact is limited. Unlike `blob:` URLs that wrap `Blob` objects, `std:` URLs are not something that can be seen by `fetch()`, `<img>`, etc.
134 | 
135 | Indeed, it's not really accurate to treat `std:x|y` URLs as URLs at all. They are more just mechanisms for interfacing with the platform's module resolution system. They do not have an origin; they are not seen by service workers; they are not affected by CSP.
136 | 
137 | This is in some ways similar to `blob:` URLs that wrap `MediaSource` objects: those are handled specially in HTML's [media resource fetch algorithm](https://html.spec.whatwg.org/#concept-media-load-resource), and never make it to platform's general fetching infrastructure (e.g. the [scheme fetch](https://fetch.spec.whatwg.org/#scheme-fetch) algorithm asserts that it never sees such `blob:` URLs).
138 | 
139 | Future work may make it possible to reuse JavaScript import specifiers in other URL-accepting contexts, e.g. people have proposed `<img src="package:bootstrap/arrow.svg">` to [integrate with package name maps](https://github.com/domenic/package-name-maps/issues/23). However, this would need separate work to integrate with `std:x|y` URLs, since we would need to preserve the property of these resources being opaque to hide their implementation details.
140 | 
141 | ### Feature detection
142 | 
143 | Feature detection of the layered API mechanism is possible using code such as the following:
144 | 
145 | ```html
146 | <script type="module">
147 | (async () => {
148 |   let layeredAPIsSupported = false;
149 |   try {
150 |     await import("std:blank");
151 |     layeredAPIsSupported = true;
152 |   } catch {}
153 | 
154 |   if (!layeredAPIsSupported) {
155 |     // Load polyfills the old-fashioned way.
156 |   }
157 | })();
158 | ```
159 | 
160 | This feature detection is asynchronous, but does not require any network round-trip.
161 | 
162 | ### Relative fallback URLs work
163 | 
164 | Although all of our examples tend to use absolute fallback URLs, since we anticipate fallbacks being located on CDNs or similar, relative URLs also work:
165 | 
166 | ```js
167 | import { storage } from "std:async-local-storage|lib/polyfills/als.mjs";
168 | ```
169 | 
170 | ```html
171 | <script type="module" src="std:virtual-list|/node_modules/virtual-list/element.mjs">
172 | ```
173 | 


--------------------------------------------------------------------------------