├── rfc-139 ├── fb-h3.png ├── youtube-h3.png ├── google-search-h3.png ├── visual-filmstrip.png ├── visual-progress.png ├── http3-browser-usage.png ├── 5_H2_2files_multiplexed.png └── QUIC-illustration-final.jpg ├── rfc-115 ├── clean-connections.png ├── connection-view.png ├── h2-dns-annotated.png ├── cold-cache-summary.png ├── the-impact-annotated.png ├── connection-view-annotated.png └── underutilised-connections-annotated.png ├── rfc-138 ├── total-bytes-search.png ├── brotli-usage-caniuse.png ├── total-bytes-homepage.png ├── visual-progress-covid.png ├── visual-progress-search.png ├── total-bytes-coronavirus.png ├── visual-progress-homepage.png ├── total-bytes-bank-holidays.png └── visual-progress-bank-holidays.png ├── rfc-171 ├── ie_usage_last_year.png └── ie_usage_last_three_years.png ├── rfc-172 └── pgexplain-query-plan-visualisation.png ├── .github └── workflows │ └── actionlint.yml ├── rfc-000-template.md ├── rfc-066-use-github-for-rfcs.md ├── rfc-047-add-another-timestamp-to-content-items.md ├── rfc-045-consolidate-sidekiq-usage-into-shared-gem.md ├── rfc-046-break-ability-to-replay-publishing-api-event-log.md ├── rfc-085-special-route-publisher.md ├── rfc-053-terminology-for-migration-progress.md ├── rfc-028-keeping-gov-uk-s-software-current.md ├── rfc-067-tagging-to-organisations.md ├── rfc-103-merge-dependabot-pull-requests-with-a-single-review.md ├── rfc-030-customise-call-to-action-text-on-simple-smart-answers-start-page.md ├── rfc-015-environment-names.md ├── rfc-088-external-content.md ├── rfc-041-separate-document-type-from-format.md ├── rfc-083-case-insensitive-routing.md ├── rfc-063-naming-new-apps-gems-on-gov-uk.md ├── rfc-017-simpler-draft-stack.md ├── rfc-035-explicitly-make-the-details-hash-non-opaque.md ├── rfc-147-enable-speedcurve-http-protocol-capture.md ├── rfc-078-re-architect-signin-permissions-in-signon.md ├── rfc-075-managing-users.md ├── rfc-025-managing-special-snowflake-urls-through-the-publishing-api.md ├── rfc-043-content-items-without-a-base-path.md ├── rfc-057-default-values-for-application-secrets.md ├── rfc-086-draft-stack-rummager.md ├── rfc-058-publishing-api-events.md ├── rfc-091-sharing-assets.md ├── rfc-050-do-end-to-end-testing-of-gov-uk-applications.md ├── rfc-165-add-api-endpoints-to-transition.md ├── rfc-159-switch-off-whitehall-apis.md ├── rfc-023-putting-detailed-guides-paths-under-guidance.md ├── rfc-022-putting-elasticsearch-backups-in-s3.md ├── rfc-042-testing-backend-applications-against-production-traffic.md ├── rfc-036-stop-preserving-order-for-links.md ├── rfc-044-unpublishing-content-items.md ├── rfc-106-docker-for-local-development.md ├── rfc-097-verify-specific-start-pages.md ├── README.md ├── rfc-055-content-history.md ├── rfc-059-workflow-for-making-changes-to-the-schemas.md ├── rfc-016-how-to-prevent-published-live-frontends-from-reading-from-the-draft-content-store.md ├── rfc-087-dealing-with-errors.md ├── rfc-027-supporting-slug-changes-in-the-publishing-api.md ├── rfc-145-unarchive-govuk_admin_template.md ├── rfc-098-csp.md ├── rfc-100-linting.md ├── rfc-064-killing-router-data.md ├── rfc-013-thoughts-on-access-limiting-in-draft.md ├── rfc-052-pull-request-merging-process.md ├── rfc-056-ordered-link-types.md ├── rfc-163-cdn-rationalisation.md ├── rfc-186-replacement-of-email-based-fact-checking-process-in-publisher.md ├── rfc-146-production-deploy-access.md ├── rfc-108-including-gem-component-assets.md ├── rfc-093-retire-govuk-cdn-logs-monitor.md ├── rfc-142-add-interest-cohort-permssion-policy-header.md ├── rfc-138-enable-brotli-compression.md ├── rfc-143-split-database-instances.md ├── rfc-136-remove-our-backup-cdn-in-gcp.md └── rfc-158-port-content-store-to-postgresql.md /rfc-139/fb-h3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alphagov/govuk-rfcs/HEAD/rfc-139/fb-h3.png -------------------------------------------------------------------------------- /rfc-139/youtube-h3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alphagov/govuk-rfcs/HEAD/rfc-139/youtube-h3.png -------------------------------------------------------------------------------- /rfc-115/clean-connections.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alphagov/govuk-rfcs/HEAD/rfc-115/clean-connections.png -------------------------------------------------------------------------------- /rfc-115/connection-view.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alphagov/govuk-rfcs/HEAD/rfc-115/connection-view.png -------------------------------------------------------------------------------- /rfc-115/h2-dns-annotated.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alphagov/govuk-rfcs/HEAD/rfc-115/h2-dns-annotated.png -------------------------------------------------------------------------------- /rfc-139/google-search-h3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alphagov/govuk-rfcs/HEAD/rfc-139/google-search-h3.png -------------------------------------------------------------------------------- /rfc-139/visual-filmstrip.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alphagov/govuk-rfcs/HEAD/rfc-139/visual-filmstrip.png -------------------------------------------------------------------------------- /rfc-139/visual-progress.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alphagov/govuk-rfcs/HEAD/rfc-139/visual-progress.png -------------------------------------------------------------------------------- /rfc-115/cold-cache-summary.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alphagov/govuk-rfcs/HEAD/rfc-115/cold-cache-summary.png -------------------------------------------------------------------------------- /rfc-138/total-bytes-search.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alphagov/govuk-rfcs/HEAD/rfc-138/total-bytes-search.png -------------------------------------------------------------------------------- /rfc-139/http3-browser-usage.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alphagov/govuk-rfcs/HEAD/rfc-139/http3-browser-usage.png -------------------------------------------------------------------------------- /rfc-171/ie_usage_last_year.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alphagov/govuk-rfcs/HEAD/rfc-171/ie_usage_last_year.png -------------------------------------------------------------------------------- /rfc-115/the-impact-annotated.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alphagov/govuk-rfcs/HEAD/rfc-115/the-impact-annotated.png -------------------------------------------------------------------------------- /rfc-138/brotli-usage-caniuse.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alphagov/govuk-rfcs/HEAD/rfc-138/brotli-usage-caniuse.png -------------------------------------------------------------------------------- /rfc-138/total-bytes-homepage.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alphagov/govuk-rfcs/HEAD/rfc-138/total-bytes-homepage.png -------------------------------------------------------------------------------- /rfc-138/visual-progress-covid.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alphagov/govuk-rfcs/HEAD/rfc-138/visual-progress-covid.png -------------------------------------------------------------------------------- /rfc-138/visual-progress-search.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alphagov/govuk-rfcs/HEAD/rfc-138/visual-progress-search.png -------------------------------------------------------------------------------- /rfc-138/total-bytes-coronavirus.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alphagov/govuk-rfcs/HEAD/rfc-138/total-bytes-coronavirus.png -------------------------------------------------------------------------------- /rfc-138/visual-progress-homepage.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alphagov/govuk-rfcs/HEAD/rfc-138/visual-progress-homepage.png -------------------------------------------------------------------------------- /rfc-139/5_H2_2files_multiplexed.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alphagov/govuk-rfcs/HEAD/rfc-139/5_H2_2files_multiplexed.png -------------------------------------------------------------------------------- /rfc-139/QUIC-illustration-final.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alphagov/govuk-rfcs/HEAD/rfc-139/QUIC-illustration-final.jpg -------------------------------------------------------------------------------- /rfc-115/connection-view-annotated.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alphagov/govuk-rfcs/HEAD/rfc-115/connection-view-annotated.png -------------------------------------------------------------------------------- /rfc-138/total-bytes-bank-holidays.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alphagov/govuk-rfcs/HEAD/rfc-138/total-bytes-bank-holidays.png -------------------------------------------------------------------------------- /rfc-171/ie_usage_last_three_years.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alphagov/govuk-rfcs/HEAD/rfc-171/ie_usage_last_three_years.png -------------------------------------------------------------------------------- /rfc-138/visual-progress-bank-holidays.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alphagov/govuk-rfcs/HEAD/rfc-138/visual-progress-bank-holidays.png -------------------------------------------------------------------------------- /rfc-172/pgexplain-query-plan-visualisation.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alphagov/govuk-rfcs/HEAD/rfc-172/pgexplain-query-plan-visualisation.png -------------------------------------------------------------------------------- /rfc-115/underutilised-connections-annotated.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alphagov/govuk-rfcs/HEAD/rfc-115/underutilised-connections-annotated.png -------------------------------------------------------------------------------- /.github/workflows/actionlint.yml: -------------------------------------------------------------------------------- 1 | name: Lint GitHub Actions 2 | on: 3 | push: 4 | paths: ['.github/**'] 5 | jobs: 6 | actionlint: 7 | runs-on: ubuntu-latest 8 | steps: 9 | - uses: actions/checkout@v4 10 | with: 11 | show-progress: false 12 | - uses: alphagov/govuk-infrastructure/.github/actions/actionlint@main 13 | -------------------------------------------------------------------------------- /rfc-000-template.md: -------------------------------------------------------------------------------- 1 | --- 2 | status: proposed 3 | implementation: proposed 4 | status_last_reviewed: 5 | --- 6 | 7 | # My RFC Title 8 | 9 | ## Summary 10 | 11 | An abstract, tl;dr or executive summary of your RFC. 12 | 13 | ## Problem 14 | 15 | Describe the problem your RFC is trying to solve. 16 | 17 | ## Proposal 18 | 19 | Describe your proposal, with a focus on clarity of meaning. You MAY use [RFC2119-style](https://www.ietf.org/rfc/rfc2119.txt) MUST, SHOULD and MAY language to help clarify your intentions. 20 | -------------------------------------------------------------------------------- /rfc-066-use-github-for-rfcs.md: -------------------------------------------------------------------------------- 1 | ## Problem 2 | 3 | - The confluence wiki is closed to people outside of GOV.UK 4 | - Not everyone is a fan of the wiki's user interface 5 | - It's hard to keep track of changes to RFCs during the proposal period 6 | 7 | ## Proposal 8 | 9 | - Move the RFCs to GitHub 10 | - RFCs are proposed as markdown documents and discussed in a pull request 11 | 12 | ## Implementation details 13 | 14 | - We'll create a repository **alphagov/govuk-rfcs** 15 | - We'll create pull requests for each existing RFC so that we keep the current numbering system 16 | 17 | -------------------------------------------------------------------------------- /rfc-047-add-another-timestamp-to-content-items.md: -------------------------------------------------------------------------------- 1 | ## Problem 2 | 3 | One of the goals of migration is to make publishing apps use the publishing api as their data store. Publishing apps contain index pages that list all of the content they manage. These index pages are sorted by "most recently updated first" to help users find content they're working on more easily. The publishing api doesn't have a timestamp for this. The closest it has is updated\_at, but this is affected when content is republished. We could insist that republishing must happen in the same order that content appears on index pages, but republishing is often done via sidekiq, which doesn't guarantee message ordering. We also display this timestamp on index pages and it could confuse users if it changes as a result of republishing. 4 | 5 | ## Proposal 6 | 7 | Introduce a new timestamp for content items in publishing api that is updated whenever there is a major or minor update, but not on republishes. This field should be called something like 'edited\_at' or 'private\_updated\_at' and is for internal use only. It should not be sent downstream to the content store. Publishing apps then specify this sort order in the request to publishing api when requesting content for index pages. 8 | 9 |   10 | 11 |   12 | 13 | -------------------------------------------------------------------------------- /rfc-045-consolidate-sidekiq-usage-into-shared-gem.md: -------------------------------------------------------------------------------- 1 | ## Problem 2 | 3 | The Publishing Platform team is currently working on adding request tracing to all apps that use the Publishing API in an asynchronous way via Sidekiq - [https://github.com/alphagov/collections-publisher/pull/194](https://github.com/alphagov/collections-publisher/pull/194) and [https://github.com/alphagov/whitehall/pull/2567](https://github.com/alphagov/whitehall/pull/2567). This will need repeating across many apps. The implementation is sufficiently complex that a mass-change of all apps in the future is not unlikely. 4 | 5 | A previous shotgun surgery on apps with Sidekiq was the adding of sidekiq-statsd - [https://trello.com/c/z2aHqwS8/48-add-sidekiq-statsd-to-apps-that-use-sidekiq](https://trello.com/c/z2aHqwS8/48-add-sidekiq-statsd-to-apps-that-use-sidekiq) which needed ~10 PRs. 6 | 7 | There are ~15 GOV.UK apps that use Sidekiq, and they all use slightly different versions, logging and configuration. 8 | 9 | ## Proposal 10 | 11 | Introduce a `govuk-sidekiq` gem that consolidates all GOV.UK Sidekiq conventions: 12 | 13 | - Use automatic request tracing (based on [https://github.com/alphagov/whitehall/pull/2567](https://github.com/alphagov/whitehall/pull/2567)) 14 | - Use sidekiq-statsd (like [https://github.com/alphagov/imminence/pull/117](https://github.com/alphagov/imminence/pull/117)) 15 | - Use logging (`sidekiq-logging-json` is used very inconsistently) 16 | - Perhaps setup & configuration 17 | 18 | This will make all our apps easier to manage and upgrade. 19 | 20 |   21 | 22 |   23 | 24 |   25 | 26 |   27 | 28 |   29 | 30 |   31 | 32 | -------------------------------------------------------------------------------- /rfc-046-break-ability-to-replay-publishing-api-event-log.md: -------------------------------------------------------------------------------- 1 | ## Problem 2 | 3 | The Event Log is a log of all requests received by the publishing api. Theoretically, we should be able to replay each of these events in turn against a blank database and end up with the same content as in the current live publishing api database. 4 | 5 | In practice, this replaying of events takes too long to be useful for disaster recovery. It also prevents in situ changes to data via data migrations and the Rails console, as these changes are not recorded in the event log, and thus wouldn't be replayed. There have already been data migrations and instances of "devopsing" in production which has broken the integrity of the Event log. 6 | 7 | ## Proposal 8 | 9 | In order to allow for more flexibility to developers to make ad hoc changes to data in ways that are not supported by the available commands, we're proposing we remove the implied contract that the event log can be replayed at any time. In practice, it is unlikely we'll ever need to do this. The event log is still very important as an audit trail of actions that have occurred in the publishing API, so this change will enable us to change how the log is stored long term. There are already almost 1.5m events in the log, and postgresql is not the most efficient solution for storing these.  10 | 11 | The event log needs to be retained for now as the primary key for the table is used as an atomic counter for content items lock versions. We could in future archive events into cold storage after a certain time period. Other necessary changes to data that would not be logged in the Event log should be performed using Rails' migration functionality. 12 | 13 |   14 | 15 |   16 | 17 | -------------------------------------------------------------------------------- /rfc-085-special-route-publisher.md: -------------------------------------------------------------------------------- 1 | # Special Route Publisher app 2 | 3 | ## Summary 4 | 5 | This RFC proposes the creation of a Special Route Publisher app to cater for routes that require registration but have no content. 6 | 7 | ## Problem 8 | 9 | There are examples of certain routes that have no direct content and therefore do not live in any particular publishing but still require a route to be registered via the the Publishing-Api -> Content Store. Examples of these can be seen in [Frontend](https://github.com/alphagov/frontend/blob/master/lib/special_route_publisher.rb) and [Rummager](https://github.com/alphagov/rummager/blob/master/lib/tasks/publishing_api.rake) where the former publishes many routes, such as site search, and the latter the site map. Each app then has a rake task which is triggered on deploy and updates the routes via the Publishing API. 10 | 11 | This creates some inconsistencies/issues: 12 | 13 | 1. Frontend apps should ideally not talk directly to the Publishing API. 14 | This has become more relevant recently whilst moving site search into Finder Frontend. 15 | 16 | 2. No central location for the administration of special routes. 17 | 18 | 3. No consensus as to which app should take responsibility for 19 | publishing any future special routes. 20 | 21 | 4. Each app that publishes these routes does so in slightly different ways. 22 | 23 | ## Proposal 24 | 25 | Create a Special Route Publisher app to handle the administration of any 26 | special routes that have no direct pieces of content and therefore 27 | cannot obviously live in any other publishing app. 28 | 29 | The app could be used to publish other types of content items in the 30 | future, but this is out of scope for now. For example, external links 31 | are not currently sent to the Publishing API but are sent to search. 32 | Tracking them as content items would allow them to be used elsewhere. 33 | -------------------------------------------------------------------------------- /rfc-053-terminology-for-migration-progress.md: -------------------------------------------------------------------------------- 1 | ## Problem 2 | 3 | So far, we've been describing the progress of Migration using the terms "phase 1" and "phase 2". These terms have no formal definition, have become overloaded and can incorrectly imply the intent for our work. They were born out of casual conversation when Migration started, but are now used formally in our roadmap. There is confusion about the definition of the terms, and they are mis-applied to applications rather than formats. The terms also mask additional complexity, "phase 1" is more complicated than a single stage of work. 4 | 5 | ## Proposal 6 | 7 | Firstly, we should stop applying these states to entire applications. The only time we're able to refer to an application as a single state on the migration journey is when all of its formats are fully migrated, at which point we can stop talking about it altogether. A publishing application is responsible for a range of formats, and due to the process of migration these formats can and will be in different states. 8 | 9 | Our proposed terminology is: 10 | 11 | | Name | Description | 12 | | --- | --- | 13 | | **Pre-migration** | Publishing API is unaware of the existence of this format. The publishing app does not send this content to the API. | 14 | | **Placeholder** | Placeholders are sent to the Publishing API to represent the existence of documents of this format. They have at least a base path (where appropriate) and a title. | 15 | | **Content complete** | The Publishing API has everything needed for a frontend application to render the static content of documents of this format. The frontend application does not yet use this content to render documents. | 16 | | **Rendered** | As above, but a frontend application makes a request to the content store in order to render the static content. The publishing application is still the source of truth for the content, not the Publishing API. | 17 | | **Migrated** | The Publishing API is the canonical source of truth for the content for this format. The publishing application treats the Publishing API as its database, and the content can be removed from the publishing application's database. | 18 | 19 | -------------------------------------------------------------------------------- /rfc-028-keeping-gov-uk-s-software-current.md: -------------------------------------------------------------------------------- 1 | --- 2 | status: superseded 3 | implementation: superseded 4 | status_last_reviewed: 2024-03-06 5 | status_notes: 'GOV.UK has a policy on keeping software current: https://docs.publishing.service.gov.uk/manual/keeping-software-current.html' 6 | --- 7 | 8 | # Keeping GOV.UK's software current 9 | 10 | One of our core values is to use secure and up to date software. This document lays out the recommendations for keeping our Ruby on Rails software current. 11 | 12 | ## Introduction 13 | 14 | We run a lot of Rails applications. This means that we have dependencies on both Rails and Ruby versions. 15 | 16 | ## Upgrading Rails 17 | 18 | It's very important that we're running a currently supported version of Rails for all applications, otherwise we **aren't covered**  by [security fixes](http://rubyonrails.org/security/). We should: 19 | 20 | - Be running on the current major version - this currently means `4.y.z` 21 | - Maintain our applications at the latest current bugfix release for the minor version we're on (expressed in Gemfile syntax as: `~> X.Y.Z`) - this currently means `4.1.8` and `4.2.3` 22 | - Keep abreast of breaking changes for the next major version (`5.y.z`), and have a plan to migrate our apps before `4.2.x` is deprecated 23 | 24 | ## Upgrading Ruby 25 | 26 | New versions of Ruby bring us improved performance and nicer syntax for certain things, but also can cause issues with the libraries etc. we use. We should: 27 | 28 | - Be running on the current major version - this currently means `2.y.z` 29 | - Maintain our applications at the current or next-to-current minor version - this means `2.2.z` or `2.1.z`, depending on your app's dependencies 30 | 31 | ## Current state 32 | 33 | The current state of the Ruby and Rails versions is: 34 | 35 | - [Listed in this versions spreadsheet](https://docs.google.com/spreadsheets/d/1FJmr39c9eXgpA-qHUU6GAbbJrnenc0P7JcyY2NB9PgU/edit#gid=1480786499) by alext 36 | - [Another spreadsheet with team ownership](https://docs.google.com/a/digital.cabinet-office.gov.uk/spreadsheets/d/17SaFqFqVEMoabq-FjEeCHpUmA5yAjqLr_Vt-lDeXMsE/edit?usp=sharing) by alexandria.jackson. 37 | 38 | -------------------------------------------------------------------------------- /rfc-067-tagging-to-organisations.md: -------------------------------------------------------------------------------- 1 | # Tagging to organisations (not implemented) 2 | 3 | --- 4 | **NOTE 2017/06/12**: This PR has not been implemented because of difficulty getting consensus about the semantics of the organisations tags. The only tag that has been added at the time of writing is the `primary_publishing_organisation`, which contains the first "lead organisation" for Whitehall documents. 5 | --- 6 | 7 | ## Problem 8 | 9 | Historically, we've had "organisation tagging" on GOV.UK, but it's a bit 10 | muddled. In particular, the value of "organisations" in the content store means 11 | different things for different publishing apps. 12 | 13 | ## Model 14 | 15 | There will be three organisation tagging options for any page on GOV.UK: 16 | 17 | - **Primary publishing organisation** This is the publisher, or "owner" of the 18 | page. 19 | - **Additional publishing organisations** In cases when there's more than 1 20 | organisation that has worked on the content. 21 | - **Related organisations** The content is related to these organisations. 22 | 23 | ## Implementation 24 | 25 | - The content store will expose the three organisation link types separately 26 | - The search API will expose the three organisation link types separately, but 27 | also expose a compound field that contains all organisations 28 | - We'll send the three types as separate dimensions to analytics, but also a 29 | compound field that contains all the organisations 30 | 31 | ## Usage 32 | 33 | - Survey team can use primary publishing org from the search API to show 34 | breakdown per org 35 | - Content performance manager can use the primary publishing org to show 36 | ownership 37 | - Content transformation will use the primary publishing org to assign content 38 | - We'll use all the organisations combined to do facetted search 39 | 40 | ## Mapping 41 | 42 | ### Whitehall 43 | 44 | - The first lead org is the primary publishing org 45 | - The other lead orgs become the additional publishing orgs 46 | - The supporting orgs become the related orgs 47 | 48 | ### Publisher 49 | 50 | - The primary publishing org will be **Government Digital Service** 51 | - The current "Organisations" become related orgs 52 | -------------------------------------------------------------------------------- /rfc-103-merge-dependabot-pull-requests-with-a-single-review.md: -------------------------------------------------------------------------------- 1 | --- 2 | status: accepted 3 | implementation: done 4 | status_last_reviewed: 2024-03-04 5 | --- 6 | 7 | # Merge dependabot pull requests with a single review 8 | 9 | ## Problem 10 | 11 | I think there are two somewhat related problems that I'd like to see 12 | addressed. 13 | 14 | Pull Requests by Dependabot, that the guidance describes as requiring 15 | a review from two people are being merged with a single review. 16 | 17 | Pull Requests by Dependabot are up for long amounts of time, without 18 | being merged, including those that could be time sensitive like 19 | security fixes. 20 | 21 | ## Proposal 22 | 23 | I think a step forward in addressing both problems would be to not 24 | treat Dependabot as an external contributor when reviewing Pull 25 | Requests, and amend the guidance on reviewing and merging Pull 26 | Requests to permit merging Pull Requests by Dependabot with a single 27 | review. 28 | 29 | ## Rationale 30 | 31 | Currently, Pull Requests by Dependabot are viewed as coming from an 32 | external contributor, and as such, should be reviewed by two people 33 | employed by GDS, working on GOV.UK. 34 | 35 | However, Dependabot is already a special case in the following ways: 36 | 37 | - It's an automated service, not a person raising the Pull Requests 38 | - Dependabot pushes changes directly to the repositories hosted on 39 | the alphagov GitHub organisation, rather than using a fork of the 40 | repository 41 | - It should only be changing files in the repositories relating to 42 | the versions of dependencies (Gemfile and Gemfile.lock in the case 43 | of Rubygems/Bundler) 44 | 45 | Therefore, to try and decrease the amount of time Pull Requests remain 46 | open, and reduce the amount of time spent reviewing them, the proposal 47 | is to amend the guidance to only require a single review on GitHub. 48 | 49 | Since the introduction of Dependabot, merging Pull Requests with only 50 | a single review has been happening, including when this doesn't adhere 51 | to the guidance. Having more approvals than necessary is good, but 52 | having fewer doesn't match up with the current guidance, so this 53 | change would make the guidance and practice line up better. 54 | -------------------------------------------------------------------------------- /rfc-030-customise-call-to-action-text-on-simple-smart-answers-start-page.md: -------------------------------------------------------------------------------- 1 | --- 2 | status: accepted 3 | implementation: done 4 | status_last_reviewed: 2024-03-06 5 | --- 6 | 7 | # Customise call-to-action text on simple Smart Answer start pages 8 | 9 | ## Problem 10 | 11 | According to a colleague people get confused by 'Start now' on some Simple Smart Answers' start pages when it's not related to the thing they're doing, eg paying or contacting a department/agency. 12 | 13 | For example, there are roughly 30 contacts per day for 1st line about the contact [dvla](https://www.gov.uk/contact-the-dvla) / [dvsa](https://www.gov.uk/contact-dvsa) Simple Smart Answers. See [feedex entries for /contact-dvsa](https://support.production.alphagov.co.uk/anonymous_feedback?path=%2Fcontact-dvsa) and [/contact-the-dvla](https://support.production.alphagov.co.uk/anonymous_feedback?path=%2Fcontact-the-dvla). 14 | 15 | ## Proposal 16 | 17 | Allow customising the text value on the call to action button in the publisher application. Currently its value is [hard-coded to 'Start now'](https://github.com/alphagov/frontend/blob/d9e2852faf4d47a26c9e9c2192f3747f90a7ed3c/app/views/root/simple_smart_answer.html.erb#L9).  18 | 19 |   20 | 21 | Introducing this change would affect at least three repos: 22 | 23 |   24 | 25 | **1) govuk\_content\_models** 26 | 27 | Simple Smart Answers would need an additional attribute to store the text value of the action button. Content designers suggest a free-text field limited to ~15 characters. 28 | 29 | All existing Simple Smart Answers would need a data migration to set this value to 'Start now'. 30 | 31 |   32 | 33 | **2) publisher** 34 | 35 | Publisher would need to show the new attribute in the UI. I've mocked up what it could look like (Action button): 36 | 37 |   38 | 39 | **3) frontend** 40 | 41 | Frontend would need to show the variable attribute value on the call to action button. 42 | 43 | Since the change would touch applications looked after by different teams, I would like to make sure it does not clash with the vision of the future of those applications. 44 | 45 | I would also like to receive comments about whether this is the right solution to the problem and I am not violating any user experience guidelines. 46 | 47 |   48 | 49 |   50 | 51 | -------------------------------------------------------------------------------- /rfc-015-environment-names.md: -------------------------------------------------------------------------------- 1 | --- 2 | status: accepted 3 | implementation: done 4 | status_last_reviewed: 2024-03-06 5 | --- 6 | 7 | # Environment names 8 | 9 | ## Problem 10 | 11 | The current naming scheme for our different environments is confusing and does not match the standard definitions. We need to come up with a set of definitive names that avoid confusion and allow us and our publishing users to distinctly identify particular environments. 12 | 13 | ## Proposal 14 | 15 | | Environment name | Live URL | Preview URL | Existing environment name | Existing live URL | 16 | | ---------------- | -------- | ----------- | ------------------------- | ----------------- | 17 | | Production | www.gov.uk (www.publishing.service.gov.uk) | www-preview.publishing.service.gov.uk | Production | www.gov.uk | 18 | | Staging | www.staging.publishing.service.gov.uk | www-preview.staging.publishing.service.gov.uk | Staging | production.alphagov.co.uk | 19 | | Integration | www.integration.publishing.service.gov.uk | www-preview.integration.publishing.service.gov.uk | Preview | preview.alphagov.couk | 20 | 21 | This involves reassigning some existing names for other purposes, and using some completely new names. In particular, what is currently known as the "Draft" stack becomes "Preview". This has two benefits: 22 | 23 | - It matches what editors are expecting to do - that is, preview their content before making it live. 24 | - It weans them off using the separate environment that we currently call "preview", allowing us to rename it to match its actual intended usage, which is for integration testing. 25 | 26 | Note that the transition will not be as painful as it could be, as no editors are currently using "draft". The main re-education task will be to stop thinking of "preview" as a scratchpad that gets reset regularly, but as somewhere to actually preview content that is going to be live. 27 | 28 | Also note that thanks to some reconfiguration that Infrastructure are doing as part of the move away from Skyscape, the domain name for all environments will change from "alphagov.co.uk" to "publishing.service.gov.uk". Since we retain control of alphagov, we will be able to redirect as appropriate. 29 | 30 | In addition, this work will give a specific URL to the staging environment, rather than using the production domain and having to edit the hosts file. 31 | -------------------------------------------------------------------------------- /rfc-088-external-content.md: -------------------------------------------------------------------------------- 1 | --- 2 | status: accepted 3 | implementation: done 4 | status_last_reviewed: 2024-03-04 5 | --- 6 | 7 | # External content 8 | 9 | ## Summary 10 | 11 | Create an `external_content` document type to represent pages that aren't part 12 | of GOV.UK but are relavent to our users. 13 | 14 | ## Problem 15 | 16 | Not everything government related is on GOV.UK. 17 | 18 | Search admin stores "External links", which can be returned for certain search terms. 19 | 20 | For example: 21 | - searches for "mp" can return the parliament.uk page http://www.parliament.uk/mps-lords-and-offices/mps/ 22 | - searches for "fit for work" can return http://fitforwork.org/ 23 | - searches for council names can return the council website 24 | 25 | We are currently changing search indexing to source all its content from the publishing API, but external links 26 | have never been part of the publishing API. 27 | 28 | ## Proposal 29 | 30 | Create an `external_content` document type so we can store these links in the publishing API. 31 | 32 | Create an `external_content` schema. 33 | 34 | The schema MUST store in the details hash: 35 | - `hidden_search_terms` - a set of search keywords/phrases that should route users to the page. [This field has already been added for smart answers](https://github.com/alphagov/govuk-content-schemas/pull/685/files). 36 | - `url` - a URL for the external resource 37 | 38 | The schema MAY store information about change history. 39 | 40 | The standard fields `title` and `description` MUST be set. 41 | 42 | In accordance with [RFC 43](https://github.com/alphagov/govuk-rfcs/blob/master/rfc-043-content-items-without-a-base-path.md) The `base_path`, `rendering_app`, `redirects` and `routes` MUST NOT be set. 43 | 44 | ## Consequences 45 | 46 | - External content can be defined centrally and reused across the platform 47 | - The standard `links` hash can be used to refer to external content. The platform does not make any distinction between internal and external content. 48 | 49 | This means that `external_related_links` in the details hash is technically redundant, and external links could be managed independently of the publishing workflow (for example, through content tagger). 50 | 51 | This RFC acknowledges but does not address this duplication. The intention is only to move the search admin concept of an external link into the publishing API. 52 | -------------------------------------------------------------------------------- /rfc-041-separate-document-type-from-format.md: -------------------------------------------------------------------------------- 1 | ## Problem 2 | 3 | The publishing system uses the `format` element of a content item for three purposes: 4 | 5 | - to identify the schema used to validate a content item 6 | - to determine the way the item is displayed on the frontend 7 | - (in phase 2) to filter lists of objects in publishing apps 8 | 9 | Although these use cases are related, it is not necessarily the case that a schema maps directly to a document type: one example is in specialist-publisher, where there is one single schema for "specialist document" but many different types of document, eg CMA cases, AAIB reports, drug safety alerts, etc, which need to be displayed separately in the publishing app. 10 | 11 | This is likely to become more significant when the work is done to consolidate formats; at that point we might only have a small selection of schemas and front-end templates, but potentially still need to distinguish sub-types in the publishing apps. 12 | 13 | Currently, specialist documents define a "document type" field in the details hash; however, because it is not at the top level of the document, it is not available for use in filtering. 14 | 15 | ## Proposal 16 | 17 | We will deprecate the current `format` field, and replace it with two new fields: 18 | 19 | - `schema_name` - determines which file in govuk-content-schemas is used to validate the item. 20 | - `document_type` - used for frontend display and for filtering in publishing apps  21 | 22 | ### Migration 23 | 24 | We will support both naming types for a period to ease transition, and modify publishing-api to supply missing data where necessary. If only the old `format` field is supplied, its value will be copied to `schema_name` and document`_type`. If `format` is not supplied, its value will be copied from `schema`. It will be an error to supply only one of `schema` and `content_type`. 25 | 26 | During the deprecation period, all three fields will be supplied to content-store, and both `document_type` and `format` will be passed from there to the frontends. There are only a small number of apps that the `format` field in the frontend; as soon as they is updated to use `document_type`, the deprecated field will be dropped from content-store and the frontend representation, and the schemas updated. 27 | 28 |   29 | 30 |   31 | 32 | -------------------------------------------------------------------------------- /rfc-083-case-insensitive-routing.md: -------------------------------------------------------------------------------- 1 | # Case insensitive routing on GOV.UK 2 | 3 | ## Summary 4 | 5 | Make base paths case insensitive for routing purposes, and only allow one case of a base path to be registered with the router and publishing-api. 6 | 7 | ## Problem 8 | 9 | Our stack currently regards all base paths to be case-sensitive. The router carries out routing on this basis. Although nginx has a configuration that redirects all-uppercase base paths to their lowercase equivalent, there are no rules for mixed-case base paths. 10 | 11 | In addition, publishing-api allows multiple content items where the base paths are only differentiated by case. 12 | 13 | This has the potential to cause confusion for end users if they try to visit a page using a mixed-case path. In most circumstances, it results in a 404 error, but in the case of prefix routes, the result can be unpredictable since the routing is handled by the backend application. 14 | 15 | It can be argued that most end users do not understand and do not care about case sensitivity, and would be surprised to learn that there is the potential for www.gov.uk/education to lead to a navigation page whereas www.gov.uk/Education leads to a 404 error, or even worse, can be claimed by a completely different app displaying a different page. 16 | 17 | Case sensitivity as a general rule made sense when all requests were for files stored on a file system which was itself case sensitive. However, given that most of our base paths are virtual and routed to apps, this argument does not apply. 18 | 19 | ## Proposal 20 | 21 | * The router should only allow one case (uppercase, lowercase or mixed case) of a base path to be registered as a route (so `/education` can be registered but `/Education` will be regarded to be a duplicate) 22 | * The router should match requested base paths to routes in a case insensitive manner (so a request for `/EDUCATION` will match `/education`) 23 | * The router should redirect requested base paths that match a route lexically but not in case with a 301 redirect to the route in the case is it registered with (so a request for `/EDUCAtion` will result in a 301 redirect to `/education`) 24 | * publishing-api should be audited to find any existing content that differs only in the case of the base path, and these issues should be resolved 25 | * publishing-api should only allow one case of a base path to be registered 26 | -------------------------------------------------------------------------------- /rfc-063-naming-new-apps-gems-on-gov-uk.md: -------------------------------------------------------------------------------- 1 | ## Problem 2 | 3 | Apps and gems on GOV.UK have history followed different naming schemes. 4 | Recently we've established a pattern we're quite happy with, but we've never 5 | written that down. 6 | 7 | This RFC is an attempt to write down the current conventions. 8 | 9 | ## Naming applications 10 | 11 | Firstly, the [service manual has good guidance on naming 12 | things](https://www.gov.uk/service-manual/design/naming-your-service). 13 | 14 | The most important rules: 15 | 16 | - The name should be self-descriptive. No branding or puns (like Rummager, 17 | Needotron and Maslow) 18 | - Use **dashes** for the URL and GitHub repo 19 | - The name of the app should be the same on GitHub, Puppet and hostname 20 | 21 | ### Publishing applications 22 | 23 | Applications that publish things are named **x-publisher**. 24 | 25 | Good: 26 | 27 | - specialist-publisher 28 | - manuals-publisher 29 | 30 | Not so good: 31 | 32 | - publisher (too generic) 33 | - contacts-admin (could be contacts-publisher) 34 | 35 | ### Frontend applications 36 | 37 | Applications that render content to end users on GOV.UK are named 38 | **x-frontend** 39 | 40 | Good: 41 | 42 | - government-frontend 43 | - email-alert-frontend 44 | 45 | Not so good: 46 | 47 | - collections (could be collections-frontend) 48 | - frontend (too generic) 49 | 50 | ### APIs 51 | 52 | Applications that just expose an API are named **x-api**. 53 | 54 | Good: 55 | 56 | - publishing-api 57 | - email-alert-api 58 | - router-api 59 | 60 | Not so good: 61 | 62 | - rummager (should be search-api) 63 | 64 | ### Admin applications 65 | 66 | Applications that "manage" things can be called **x-manager** or **x-admin** or 67 | **thing-doer**. 68 | 69 | Good: 70 | 71 | - search-admin 72 | - local-links-manager 73 | - content-tagger 74 | 75 | No so good: 76 | 77 | - signonotron2000 78 | - maslow (needs-manager) 79 | 80 | ## Naming gems 81 | 82 | - Use the official [Rubygems naming 83 | convention](http://guides.rubygems.org/name-your-gem/) 84 | - Use underscores for multiple words 85 | - Use `govuk_` prefix if the gem is only interesting to projects within GOV.UK 86 | 87 | Good: 88 | 89 | - govuk\_sidekiq 90 | - govuk\_content\_models 91 | - govuk\_admin\_template 92 | - vcloud-edge\_gateway 93 | 94 | Not so good: 95 | 96 | - slimmer 97 | - plek 98 | - gds-sso (should be gds\_sso, or govuk\_sso) 99 | -------------------------------------------------------------------------------- /rfc-017-simpler-draft-stack.md: -------------------------------------------------------------------------------- 1 | --- 2 | status: superseded 3 | implementation: superseded 4 | status_last_reviewed: 2024-03-06 5 | status_notes: We no longer use these infrastructure components. 6 | --- 7 | 8 | # Simpler draft stack 9 | 10 | ## Problem 11 | 12 | The current draft stack design is laid out as an exact copy of the `frontend`, `API`, and `management` VDCs in the GOV.UK vCloud Organisation. Because we've replicated all our data store clusters (Mongo and ElasticSearch), as well as our monitoring stack, we have over 30 machines in this VDC. 13 | 14 | Additionally, because we've hit the limit of VDCs in our current vCloud Org, we've had to build this VDC in a new org. 15 | 16 | Creating a new org has some extra implications to our setup, namely: 17 | 18 | - It creates deployment complexity - we need extra Jenkins instances to maintain a privileged connection to the new Org, and chain Jenkins jobs across Orgs together to deploy 19 | - It creates operational complexity 20 | - we have to create extra SSH configuration and deploy this to all clients to allow us to easily get on the machines 21 | - we now have at least 2 monitoring stacks to watch (in different orgs) 22 | - we now have at least 2 Kibana instances, so logs are no longer all in one place 23 | - Data synchronisation is a problem because we have to create VPNs between Orgs 24 | 25 | ## Proposal 26 | 27 | To reduce the complexity, we propose to, within the main GOV.UK Production vCloud Org: 28 | 29 | - Add 2 machines to the cache VDC: `draft-cache-{1,2}` 30 | - Add 2 machines to the frontend VDC:` draft-frontend-{1,2}` 31 | - Add 2 machines to the API VDC:` draft-content-store-{1,2}` 32 | 33 | The applications that depend on data stores (`router, content-store, router-api`) will use the existing clusters, but use different database names (eg `draft_content_store_production`). 34 | 35 | This has a few advantages over the previous design: 36 | 37 | - It requires significantly less machines (6 vs 30+) 38 | - It simplifies deployments - we're deploying into a single Org so no need for extra Jenkins machines 39 | - It simplifies operations - we can use the existing Errbit, Icinga, and Kibana setups; and for 2nd line staff no change is required to SSH configuration 40 | 41 | it also has one main negative - the security model previously offered (total separation of content and data/requests) is no longer as simple to come by. We plan to address this using vShield Edge firewalling (see ). 42 | 43 | -------------------------------------------------------------------------------- /rfc-035-explicitly-make-the-details-hash-non-opaque.md: -------------------------------------------------------------------------------- 1 | ## Problem 2 | 3 | In the JSON schemas we use to send data between our systems, we've traditionally tried to treat the `details` part of the payload as entirely opaque (eg no special knowledge should be required of anything between the publishing tool sending and the frontend rendering the payload). 4 | 5 | The reality is that in the [govuk-content-schemas](https://github.com/alphagov/govuk-content-schemas) repository, almost all formats have required fields in the details hash. Examples: 6 | 7 | - [case studies](https://github.com/alphagov/govuk-content-schemas/blob/b02afaac06ddd965e114b3ff577faf1952c628e0/formats/case_study/publisher/details.json#L6-L7) 8 | - [finders](https://github.com/alphagov/govuk-content-schemas/blob/b02afaac06ddd965e114b3ff577faf1952c628e0/formats/finder/publisher/details.json#L6-L7) 9 | - [specialist documents](https://github.com/alphagov/govuk-content-schemas/blob/b02afaac06ddd965e114b3ff577faf1952c628e0/formats/specialist_document/publisher/details.json#L6-L8) 10 | 11 | All these required fields are for the rendering apps to have enough data to fulfil the business logic required of the format. 12 | 13 | In order to provide bespoke validation of the schemas within the publishing API, we need to inspect the JSON payloads and validate them using JSON schema, and provide feedback on specific errors. 14 | 15 | In the future we may also want to provide additional parsing and understanding (eg a future dependency resolver will want to know which fields are `govspeak` vs plain text in order to rewrite links and dependencies, or when attachments get out of virus scanning). 16 | 17 | ## Proposal 18 | 19 | Therefore we propose to no longer treat the `details` part of the payload as entirely opaque, but instead allow for inspection and enhancement when required of the format. 20 | 21 | This requires no work as we're already doing this in code in some places - but we would change our expectations about certain fields (eg datetime fields, govspeak fields, attachments). 22 | 23 | These fields will be marked as custom types in the content schemas (code TBD), and documented, with their addition to be discussed as part of code review. They  **will not** be added on a per format basis - any special fields added should be re-usable across the entire dataset. 24 | 25 |   26 | 27 |   28 | 29 |   30 | 31 |   32 | 33 |   34 | 35 |   36 | 37 |   38 | 39 | -------------------------------------------------------------------------------- /rfc-147-enable-speedcurve-http-protocol-capture.md: -------------------------------------------------------------------------------- 1 | --- 2 | status: accepted 3 | implementation: done 4 | status_last_reviewed: 2024-03-04 5 | --- 6 | 7 | # Allow SpeedCurve to capture user HTTP Protocol Data 8 | 9 | ## Summary 10 | 11 | We use Speedcurve RUM to capture detailed user performance data on GOV.UK. Fastly ammounced [HTTP/3 + QUIC](https://twitter.com/fastly/status/1520139864032874497) is now availabe for all customers for free as it is now out of beta. We already have an [RFC in draft](https://github.com/alphagov/govuk-rfcs/pull/139) about enabling this on GOV.UK. 12 | 13 | But before we enable HTTP/3 + QUIC on GOV.UK I'd like to be able to capture what protocol a user is using via SpeedCurve. This will alow us to quantify if the change has made any difference for users (especially those on unreliable connections). I feel this could be a great bit of research / PR for how we are improving performance for all users of GOV.UK, no matter what their connection or device. 14 | 15 | ## Problem 16 | 17 | Data capture to quantify the change from HTTP/2 + TCP to HTTP/3 + QUIC. 18 | 19 | ## Proposal 20 | 21 | I'd like to include a small piece of JavaScript in the current SpeedCurve RUM implimentation that will only fire when a user accepts the cookie banner. This JavaScript will look at the current HTTP protocol the user is using and push this anonymous data into SpeedCurve using their API. 22 | 23 | This will then give us an additional dimension in the SpeedCurve GUI with which to compare before / after the change is made. 24 | 25 | The actual JavaScript is small and will have a negligible effect of page performance as it is simply reading a string value from an object in a supporting browsers [Navigation Timing Level 2 API](https://www.w3.org/TR/navigation-timing-2/). 26 | 27 | The actual code to include looks like this: 28 | 29 | ```js 30 | // use the LUX.addData method to capture HTTP protocol information. 31 | LUX.addData("http-protocol", performance.getEntriesByType('navigation')[0].nextHopProtocol); 32 | ``` 33 | 34 | We should report the contents of `performance.getEntriesByType('navigation')[0].nextHopProtocol` using the `LUX.addData()` function - SpeedCurve have some good example of this [in their recipes][1]. 35 | 36 | Once this code is added, we should then have information about what protocol the user used and the difference it made to the performance, allowing us to compare aggregate data for HTTP/2 and HTTP/3 users over a set period of time (likely 1-2 months). 37 | 38 | [1]: https://support.speedcurve.com/recipes/track-size-for-a-single-resource 39 | -------------------------------------------------------------------------------- /rfc-078-re-architect-signin-permissions-in-signon.md: -------------------------------------------------------------------------------- 1 | # Re-architect signin permission in signon 2 | 3 | ## Summary 4 | 5 | Change the 'signin' permission in signon to be one that signon itself checks for as part of the oauth handshake and reject users trying to login to applications they don't have access to. 6 | 7 | ## Problem 8 | 9 | In signon all apps have a 'signin' permission and the UI for managing permissions on a user strongly suggests that if the user is not granted this permission for an application they will not be able to access that application. The reality is that it is up to the applications themselves to care about the 'signin' permission via the gds-sso gem and the `require_signin_permission!` controller method. 10 | 11 | This means that for some applications unchecking the "has access to?" box for a user will block them from logging in, but for others it won't. There's nothing in signon that can explain this to an admin, because signon doesn't know which applications are coded to use `require_signin_permission!` and which allow any signon user. Only applications that a user has explicit 'signin' permission for are listed on their dashboard so signon users may not even know they have access to an application that allows all users because it won't be listed if they haven't been granted 'signin' permission. 12 | 13 | More importantly, for those apps that allow any signon user we have no mechanism to stop a user from accessing that application other than suspending their entire account and blocking them from using all applications that use signon. 14 | 15 | ## Proposal 16 | 17 | The proposed solution is that signon handles the 'signin' permission during the oauth handshake and applications cannot circumvent this. This means that the existing signon UI represents the truth of the situation and all applications a user has access to will be listed in their dashboard. To cater for those applications that do want to allow all users to access them signon will provide bulk permission granting and default permission granting functionality. 18 | 19 | * Signon MUST check for 'signin' permission during the oauth handshake with an application and reject users that do not have this permission. 20 | * Signon MUST allow a super admin to grant permissions to all users in one go. 21 | * Signon SHOULD allow super admins to mark permissions for an application as 'default' meaning it is added to all new users. 22 | * gds-sso SHOULD deprecate `require_signin_permission!`. 23 | * applications using signon SHOULD be audited to make sure they will continue to function after these changes are made 24 | * applications using signon SHOULD be audited to collect the set of default permissions that should be created and bulk granted to all existing users 25 | 26 | -------------------------------------------------------------------------------- /rfc-075-managing-users.md: -------------------------------------------------------------------------------- 1 | # Managing users on GOV.UK 2 | 3 | ## Summary 4 | 5 | Create a central repository of users in GOV.UK. Use that to verify that the 6 | correct people have access. 7 | 8 | ## Context 9 | 10 | Tech staff on GOV.UK can have access to dozens of things, like SSH access, GitHub organisation membership, and AWS accounts. When someone leaves access is revoked [using a check list][leaver] in a leavers card on the 2nd line Trello. 11 | 12 | ## Problem 13 | 14 | 1. The leavers process is slow and manual (and a bit boring). This means sometimes leavers aren't removed as quickly as we want. 15 | 2. The process of removing people from access lists is based around making _changes_. There is no system in place that verifies that everyone who currently has access is actually allowed to. This means that if someone processes a leaver ticket and forgets a step, a user account can linger around forever. 16 | 3. We'd like to use [GitHub teams for authentication to our Jenkins][jenk]. There are some concerns around that because it's very easy to add people to teams, and there is little visibility around the membership changes. 17 | 18 | [jenk]: https://github.com/alphagov/govuk-puppet/pull/5910 19 | 20 | ## Proposal 21 | 22 | We make a central list of GOV.UK users. We use this list to periodically verify that only the correct people have access. We do this by creating a Jenkins job that alerts Slack and Icinga if there are unexpected users with access to something. 23 | 24 | This will make the leavers process more deterministic. It also prevents people from being added to the GitHub team outside our normal process. 25 | 26 | There are a number of potential extensions to this, like adding an expiry date to the user accounts, which means we would get alerted when to remove a user (a concept that bob found in Gitlab). 27 | 28 | ## Implementation 29 | 30 | This is a proof of concept: 31 | 32 | 33 | 34 | It defines a list of users like this: 35 | 36 | ```yaml: 37 | - username: johndoe 38 | github_username: john-code 39 | ``` 40 | 41 | The script in the repo will check: 42 | 43 | 1. SSH access to (old) CI machines 44 | 1. SSH access to mirrors 45 | 1. SSH access to backup machines 46 | 1. SSH access to integration 47 | 1. Access to Jenkins 48 | 1. Access to AWS in integration, staging & production (via Terraform) 49 | 1. GitHub team membership (the GOV.UK team) 50 | 51 | These aren't all the places access lists are defined (see the [rest of the leaver ticket template][leaver]), but it's a start. Once our credential repos are moved to public GitHub, production access can also be checked. Possibly we could create an endpoint for Signon to allow us to verify the admins/superadmins. 52 | 53 | [leaver]: https://trello.com/c/PmVyofn8/3-template-leaver-dev-webops 54 | -------------------------------------------------------------------------------- /rfc-025-managing-special-snowflake-urls-through-the-publishing-api.md: -------------------------------------------------------------------------------- 1 | --- 2 | status: superseded 3 | implementation: superseded 4 | status_last_reviewed: 2024-03-06 5 | status_notes: Content Store content is kept in sync with Publishing API so there is no longer a choice. 6 | --- 7 | 8 | # Managing special snowflake URLs through the Publishing API 9 | 10 | ## Problem 11 | 12 | There are a [number of URLs on GOV.UK](https://docs.google.com/spreadsheets/d/1LUpym0SVeOkom-k6qnqUic1tyIRhnHgKrAsXwr1kX5w/edit#gid=0) which do not fall into the category of traditional leaf-node published content e.g. robots.txt, search, or the homepage. 13 | 14 | These URLs are frequently registered with the router directly.  Since they aren't entered into the URL arbiter this sometimes causes clashes and/or [downtime if important routes are overwritten](https://docs.google.com/document/d/1Ev_axmMdvsg3WTnYpdYVBciSAzM3q64QklB8MOZl7Go). 15 | 16 | ## Proposal 17 | 18 | The minimum required to add safety to this process is to have the routes pass through the URL arbiter.  We could have all applications which register these snowflake routes directly instead check with the arbiter first, but this is an untidy option and open to error.  We'd also like to reduce the number of applications which talk directly to the router API as much as possible.  This suggests we use the publishing API as the registration mechanism. 19 | 20 | So far there are two proposals on how to do this: 21 | 22 | 1. **Use the publishing API as an endpoint but don't write to the content store**.  This would require a new endpoint on the API which engages in URL arbitration but does not push to either live or draft content stores.  It would also require the API to speak to the router API directly, something which it currently delegates to the content store. 23 | 2. **Add content store entries**.  This would require no changes to the publishing API.  The owning applications would publish a "snowflake" format document to the publishing API on deploy (or whenever else they normally do this, e.g. via rake tasks) containing some human-readable information about the path and why it exists (e.g. "`/search` handles the display of search results across the site" or "`/info` is a prefix to provide statistics about other routes on GOV.UK") and allow the standard publishing pipeline to deal with URL arbitration and route registration.  This could later allow some of the routes to be not only registered via the publishing pipeline but also rendered out of the content store, for example in the case of `robots.txt`. 24 | 25 | My personal preference is for proposal #2.  Here's an example of what the document might look like for `robots.txt`: [https://gist.github.com/elliotcm/cddd6ea1f4e3989009bd](https://gist.github.com/elliotcm/cddd6ea1f4e3989009bd) 26 | 27 | -------------------------------------------------------------------------------- /rfc-043-content-items-without-a-base-path.md: -------------------------------------------------------------------------------- 1 | ## Problem 2 | 3 | There's a need for "unadressable content" on GOV.UK. These are content items without a URL (base path) that won't be visible on GOV.UK. They would be used in the links hash and for dependency resolution. 4 | 5 | The need for having these first surfaced when talking about modelling governments in the publishing-api. Having content-items of document type "government" in the publishing-api would allow us to use the links hash to specify to which government a piece of content belongs. With dependency resolution we can then put everything related to one government in history mode. 6 | 7 | Other use cases for this are: 8 | 9 | - **The tags for the new single taxonomy**. Linking to things without a base path means we don't need a page for each "tag", but we can have a more flexible architecture where we display multiple tags on one page. 10 | - **Adding external links to the search index**. These currently live in [recommended links repo](https://github.com/alphagov/recommended-links) and are put directly into rummager. In the future we'll populate the search index from the message queue exclusively. This means that everything that should be in search also needs to be in the publishing-api. 11 | - **Councils**  may be added as part of the work the Custom team is doing to [rebuild local transactions](https://gov-uk.atlassian.net/wiki/display/GOVUK/RFC+33+Local+transactions+migration+approach). 12 | 13 | There has been some work done on this: [https://trello.com/c/b77KFGgc/523-add-support-for-nil-base-paths-in-publishing-api](https://trello.com/c/b77KFGgc/523-add-support-for-nil-base-paths-in-publishing-api).  14 | 15 | This RFC is intended to give us the opportunity to really think this through, as this is a important architectural feature. 16 | 17 | ## Proposed requirements 18 | 19 | 1. Publishing API should support these content items without knowing about the formats. Instead, whether or not the format is addressable or not should be set in the content-schemas. 20 | 2. The content item needs testable with [govuk-content-schemas](https://github.com/alphagov/govuk-content-schemas) (the current build process assumes that all schemas will have a required `base_path`) 21 | 3. Publishing API validates that the items don't have a `base_path`, `rendering_app`, `redirects` and `routes` when writing 22 | 4. Publishing API doesn't include `base_path`, `rendering_app`, `redirects` and `routes` in the GET responses and message queue payload 23 | 24 | 25 | Open questions 26 | 27 | - Do we add a explicit attribute on the presented content item that it's addressable/non-addressable? 28 | - Can all message queue consumers cope with non-addressable content? 29 | - Are there any other places where we assume that content items have a base path? 30 | 31 |   32 | 33 |   34 | 35 |   36 | 37 | -------------------------------------------------------------------------------- /rfc-057-default-values-for-application-secrets.md: -------------------------------------------------------------------------------- 1 | ## Problem 2 | 3 | We discovered that the EFG application had 2 Devise configuration files in production. This is because it uses alphagov-deployment to copy configuration files into place during deployment. Each of these Devise configuration files had a `secret_key` value (one was in the public repo and used for development, the other was from alphagov-deployment). By luck, we were using the value of the `secret_key` which was not public (this was due to the order that Rails loaded the initializers). 4 | 5 | If we had been using the public value of the secret key this would have been a security incident for GOV.UK which may have resulted in all 3000 EFG users having their passwords reset. 6 | 7 | ## Proposal 8 | 9 | "Secrets" are defined as any values which would result in a security incident if disclosed. These can include, but are not limited to, cookie encryption seeds and password encryption seeds. 10 | 11 | ### Configuration 12 | 13 | This is a reinforcement of [RFC 26: 12-Factor Rails Apps](https://gov-uk.atlassian.net/wiki/display/GOVUK/RFC+26%3A+12-Factor+Rails+Apps). Applications must retrieve their secrets from the environment, not from files. 14 | 15 | ### Default values for secrets 16 | 17 |  Application code must not contain default values for secrets. For example: 18 | 19 | - 20 | ``` 21 | Bad: GovernmentApp::Application.config.secret_token = 'ohqu1iejohTh9oophieFah9UeDik0neixizeeVooSuush1xu' 22 | ``` 23 | - 24 | ``` 25 | Bad: GovernmentApp::Application.config.secret_token = ENV['SECRET_TOKEN'] || 'ohqu1iejohTh9oophieFah9UeDik0neixizeeVooSuush1xu' 26 | ``` 27 | - 28 | ``` 29 | Good: GovernmentApp::Application.config.secret_token = ENV['SECRET_TOKEN'] 30 | ``` 31 | 32 | ### Maintaining dev-prod parity for secret configuration 33 | 34 | Secrets must be configured in the environment for development as well as in production. An acceptable alternative to using environment variables in development is to use the [`config/secrets.yml` functionality available in Rails 4.1 and later](http://edgeguides.rubyonrails.org/4_1_release_notes.html#config-secrets-yml): 35 | 36 | ``` 37 | development: secret_key_base: development_example_secret_keyproduction: secret_key_base: ENV['SECRET_KEY_BASE'] 38 | ``` 39 | 40 | This maintains close enough configuration between development and production without needing environment variables in every environment. 41 | 42 | ### Default behaviour when secrets are not present 43 | 44 | If a piece of secret configuration being missing could result in a security incident, applications must fail to start if it is not present at boot or is set to an empty string. A good example of this is an empty cookie token, which in a bad framework could result in unencrypted cookies. For example: 45 | 46 | ``` 47 | if Rails.application.secrets[:secret_key_base].blank? raise 'Required setting for secret_key_base is not set'end 48 | ``` 49 | 50 | -------------------------------------------------------------------------------- /rfc-086-draft-stack-rummager.md: -------------------------------------------------------------------------------- 1 | # Draft Stack Rummager 2 | 3 | ## Summary 4 | 5 | RFC 86 proposes setting up a instance of [Rummager][rummager] for the 6 | [draft stack][draft-stack-docs]. 7 | 8 | ## Problem 9 | 10 | The [draft stack (content preview)][draft-stack-docs] part of GOV.UK 11 | relies in places on the [rummager service (Search API)][rummager]. 12 | 13 | There is only one instance of this service, which is primarily used 14 | for the live stack, but is also currently used from some services on 15 | the draft stack (see [govuk-puppet][puppet-frontend-rummager-config]). 16 | 17 | The current Rummager instance may expose some associations between 18 | live content and draft content (e.g. draft mainstream browse pages or 19 | taxons), depending on the internal index which is used for the live 20 | content. However, the current trend is that associations with draft 21 | content will be supported for less live content, as more types of 22 | content are migrated to the `govuk` index (see [ADR 23 | 4][rummager-adr-4]), as the process of populating this index does not 24 | involve fetching draft content from the Publishing API. 25 | 26 | Having some way of including draft content in the responses from 27 | Rummager on the draft stack would make the draft stack more 28 | useful. Previewing pages that use Rummager for some of the data won't 29 | give an accurate representation of what the page would look like if 30 | the relevant content is published. 31 | 32 | ## Proposal 33 | 34 | The proposal made here to improve the user experience on the draft 35 | stack, is to setup a draft instance of rummager, populated with the 36 | appropriate content via the Publishing API, and then switch to using 37 | this from draft services. 38 | 39 | ## Action Plan 40 | 41 | 1. Deploy rummager to the draft stack 42 | - Involves changing `govuk-puppet` 43 | - Possibly requires some new machines 44 | 2. Send draft content to a new `draft_documents` RabbitMQ Exchange 45 | - From the Publishing API, for at least the PutContent, and 46 | Unpublish commands 47 | - Don't send any access limited content, to avoid Rummager 48 | indexing it 49 | 3. Subscribe the draft instance of Rummager to the `draft_documents` 50 | exchange, as well as the existing `published_documents` exchange 51 | 52 | If the above work is done, at this point, there will be a draft 53 | Rummager service. To switch the draft frontend apps to use this 54 | instead, the next step is to make an assessment regarding whether it 55 | will work better than the live instance. This will probably depend on 56 | how much content has been migrated to the `govuk` index. 57 | 58 | [rummager]: https://github.com/alphagov/rummager 59 | [draft-stack-docs]: https://docs.publishing.service.gov.uk/manual/content-preview.html 60 | [puppet-frontend-rummager-config]: https://github.com/alphagov/govuk-puppet/blob/3a874ba1afec98c0aeb7f34c9fe34128340e7363/modules/govuk/manifests/node/s_draft_frontend.pp#L24 61 | [rummager-adr-4]: https://github.com/alphagov/rummager/blob/master/doc/arch/adr-004-transition-mainstream-to-publishing-api-index.md 62 | -------------------------------------------------------------------------------- /rfc-058-publishing-api-events.md: -------------------------------------------------------------------------------- 1 | ## Problem 2 | 3 | An event is created for every request that comes into the publishing API. 4 | These events are used as a form of Event Sourcing to track the application state 5 | as a series of events over time, and can be used to reconstruct past states. 6 | 7 | One of the problems is the sheer number of events that go through the 8 | publishing API, roughly growing at 600MB a month, currently sitting at 6GB, 9 | 4.2 million events, which has implications for replication, and for developers 10 | copying that data to their local environments. 11 | 12 | The other problem being that the event information, as it stands, starts losing 13 | its effectiveness over time, with code changes, schema changes and API version 14 | changes, the ability of being able to replay the events from scratch gets lost. 15 | 16 | ## Proposal 17 | 18 | The event information is useful for replaying events and debugging but we would 19 | 20 | like to archive the events in the events table, but with still having ability to: 21 | 22 | 1. Easily search through events for debugging 23 | 2. The ability to retrieve events, to replay in the case of failure. 24 | 3. Store future events 25 | 26 | ### Option A: 27 | Start storing events in Elasticsearch. 28 | 29 | This would imply that we no longer store events in the Publishing API, but rather 30 | every time an event occurs, we asynchronously store it in Elasticsearch. We would 31 | need to decouple events from any command, and store the event in the same 32 | transaction. Which has further implications of sending versioned data downstream 33 | to the content store, which currently is managed by the event id. 34 | 35 | pros: 36 | 37 | - Ease of storing and retrieving events and rich query language makes this useful 38 | for debugging 39 | - No development time needed to build an interface (Kibana) 40 | - Already part of our infrastructure 41 | - Easily retrieve events to replay 42 | 43 | cons: 44 | 45 | - Reliability, if someone accidentally deletes the index 46 | - Effectively becomes the primary data source of events 47 | - Might also need archiving as a backup 48 | 49 | ### Option B: 50 | Log payload params directly to Logstash 51 | 52 | This would mean simply logging the request params in the log, and loses any 53 | concept of an event 54 | 55 | pros: 56 | 57 | - Very little work initially 58 | - No other Elasticsearch clusters 59 | 60 | cons: 61 | 62 | - Difficult to backdate events 63 | - More difficult to replay events 64 | - Not as easy to query since it will be interspersed with other data 65 | 66 | 67 | ### Option C: 68 | 69 | Another Postgres DB in the Publishing API 70 | 71 | By establishing a different connection in the Events model 72 | [http://api.rubyonrails.org/classes/ActiveRecord/Base.html](http://api.rubyonrails.org/classes/ActiveRecord/Base.html) 73 | 74 | pros: 75 | 76 | - Same infrastructure as current Publishing API 77 | - Little to change 78 | 79 | cons: 80 | 81 | - Doesn't solve the problem of data replication 82 | - Increases complexity 83 | 84 | 85 | ### Other Options: 86 | 87 | Archive to S3 and carry on as normal 88 | -------------------------------------------------------------------------------- /rfc-091-sharing-assets.md: -------------------------------------------------------------------------------- 1 | --- 2 | status: unclear 3 | implementation: abandoned 4 | status_last_reviewed: 2024-03-04 5 | status_notes: We're currently thinking about consolidating apps as a better alternative to changing the way we share assets. 6 | --- 7 | 8 | # Sharing assets 9 | 10 | ## Summary 11 | 12 | We need to share assets to make sure users only have to download a thing once. To do this, we'll upload each application's assets to the same location. 13 | 14 | ## Problem 15 | 16 | In [RFC84 we proposed replacing Static with a gem](https://github.com/alphagov/govuk-rfcs/pull/84). One of the architectural challenges with this approach is that by moving assets (like CSS, JS and images) into the gem, the assets won't be shared for users. This means that users visiting GOV.UK would have to download assets multiple times, depending on what app renders the page. 17 | 18 | ## Proposal 19 | 20 | We upload the assets to a central location. For example, if both `collections` and `government-frontend` have a file named `foo.css`, they will upload the file to `assets.publishing.service.gov.uk/shared/foo.css`. 21 | 22 | Because [Rails fingerprints assets](http://guides.rubyonrails.org/asset_pipeline.html#what-is-fingerprinting-and-why-should-i-care-questionmark), the asset actually uploaded will be `assets.publishing.service.gov.uk/shared/foo-8d811b8c3badbc0b0e2f6e25d3660a96cc0cca7993e6f32e98785f205fc40907.css`, unless the files are different, in which case they'll have different fingerprints and won't overwrite. 23 | 24 | As part of this solution we'd need a way to ensure or encourage different frontend apps to use assets in a way that is shareable. For example, applications could include a separate stylesheet with all the component styles (`components.css`), which would be shared across apps. 25 | 26 | How to achieve this: 27 | 28 | * In govuk-app-deployment, we configure capistrano to precompile the assets and then upload the resulting files to a central location. 29 | * Proxy assets.publishing.service.gov.uk/shared to this location 30 | * Point applications to use this URL as asset host so that it's used in production 31 | 32 | Upsides 33 | 34 | * We remove Nginx config to proxy to individual applications, perhaps removing some hops 35 | * Whenever there's an asset to share, it will be shared automatically. If an application copies an image from another app, on production they'll have the same URL. 36 | * Each application ensures that its own assets are present on the shared bucket. 37 | * Applications don't serve files anymore. 38 | 39 | Downsides 40 | 41 | * We'll end up with a directory full of assets which are of unknown origin. 42 | 43 | ## Alternatives 44 | 45 | ### 1. Do nothing 46 | 47 | The simplest solution is to not share assets between applications. This would have significant impact on the user, especially if the font will be distributed in the gem. 48 | 49 | ### 2. Deploy the gem 50 | 51 | This would work roughly that every time we publish a new gem version, we'll deploy the resulting assets to a central place. In production apps point to that shared asset URL instead of 52 | 53 | * Disparity between development and production 54 | * Inevitable deploy dependency - the gem deploy needs to happen before the application is deployed 55 | -------------------------------------------------------------------------------- /rfc-050-do-end-to-end-testing-of-gov-uk-applications.md: -------------------------------------------------------------------------------- 1 |   2 | 3 |   4 | 5 | --- 6 | status: "OPEN" 7 | notes: "Open for comments and questions" 8 | --- 9 | 10 | ## Background 11 | 12 | We have almost finished migrating Specialist Publisher onto the Publishing Platform. 13 | 14 | In doing so, we've written lots of tests to ensure the application meets its requirements. 15 | 16 | The majority of our tests follow this pattern: 17 | 18 | 1. Stub external services to return a canned response 19 | 2. Perform some action in the publishing application 20 | 3. Assert that the correct requests were made to external services 21 | 22 | ## Problems 23 | 24 | **Unable to test user-journeys** 25 | 26 | Because we're returning canned responses, we're unable to test user-journeys within the publishing app. We **can't** write tests like this: 27 | 28 | 1. Visit the new document page 29 | 2. Create a new document 30 | 3. Assert that the document is visible in the interface 31 | 32 | We can't write these tests because no state is preserved after we have created the document. The next time a request is made to get that document, it will return the original response that the stub set up and won't reflect that change that would have happened in the Publishing API if we were interacting with the application for real. 33 | 34 | The same is true for all user-journeys in the application. There are no tests that span more than a single page of the publishing app.. 35 | 36 | **Unable to test side-effects** 37 | 38 | Similarly, we are unable to test that side-effects actually happen. We have no way to test that an email has actually been sent. Side-effects include: 39 | 40 | 1. Checking that draft content is visible 41 | 2. Checking that published content is visible 42 | 3. Checking that an email has been sent 43 | 4. Checking that documents appear in the search index 44 | 45 | The closest we can get is to assert that we have sent requests to the appropriate services, but this doesn't actually check that these things have happened. 46 | 47 | **Interface brittleness** 48 | 49 | When interfaces change on those external services, all of our tests will continue to pass, but the application will be broken. 50 | 51 | We may be able to do lots of manual testing to plug the holes that we have with end-to-end testing, but that doesn't account for changes to interfaces that might happen in the future. 52 | 53 | ## Proposal 54 | 55 | I propose that we consider how to do end-to-end testing of applications on GOV.UK. 56 | 57 | I think that we should have a high-level test-suite that can orchestrate more than one application and make assertions at external boundaries of the system, such as user-interfaces or email inboxes. 58 | 59 | Ideally, it would treat the system as a black-box and only care about the user-facing behaviour of the system, giving us the freedom to change the internals of the system independently of this end-to-end testing process. 60 | 61 | When developing applications, writing these high-level tests should be an inherent part of that process, rather than an after-thought. 62 | 63 | ## Solution 64 | 65 | I don't want to prescribe what the technical solution should be. 66 | 67 | If this RFC is accepted, I think we should have a separate forum to figure out what to do. 68 | 69 |   70 | 71 |   72 | 73 | -------------------------------------------------------------------------------- /rfc-165-add-api-endpoints-to-transition.md: -------------------------------------------------------------------------------- 1 | --- 2 | status: to_be_deleted 3 | implementation: not_planned 4 | status_last_reviewed: 2024-03-04 5 | status_notes: > 6 | People aren't clear whether the value of doing this outweighs the cost of implementing it. 7 | Given that disagreement, it seems unlikely that any team is going to pick this up. 8 | --- 9 | 10 | # Add API endpoints to Transition for consumption by Bouncer 11 | 12 | ## Summary 13 | 14 | This proposes that the Bouncer application should access transition data via 15 | API endpoints on Transition instead of via direct database access. 16 | 17 | ## Problem 18 | 19 | The [Transition 20 | System](https://docs.publishing.service.gov.uk/manual/transition-architecture.html) 21 | is built to transition government websites to GOV.UK. It includes: 22 | - Transition - Ruby on Rails application used by admins to create mappings from 23 | old URLs to pages on GOV.UK. 24 | - Bouncer - Rack-based application that uses the mappings created by Transition 25 | and handles requests to those old domains. 26 | 27 | Currently, Bouncer retrieves the mappings written by Transition through direct 28 | read-only access to the database shared by the two applications. 29 | 30 | This is problematic as any code related to database access must be kept in sync 31 | between Transition and Bouncer. 32 | This includes the use of ActiveRecord; we must take the effort to keep 33 | ActiveRecord subclasses and dependency versions consistent. 34 | 35 | As this approach is unusual in the wider context of GOV.UK, this also has the 36 | potential to make working with this system more difficult to grasp and process 37 | for software developers. The additional cognitive load that this places on 38 | individuals and teams may hurt productivity and make tasks such as onboarding 39 | new developers more difficult. 40 | 41 | ## Proposal 42 | 43 | Bouncer should retrieve data from Transition via an API. This involves: 44 | - Adding API endpoints that allow us to retrieve the minimum amount of data for 45 | Bouncer to function. 46 | - Adding adapters to GDS API Adapters. 47 | - Removing database dependencies from Bouncer. 48 | 49 | There has been a push recently to fully separate our frontend and backend apps. 50 | This approach would fit in well with this theme, essentially turning Transition 51 | into a publishing app and Bouncer into a platform concern. 52 | 53 | This approach is an alternative to merging to the Bouncer codebase into 54 | Transition entirely. Merging the applications is a more complex task that 55 | involves: 56 | - Rewriting large parts of Bouncer as it's currently not a Rails app. 57 | - Handling of routes that are the same in Bouncer and Transition. 58 | - Dealing with infrastructure complexity - references to Bouncer are littered 59 | through the infrastructure. 60 | 61 | The documentation in the Transition and Bouncer repositories must be updated to 62 | reflect these changes. 63 | 64 | ## Consequences 65 | 66 | - Bouncer's dependency on the database shared with Transition is removed. 67 | - Bouncer's codebase will be less complex. 68 | - This approach will likely be less performant than the current approach. We 69 | plan to take this into consideration during development by identifying and 70 | measuring relevant performance metrics. 71 | - Bouncer depends on the availability of Transition's API. 72 | -------------------------------------------------------------------------------- /rfc-159-switch-off-whitehall-apis.md: -------------------------------------------------------------------------------- 1 | --- 2 | status: accepted 3 | implementation: done 4 | status_last_reviewed: 2024-03-04 5 | --- 6 | 7 | # Switch off Whitehall's public APIs 8 | 9 | ## Summary 10 | 11 | Whitehall has a number of public APIs, as listed below: 12 | 13 | - `/api/governments` 14 | - `/api/governments/{slug}` 15 | - `/api/world-locations` 16 | - `/api/world-locations/{slug}` 17 | - `/api/world-locations/{slug}/organisations` 18 | - `/api/worldwide-organisations/{slug}` 19 | 20 | These APIs are not advertised publicly, except in [some documentation in Whitehall’s repository](https://docs.publishing.service.gov.uk/repos/whitehall/api.html) and in an [‘alpha’ API catalogue](https://www.api.gov.uk/gds/gov-uk-governments/#gov-uk-governments). Only one of these endpoints (`/api/world-locations`) is used internally by GOV.UK applications (via. the `gds-api-adapters` gem). 21 | 22 | Usage is low, with the vast majority of traffic from robots. Over a 14 day period in April/May 2023, the number of hits to origin machines was as follows (after excluding those from our team whilst investigating these APIs): 23 | 24 | - `/api/governments`: 114 hits on index page 25 | - `/api/government/{slug}`: 0 hits 26 | - `/api/world-locations`: 290 hits (excluding requests from other GOV.UK apps) 27 | - `/api/world-locations/{slug}/organisations`: 0 hits 28 | - `/api/worldwide-organisations/{slug}`: 34 hits 29 | 30 | ## Problem 31 | 32 | Publishing Platform are aiming to switch off Whitehall Frontend (and remove all of the associated rendering code from the application) before the end of Q1 2023. The rendering of these APIs would need to be migrated into other applications, which would involve development cost. The traffic is very low, therefore the development cost would outweigh the benefit of migrating these to another application. 33 | 34 | However users are already able to retrieve the same data through alternative means, except for one of these endpoints: 35 | 36 | - `/api/governments` → not available (we could probably add this into a content item, if there’s a user need to maintain this without having a specific API for it) 37 | - `/api/government/{slug}` → `/api/content/government/{slug}` 38 | - `/api/world-locations` → `/api/content/world` (available soon, once we’ve migrated the page out of Whitehall) 39 | - `/api/world-locations/{slug}` → `/api/content/world/{slug}` 40 | - `/api/world-locations/{slug}/organisations` → `/api/search.json?filter_format=worldwide_organisation&fields=title,format,updated_at,link,slug,world_locations` (although this doesn’t filter by World Location, so the user will need to filter this themselves once they’ve got the results) 41 | 42 | ## Proposal 43 | 44 | Update gds-api-adapters to not need the `/api/world-locations` endpoint of the Whitehall API and return the world locations (and their organisations) by using the relevant alternatives listed above instead. This will end our internal reliance on these APIs. 45 | 46 | Switch off the APIs without migrating to another application in a two-step process: 47 | 48 | 1. Replace the existing APIs with a ‘gone’ response and provide content that states how users are able to access the same information (as detailed in the ‘Problem’ section above). This will be in place for one month. 49 | 2. After one month, switch off the APIs and return a ‘gone’ response to any requests with no content. 50 | -------------------------------------------------------------------------------- /rfc-023-putting-detailed-guides-paths-under-guidance.md: -------------------------------------------------------------------------------- 1 | --- 2 | status: accepted 3 | implementation: done 4 | status_last_reviewed: 2024-03-06 5 | --- 6 | 7 | # Putting Detailed Guide paths under /guidance 8 | 9 | ## Problem 10 | 11 | Currently detailed guides live at urls like: [https://www.gov.uk/british-forces-overseas-posting-cyprus](https://www.gov.uk/british-forces-overseas-posting-cyprus) 12 | This can cause problems because editors are able to create documents and claim high-profile slugs which shouldn't really be used for guidance 13 | Instead they should be published as https://www.gov.uk/guidance/[thing] 14 | 15 | Note that currently manuals are also published under `/guidance` so we will need to check using url arbiter or equivalent to ensure the path can be claimed. 16 | 17 | ## Proposal 18 | 19 | ### Ensuring new detailed guides are served under guidance/ 20 | 21 | in Whitehall, adding 'guidance/' in front of the detailed\_guides#show route in routes.rb. However keeping the old route live to cover deploy time (to be removed 30mn+ after deploy). It will be the same as the current route, without 'as:detailed\_guides'. 22 | 23 | in Whitehall, update the [presenter](https://github.com/alphagov/whitehall/blob/master/app/models/registerable_edition.rb#L26-L32)for sending the paths to panopticon to reflect the changes in the paths 24 | 25 | In Panopticon, we might also need to [update the slug validation code](https://github.com/alphagov/govuk_content_models/blob/master/app/validators/slug_validator.rb) as it may not accept detailed\_guide artefacts with a `/` in the slug 26 | 27 | NB: in the routes, /specialist/detailed-guide-slug redirects to root/detailed-guide-slug, so as another story we should make it redirect to /guidance/detailed-guide-slug directly. 28 | NB: also, in govuk\_content\_models, we will need to disallow URLs at the root once the migration of old guides is complete 29 | 30 | ### Ensuring the existing detailed guides are served under guidance/ 31 | 32 | Panopticon migration to reslug all detailed guides and avoid creating duplicates 33 | 34 | We need to create a rake task to republish to publishing-api. There is already a [PublishingApiRepublisher](https://github.com/alphagov/whitehall/blob/master/lib/data_hygiene/publishing_api_republisher.rb) class in lib/data\_hygiene that takes an edition scope (eg DetailedGuide.published) and republishes them - we would need to call that from the rake task. 35 | 36 | In Whitehall, run the [rummager::reset::detailed](https://github.com/alphagov/whitehall/blob/master/lib/tasks/rummager.rake#L44) rake task to reindex the detailed guides in search. 37 | 38 | In collections-publisher, run a data migration to update all api-urls for when detailed guidance is curated into topics 39 | 40 | ### Redirecting old paths of existing detailed guides 41 | 42 | Several options: 43 | 44 | - extract the URLs to a CSV and add them into router-data 45 | - in Whitehall, add a "was\_previously\_under\_root" boolean attribute to the DetailedGuide model; for each detailed guide which has this attribute as true, push a redirect to the Publishing API 46 | - add redirects as a model so that a detailed guide has a redirects association, and the redirects are pushed to the Publishing API when a detailed guide is published. 47 | 48 | ## To do 49 | 50 | Arrange the above steps in such a way that nothing breaks. 51 | -------------------------------------------------------------------------------- /rfc-022-putting-elasticsearch-backups-in-s3.md: -------------------------------------------------------------------------------- 1 | --- 2 | status: accepted 3 | implementation: done 4 | status_last_reviewed: 2024-03-06 5 | --- 6 | 7 | # Putting Elasticsearch backups into S3 8 | 9 | ## Problem 10 | 11 | Currently, we don't have good (any?) backups for our elasticsearch indexes.  This is particularly critical for the index which powers the main site search, since it takes at least 2 hours to rebuild the indexes for the "/government" content from scratch, and requires lots of separate apps to be prodded to rebuild the indexes for the mainstream index. 12 | 13 | Our current mechanism for syncing the search indexes to other environments, and to developer machines, is: 14 | 15 | - slow (can take 90 minutes to perform the sync, because it rebuilds the indexes from JSON, and we now perform lots of complicated analysis in elasticsearch) 16 | - frequently fails (lots of dependencies and moving parts) 17 | - hard to maintain 18 | - requires ssh access to preview machines to get developer builds 19 | - developers can't get an up-to-date copy of the search index - latency of at least a day (more when nightly jobs have failed) 20 | 21 | ## Proposal 22 | 23 | Since elasticsearch 1.0, elasticsearch has supported ["Snapshot and Restore"](https://www.elastic.co/guide/en/elasticsearch/reference/1.4/modules-snapshots.html), which allows a snapshot of a search index to be copied to either a shared filesystem, or to amazon S3.  I propose that we use S3, for its ease of setup, and because it will allow us to share the resulting backups easily. 24 | 25 |   26 | 27 | In detail: 28 | 29 | - snapshots are configured per-index. each index should have a snapshot set up to copy it from production to S3. 30 | - the snapshot should be updated frequently (eg, every 5 minutes) 31 | - the snapshots should be made readable to everyone (ie, no access control needed to access the snapshots). This is much more convenient than having to give all developers access to a special set of S3 keys. All the indexes which currently exist contain only publically available content, so there is no confidentiality issue with this right now. If draft ("preview") content is put into the search system, it should be put into a separate elasticsearch index, which will **not** be made publically readable - a separate S3 bucket would be used for this, and this might require accreditation approval since it would not be public data. 32 | - the sync process from production to staging and preview environments will be replaced with a restore process.  This can be made "atomic" (I'm not sure if the restore process is atomic by default, but if not we can make it restore to a new index, and use aliases to switch it in place once the restore complete, much as we currently do). 33 | - we should also ensure that the scripts for restoring to staging and preview work for restoring to production, for disaster recovery. 34 | - sync to developer machines should also be replaced with using the restore process 35 | 36 | As well as fixing existing problems with syncing, this proposal has the following benefits: 37 | 38 | - We would have working backups of the search index, and since we'd be using them for data sync, we'd notice quickly if the backups stopped being viable. 39 | - Being open about this should enable other uses to be made of the data (for example, I know academics who would love to have a local copy of the GOV.UK search index so that they can perform experiments on it). 40 | - Future upgrades of elasticsearch should become much easier and safer, since we could easily revert an upgrade (and go back to an earlier index snapshot), and could also perform an upgrade by building a new cluster and switching over easily, if desired. 41 | 42 |   43 | 44 |   45 | 46 | -------------------------------------------------------------------------------- /rfc-042-testing-backend-applications-against-production-traffic.md: -------------------------------------------------------------------------------- 1 |   2 | 3 |   4 | 5 | --- 6 | status: "OPEN" 7 | notes: "Open for review" 8 | --- 9 | 10 | ## Problem 11 | 12 | We replay frontend traffic from production to staging to expose problems with new code during deploy. We do not replay backend traffic of any kind, and we only replay frontend traffic for GETs. 13 | 14 | We would like a way to expose new code to backend traffic from production. 15 | 16 | Traffic which is not currently replayed falls into 2 groups: 17 | 18 | 1. Backend GET requests 19 | 2. Frontend and backend non-GET requests 20 | 21 | Category 1 is easy to fix as the GET requests do not change state or cause any side-effects. Category 2 is more difficult. 22 | 23 | Staging's data is not kept consistent with production because: 24 | 25 | 1. Data is sync'd nightly from production to staging 26 | 2. Developers often make small modifications in staging when testing a deploy. 27 | 28 | Replaying requests if the data doesn't match could cause errors or other behaviour which would not happen in production. 29 | 30 | Floods of errors could also make life tougher during deploys, masking more important problems with staging. 31 | 32 | While we could just turn on the traffic replay and see what happens, but we'd rather have something more structured. 33 | 34 | ## Proposal 35 | 36 | Github have a gem called [Scientist](https://github.com/github/scientist) for A/B testing critical code paths. This is in contrast to other libraries which aim to increase conversion. 37 | 38 | The gem checks the result of the old code against the new, finding inconsistencies and performance problems. 39 | 40 | The catch is that the code has to be free of side effects like writing to the database.  Running code with side effects twice would lead to unexpected behaviour and bad data. 41 | 42 | If we wanted to run these two paths of side-effect–having code we would need separate environments for each path.  Our small- and micro-services can help us here, as we can run separate instances of applications which are changing and use the GDS API adapters as our point of test. 43 | 44 | Another problem is that we don't want performance regressions to slow down our A-branch.  Currently Scientist runs both paths in the same thread and checks both results at the end.  We'd want to offload the B-branch to a worker and have the A-branch upload its results to a shared database (like Redis) so the worker can do the science outside the main request/response cycle. 45 | 46 | This also provides time to do the more sophisticated checks that might be required to make sure a change hasn't caused problems.  Have the right number of emails been sent (via a sandbox GovDelivery account)?  What does the database look like?  etc. 47 | 48 | #### Required work 49 | 50 | - We would need to make spinning up applications in standalone environments easier and more automated.  We discussed changing Puppet to easily spin up a "B-branch" version of an app and its dependencies, but that seems to be as much or more work than being able to spin up in AWS or via Docker, so we'll likely take the latter approach. 51 | - We would need to write some helper code to wrap Scientist in a way that enables the testing of side-effect code.  This would likely take the form of a standard worker which can be expanded upon and some entry point code for the API adapters. 52 | 53 | The strong benefit of this whole approach is to safely test alternate branches from the production environment.  Since we'll be able to target our testing more carefully for both side-effect and functional code it might also remove the need for a fully-duplicated staging environment. 54 | 55 |   56 | 57 |   58 | 59 | -------------------------------------------------------------------------------- /rfc-036-stop-preserving-order-for-links.md: -------------------------------------------------------------------------------- 1 | ## Problem 2 | 3 | A links hash in a content item looks like this: 4 | 5 | `"links": {` 6 | `  "lead_organisation": ['ORG-CONTENT-ID'],` 7 | `  "organisations": ['ORG-CONTENT-ID', 'ANOTHER-ORG-CONTENT-ID'],` 8 | `  "topics": ['TOPIC-CONTENT-ID'],` 9 | `  "available_translations": [... automatically generated ...]` 10 | `}` 11 | 12 | The array of links is currently guaranteed to preserve its order when sent to the publishing-api. From the content-store documentation: 13 | 14 | > The list\_of\_links is an array of content items, order is preserved. 15 | > 16 | > [https://github.com/alphagov/content-store/blob/4b4a82a279de11a2af27b05dfc61d5f18a250c75/doc/content\_item\_fields.md#links](https://github.com/alphagov/content-store/blob/4b4a82a279de11a2af27b05dfc61d5f18a250c75/doc/content_item_fields.md#links) 17 | 18 | The order preserving complicates the following things: 19 | 20 | 1. **Complicates tagging tools**. We are building a generic tagging tool that defines the relationships between content items. It shouldn't be concerned with how these relationships are presented on the site. 21 | 2. **Complicates implementation.**  The publishing-api currently saves the links array and forwards it without manipulating it. To build a more flexible system, the links are being extracted into it's own table ([https://trello.com/c/zppxFP6p](https://trello.com/c/zppxFP6p)). We'll lose the "free" preservation with that change and will have to add code specifically to preserve the ordering. 22 | 3. In most cases, the ordering of the links should be a **presentation concern** anyway. For example, the [collections-publisher app sorts the related topics by title](https://github.com/alphagov/collections-publisher/blob/37830fd561b9cd8c212a9c63b126ed93bb655dc1/app/presenters/mainstream_browse_page_presenter.rb#L15) before sending the links to the publishing-api, which effectively reserves the `related_topics` for this use. Contrived example: if we were to use the related\_topics on a prototype sorted chronologically, it would need to "override" the ordering specified in collections-publisher. It would be confusing that sometimes the ordering is defined on the frontend, and sometimes by the publisher. 23 | 4. It's **easily abused to add meaning**. We use the first item in the `sections` tag in govuk\_content\_models for the breadcrumb ([code](https://github.com/alphagov/govuk_content_models/blob/master/app/traits/taggable.rb#L29-L48)). This means we can't easily query "pages that have x as their breadcrumb". 24 | 5. It may make  **bulk tagging**  more difficult. (we don't have a specific plan for that, but I can imagine a case where a bulk-action of "remove mainstream browse page tag x from these content items" would change the breadcrumb for some items, but not others) 25 | 26 | 27 | ## Proposal 28 | 29 | - We stop guaranteeing the order of the links. 30 | - During the tagging migration we get rid of the usage of the first item as breadcrumb.   31 | 32 | ## Impact 33 | 34 | - Audit pages using links to make sure nothing is using it. 35 | 36 | ## Impact on breadrumbs 37 | 38 | Our proposal is to add a `parent` tag-type and populate it [during the tagging migration](https://github.com/alphagov/panopticon/blob/8d0c3bf8fe013ad06a61a6adb4f773ee6b3e60f5/lib/tagging_migrator.rb#L31) with the primary mainstream browse page. We also make sure this tag is merged back into the section tags ([in the TagUpdater](https://github.com/alphagov/panopticon/blob/893857e2eb7c1f21e7382f761dde806fdd2cd8b0/app/queue_consumers/tagging_updater.rb#L46)) to keep current breadcrumbs intact. 39 | 40 | This `parent` type would then be usable by all apps to populate a breadcrumb. Content-store would recursively resolve all the parents to return this in the item: 41 | 42 |   43 | 44 |   45 | 46 |   47 | 48 |   49 | 50 | -------------------------------------------------------------------------------- /rfc-044-unpublishing-content-items.md: -------------------------------------------------------------------------------- 1 |   2 | 3 |   4 | 5 | --- 6 | status: "IN DEV" 7 | notes: "Development has started (2nd May, 2016)" 8 | --- 9 | 10 | ## Problem 11 | 12 | The workflow around unpublishing and withdrawing content items in the publishing-api is unclear and does not cover all the use cases of the different publishing apps. We need to simplify the processes, while making sure that all current actions are supported. 13 | 14 | There are various different types of withdrawal action. They can be grouped as follows: 15 | 16 | 1. **Withdrawing with a message** : This is used for content that was previously valid but is no longer current. The content continues to display, with the addition of a large message block at the top stating that the content is withdrawn, and it is no longer findable in search. (Whitehall) 17 | 2. **Unpublishing with a message** : The content is removed from the site, and an explanatory message is displayed at that URL instead. The message can optionally contain an alternative URL which is displayed as a link, but does not redirect automatically. (Whitehall) 18 | 3. **Unpublishing with a redirect** : The page is replaced by an automatic redirect to a different URL. (Whitehall, Publisher) 19 | 4. **Unpublishing with a Gone** : The page is replaced by a 410 Gone page. (Publisher, Specialist Publisher) 20 | 5. In addition, Whitehall sends a Gone for items that were scheduled to be published but then unscheduled, as long as there is no previously published version, and when non-edition items such as organisations or persons are deleted. 21 | 22 |   23 | 24 | Proposal 25 | 26 | Currently it is up to the publishing apps to send the relevant formats to publishing-api in each of these cases above. This involves sending a new draft item - of the original format updated to include the message for case 1, an Unpublishing item for case 2, a Redirect for case 3 and a Gone for cases 4 and 5 - and then a second call to publish. 27 | 28 | We recommend the following steps: 29 | 30 | 1. **Clarify our language**.  We adopted "withdrawing" as a verb in the publishing API when really that should be constrained to occasions when the content remains on the site but is removed from the canon of the current government.  Everything else is "unpublishing". 31 | 2. **Flag documents as withdrawn**  from the publishing app rather than supporting "withdrawn" as a first-class state, so the front-ends can render the appropriate banner.  This should be handled the same way when we do "historic" documents. 32 | 3. **Refuse to clobber existing documents**.  We currently allow gones/redirects to take the place (base\_path) of documents, and vice versa.  This provides an in-app way to implement unpublishing (and republishing).  We should remove the first case of this functionality and require the existing content to be unpublished via the new endpoints `/v2/content/:id/unpublish` or moved by altering its base\_path and publishing (which automatically creates a redirect).  Allowing gones/redirects to be clobbered still makes sense as it is a reclamation of an unused path. 33 | 4. **Create `/v2/content/:id/unpublish`**  accepting a POST with \, which returns the content item to draft and creates & publishes the resulting format (gone/unpublishing/redirect).  Question: should we autodiscard an existing draft if there is one, or refuse to take action? 34 | 5. **Populate gone items** with enough information to render the page seen after unpublishing, which in the case of Whitehall is the reason for unpublishing and/or alternate URL.  This page presentation can then be tidied up and standardised across publishers / frontends. 35 | 6. **Indicate unpublished state** back to the user.  We need to be able to indicate that a piece of content was More thought warranted on this. 36 | 37 |   38 | 39 |   40 | 41 |   42 | 43 | -------------------------------------------------------------------------------- /rfc-106-docker-for-local-development.md: -------------------------------------------------------------------------------- 1 | --- 2 | status: accepted 3 | implementation: done 4 | status_last_reviewed: 2024-03-04 5 | --- 6 | 7 | # RFC 106: Use Docker for local development 8 | 9 | ## Summary 10 | 11 | Adopt a Docker-based approach for local development instead of the Vagrant VM. 12 | 13 | ## Problem 14 | 15 | We currently rely on the Vagrant development VM for the day to day dev cycle. 16 | 17 | - It has [many documented issues](https://docs.publishing.service.gov.uk/manual.html#development-vm) 18 | - Changes to puppet often result in a broken VM, because the infrastructure team don't use it. 19 | - We are adopting more cloud-based services into our stack, like S3 and Amazon's Elasticsearch. This means that our development environment doesn't look like production anymore, but we still need code in govuk-puppet. 20 | - It's hard to keep updated. Running the puppet command will often error and take a long time. 21 | - GOV.UK is moving to a containerised infrastructure (likely the [GDS Supported Platform](https://github.com/alphagov/gsp) (GSP)) 22 | 23 | ## Possible approaches to local development 24 | 25 | - An approach based on **Docker and docker-compose**. Ben Thorner has a working prototype called [govuk-docker](https://github.com/benthorner/govuk-docker) and we have the [end-to-end tests](https://github.com/alphagov/publishing-e2e-tests) running in the same. Other GDS programmes like GOV.UK Pay use this approach. 26 | - An approach based on **GNU Guix** package manager, which Chris Baines uses via govuk-guix. 27 | - Because they are read-only, frontend applications on GOV.UK can be **run against production** using the startup.sh --live flag. 28 | - Developers on the GOV.UK PaaS work by developing against a **remote set of services** - we’d run many versions of GOV.UK in the cloud, which developers can use in development. 29 | - Applications can be run locally by **installing dependencies manually**. This is what some of GOV.UK’s Linux users do. 30 | - The RE team has been working on local development tooling called **[gsp-local](https://github.com/alphagov/gsp/blob/master/docs/gds-supported-platform/getting-started-gsp-local.md)**. 31 | 32 | Approach | Pro | Cons 33 | -- | -- | -- 34 | Vagrant VM | Can look like production | Resource intensive, brittle 35 | Docker | Aligns with future hosting. Reproducible (with caveats). | Slow because on a Mac it needs a VM. Development won’t match production/CI exactly until we migrate onto a container-based platform. 36 | Guix | Is almost feature complete (runs Signon, TLS) | Technology unknown in GOV.UK 37 | Against production | Low setup, fast | Only works for frontend applications. Still needs local install. Limits you to testing one application at a time. 38 | Remote dependencies | Low setup. Can match Production exactly in terms of technologies used | Expensive, no offline development 39 | Manual install | Fast in development | Lots of setup, brittle 40 | gsp-local | GOV.UK will move to the GSP at some point | Multiple new layers of technology, quite new 41 | 42 | ## Proposal 43 | 44 | We'll adopt a Docker based local development environment. Docker is widely supported in the community and is most aligned with our tech strategy. In the long term we might replace it with a local environment of the GDS Supported Platform. 45 | 46 | ## Impact and follow up work 47 | 48 | We'll commit to doing these things: 49 | 50 | - Document how to test puppet locally using Vagrant now that the development VM is going away. 51 | - Come up with a process of ownership so that the tooling improves over time. 52 | 53 | Things we'll need to look out for: 54 | 55 | - Make sure we can limit the number of dependencies we run, to avoid having the environment being too resource intensive. 56 | - Our local development will diverge more from our production environment. This might lead to more testing on integration. We should be sure this is viable and doesn't become a blocker if lots of people are testing things at the same time. 57 | -------------------------------------------------------------------------------- /rfc-097-verify-specific-start-pages.md: -------------------------------------------------------------------------------- 1 | --- 2 | status: accepted 3 | implementation: done 4 | status_last_reviewed: 2024-03-04 5 | --- 6 | 7 | # Providing alternative GOV.UK service start pages for the GOV.UK Verify single IDP journey 8 | 9 | ## Summary 10 | 11 | This document describes an approach for providing alternative service start pages 12 | that will be displayed to users on a GOV.UK Verify single IDP journey. 13 | 14 | ## Terms 15 | 16 | * IDP - Identity Provider (also known as a Certified Company) 17 | * RP - Relying Party (also known as a transaction or service, e.g. view your driving license or check your state pension) 18 | 19 | ## Problem 20 | 21 | We have recently introduced a new type of journey to Verify. This journey type allows Identity Providers (IDPs) to target their existing customers in order to drive customers into using their identity service with GOV.UK services. Unlike traditional Verify journeys, the user will actually start on a page at, or an e-mail from, a specific IDP where they will choose which service to use. During this journey Verify restricts their ability to choose an alternative IDP. This is know as the Single IDP journey and is described in Verify [RFC-041](https://github.com/alphagov/verify-architecture/blob/master/rfcs/rfc-041-single-idp-journey.md). 22 | 23 | When following a single IDP journey, the user needs to be redirected via the RP (service) in order to pick up a valid SAML request. Initial thoughts were to direct the RP headless start page, which generates the SAML request and results in an immediate redirect back to the Verify hub. 24 | This approach, however, is not desirable as there is often important information given to the 25 | user on the service's normal start page. Similarly, showing the normal start page is not 26 | desirable as it may give service sign-in options that would direct the user away from Verify and the 27 | user's chosen IDP (for example, if the user chooses to login with Government Gateway). 28 | 29 | The approach described in this document is to implement alternate service start pages, that 30 | will display all important information to the user but only give them the option of using 31 | Verify and, hence, their chosen IDP. 32 | 33 | ### Considerations 34 | 35 | #### Discoverability 36 | 37 | Any customised content is designed to be solely reachable, indirectly via the hub, using hyperlinks IDPs publish to their users. 38 | 39 | Therefore the new content MUST not be indexed and reachable by performing, for example, a Google search or GOV.UK search. 40 | 41 | #### Content Schema 42 | 43 | The GOV.UK content schema used for service start pages is the `transaction` schema, the changes proposed will require changes to this schema which will, potentially, require changes to be made to existing data that conforms to this schema. 44 | 45 | ## Proposal 46 | 47 | The solution would be for GOV.UK to support 'variants' of content in the transaction content schema. This should allow a variant of the page to be requested in the URL, as shown, `http://www.gov.uk//`. For example, `http://www.gov.uk/my-rp-transaction-page/verify`. 48 | 49 | This would require multiple changes to GOV.UK including schemas, publisher and government frontend. 50 | 51 | Publisher already supports the concept of a 'parted' content item, so the publisher app should be modified to add the 'parted' functionality to the transaction editor. Corresponding changes would also need to be made to the `transaction` content schema to ensure that the publishing API accepts multiple parts. Government frontend would also have to be modified to render the pages correctly. 52 | 53 | #### Searching 54 | 55 | Even though the new alternate parts will not be linked from anywhere, it should not be discoverable by crawlers, GOV.UK does publish a `sitemap.xml`. We need to ensure the variant pages include the `ROBOTS` meta tag with a `NOINDEX` value. 56 | 57 | This will be achieved by allowing a variant/part to be flagged with a "Exclude from Index" option. If present on a variant, when Government frontend renders the page it should include the NOINDEX value in the ROBOTS meta tag. 58 | 59 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # GOV.UK Request For Comments 2 | 3 | GOV.UK staff use this repository as a forum to discuss and make technical decisions. The outcomes of these discussions can be either an action plan, or a new standard that GOV.UK should follow. This repository is open as a reference for other teams within GDS and wider government. 4 | 5 | ## Process 6 | 7 | 1. Create a new branch on this repo and copy `rfc-000-template.md` to `rfc-000-my-proposal.md` and edit. 8 | 2. Include any images etc in a separate directory named `rfc-000` and link to them. 9 | 3. Make a Pull Request (PR) for your branch and open as a draft. 10 | 4. Rename your file and directory with the number of the PR and push changes. 11 | 5. When your RFC is ready to be commented on, mark you PR as "Ready for review" and set a deadline for comments (2 weeks is a common choice) and record that in the PR description. 12 | 5. Post a link to your PR in #govuk-developers on Slack and to the [govuk-tech-members][govuk-tech-members] Google Group. 13 | 6. GOV.UK members discuss your proposal using both inline comments against your RFC document and the general PR comments section. Non-technical staff will need to create a free Github account in order to comment. 14 | 7. As changes are requested and agreed in comments, make the changes in your RFC and push them as new commits. 15 | 8. Stay active in the discussion and encourage other relevant people to participate. If you’re unsure who should be involved in a discussion, ask your Tech Lead or a Lead Developer. If you start an RFC it’s up to you to push it through the process and engage people. 16 | 9. Once the deadline is reached you are able to merge if the PR has been approved by a member of GOV.UK Senior Tech (if they haven't or you're not sure, you can contact them on Slack with @govuk-senior-tech-people) - this action is the accepting of an RFC. 17 | 18 | ### If consensus isn't reached 19 | 20 | Should consensus not be reached by the deadline, an RFC can either have the deadline extended or the RFC can be closed as something not approiate for accepting now. For a deadline extension, you should make an effort to inform #govuk-developers about this status and request extra input. If the RFC is being closed, leave a comment explaining the status and close the PR. 21 | 22 | ### If the proposal is no longer needed 23 | 24 | If the proposal is no longer applicable the RFC PR can be closed. Please include a comment explaining why to help anyone considering the problem in future. 25 | 26 | ## Editing past RFCs 27 | 28 | RFCs should not be substantially altered after they are accepted as they intended to be kept as a point-in-time record of a decision. There are however a few reasons why you may change one that has been accepted: 29 | 30 | - to fix typos and other minor mistakes 31 | - to record a status change of the RFC in the YAML frontmatter (remember to update the status_last_reviewed date) 32 | - to mark an RFC as being superseded with a link to the RFC that supersedes it 33 | - any relevant post implementation, or post abandonment, supplementary details that would be useful for someone interested in the area. 34 | 35 | ## Historical RFCs 36 | 37 | Some RFCs in this repository were migrated from Confluence. They’ve been automatically converted to Markdown, so some formatting might be incorrect. Please fix any issues as you find them in new PRs. 38 | 39 | ## RFC metadata as YAML frontmatter 40 | 41 | Some RFCs have YAML frontmatter which allows us to track their status / implementation etc. 42 | 43 |
44 | Script to list all RFC metadata 45 | 46 | ```ruby 47 | #!/usr/bin/env ruby 48 | 49 | require "csv" 50 | require "yaml" 51 | 52 | frontmatter_columns = %w[status implementation status_last_reviewed status_notes] 53 | CSV do |csv| 54 | csv << ["filename", *frontmatter_columns] 55 | Dir.glob("rfc-*.md") do |filename| 56 | first_line = File.readlines(filename).first 57 | frontmatter = {} 58 | frontmatter = YAML.load_file(filename, permitted_classes: [Date]) if first_line =~ /^---$/ 59 | csv << [filename, *frontmatter.values_at(*frontmatter_columns)] 60 | end 61 | end 62 | ``` 63 | 64 |
65 | 66 | [govuk-tech-members]: https://groups.google.com/a/digital.cabinet-office.gov.uk/forum/#!forum/govuk-tech-members 67 | -------------------------------------------------------------------------------- /rfc-055-content-history.md: -------------------------------------------------------------------------------- 1 | ## Problem 2 | 3 | Publishing apps store change history for content items in a variety of different ways. As we move to a more centralised workflow, we need a consistent way of recording this data in the publishing-api and providing it for display to editors in publishing apps and to users in the frontends. 4 | 5 | ## Current behaviour 6 | 7 | Content history is usefully split into two mostly separate types. Public change history is exposed to users on the front end, and usually consists of a list of major update dates and manually entered change notes. Version history is shown to users in the publishing apps, and consists of dates when the content was changed or versioned; it might also include notes to other editors or fact checkers. 8 | 9 | The existing apps support a mixture of these history types: 10 | 11 | **Whitehall** 12 | 13 | - Public change note: note, public timestamp - one per edition 14 | - Fact check requests: instructions (i.e. note from editor), comments (i.e. response from fact-checker) 15 | - Editorial remarks: body, author 16 | - Versions: event (create, update), state (draft, published, submitted), user 17 | 18 | **Travel Advice Publisher** 19 | 20 | - Version history and public change notes only 21 | 22 | **Publisher** 23 | 24 | - Public change note is available as a field to edit but it is not displayed anywhere 25 | - Important note, edition note and action are all the same model: recipient, requester, comment, request type (new\_version, assign, request\_review, important\_note...) 26 | - Fact check responses are stored as actions but requests are not currently stored 27 | 28 | **Specialist Publisher** 29 | 30 | - Public change history only: date, change note (currently being sent embedded in details, built up manually) 31 | 32 | **Service Manual Publisher** 33 | 34 | - Public change history: date, change summary, reason for change 35 | - State change events: new draft, assign author, add comment, request review, approve, publish 36 | 37 | **Content tagger** 38 | 39 | - No history 40 | 41 | ## Proposal 42 | 43 | ### 44 | Public change history 45 | 46 | The `change_history` element will be deprecated in the publisher schemas; apps will stop building up this history themselves, and instead just send the current public change note.  47 | 48 | Publishing API will record, in a separate table, the set of **change notes**associated with a specific content ID. The `publish` command will, for major versions only, add an entry in this table with the contents of the `change_note` in the ContentItem to be published and the publish time. To support Service Manual Publisher, `reason_for_change` will be accepted in the schemas directly at the same level as `change_note` and recorded alongside it in the change notes table. 49 | 50 | The downstream presenters will assemble the public content history from the list of change notes. 51 | 52 | ### Version history 53 | 54 | Publishing API should support the concept of **actions**. An action will link to a ContentItem or LinkSet and record all the activity that happens to that version - create, update, publish, unpublish etc - along with the ID of the user who performed that action, the email address of any recipient, and the text of any note/remark. Actions could also link to the Event that caused the change. 55 | 56 |  will be each create an action when they are called. In addition, we could store the diff between the current and previous versions on each change; a [spike into this](https://github.com/alphagov/publishing-api/compare/content_history) already exists. 57 | 58 | The list of action types will be a superset of all those supported by the publishing apps, and no extra validation will be carried out to ensure that the action makes sense given the current state; at this point we are only recording history, we are not providing workflow or a state machine. 59 | 60 | Whitehall (and Publisher once we start migrating it) will also need to start sending data specifically for those actions that do not result from existing commands - eg add note/remark, send for fact check, etc. This will probably need to be on a new `action` endpoint; we might later decide to split these out into separate endpoints when we start implementing the workflow itself, but it will be helpful to start storing the data now. 61 | 62 | -------------------------------------------------------------------------------- /rfc-059-workflow-for-making-changes-to-the-schemas.md: -------------------------------------------------------------------------------- 1 | ## Problem 2 | 3 | Recently the publishing-api has begun to validate incoming payloads against the 4 | [govuk-content-schemas][]. 5 | 6 | How it works now: 7 | 8 | - The schemas are manually "deployed" using a [task in Jenkins][jenkins-task] 9 | that [copies][deploy-script] over the schemas to the publishing-api 10 | - Pull requests on publishing apps are tested against the master branch of 11 | govuk-content-schemas ([example][example-1]) 12 | - Pull requests on govuk-content-schemas are tested against the master branch 13 | of the downstream applications ([example][example-2]) 14 | 15 | [govuk-content-schemas]: https://github.com/alphagov/govuk-content-schemas 16 | [jenkins-task]: https://deploy.integration.publishing.service.gov.uk/job/Deploy_GOVUK_Content_Schemas/ 17 | [deploy-script]: https://github.com/alphagov/govuk-content-schemas/blob/master/deploy.sh 18 | [example-1]: https://github.com/alphagov/calendars/blob/51a9583b4de80aeca53c9f3762f6412c24a3c951/jenkins.sh#L45 19 | [example-2]: https://ci.dev.publishing.service.gov.uk/job/govuk_business_support_finder_schema_tests/configure 20 | 21 | This opens up two issues that could cause the publishing-api to reject valid 22 | content, causing errors or delays for editors. 23 | 24 | ### 1. Undeployed changes in schemas 25 | 26 | Example: 27 | 28 | - You add a field to govuk-content-schemas, but don't deploy 29 | - You add the field to the content item payload in the publisher application. 30 | The PR will be green because the new payload is valid to the content schema 31 | on master 32 | - PR gets merged and deployed 33 | - Now the app will fail on production because publishing-api doesn't know about 34 | your new attribute yet 35 | 36 | ### 2. Making schema changes with undeployed apps 37 | 38 | Example: 39 | 40 | - You want to remove an attribute from govuk-content-schemas. You remove it 41 | from the payload in the publisher application. 42 | - The PR is merged, but the application is not yet deployed to production. 43 | - Raise a PR on content-schemas to remove the payload. Downstream apps pass 44 | because the publisher app master isn't sending the attribute anymore. 45 | - When you deploy govuk-content-schemas the publisher app will be sending 46 | invalid content. 47 | 48 | ## Proposal 49 | 50 | This RFC proposes: 51 | 52 | - Application tests are to be run against the **deployed** version of 53 | govuk-content-schemas 54 | - Pull requests on govuk-content-schemas are to be tested against the 55 | **deployed** version of the downstream applications 56 | 57 | The scenarios above now can't happen: 58 | 59 | ### 1. Undeployed changes in schemas 60 | 61 | Example: 62 | 63 | - You add a field to govuk-content-schemas, but don't deploy 64 | - You add the field to the content item payload in the publisher application. 65 | The PR will **not pass** because the new payload is invalid against the 66 | released-to-production branch of content-schemas 67 | 68 | ### 2. Making schema changes with undeployed apps 69 | 70 | Example: 71 | 72 | - You want to remove an attribute from govuk-content-schemas. You remove it 73 | from the payload in the publisher application. 74 | - The PR is merged, but the application is not yet deployed to production. 75 | - Raise a PR on content-schemas to remove the payload. Downstream apps **do not 76 | pass** because the publisher app still has the attribute in 77 | released-to-production 78 | 79 | Because we don't deploy automatically to production, there's a situation that 80 | will cause a PR to get merged that will fail on production: 81 | 82 | 1. Add an attribute to govuk-content-schemas. Merge the PR & deploy to 83 | production. 84 | 2. Raise PR to send this attribute from the publisher application. Your tests 85 | pass (because it's testing against released-to-production schemas) 86 | 3. Merge the PR on the publisher application (but do not deploy) 87 | 4. Now raise a PR to remove attribute from content schemas. Because the change 88 | hasn't been deployed the tests pass **will pass**. Merge it. 89 | 5. If you deploy the PR on the publisher application now, the publisher app 90 | will start sending invalid data to the publishing-api and fail 91 | 92 | To illustrate how schema changes take place, [we've made some diagrams 93 | describing the process][diagrams]. 94 | 95 | [diagrams]: https://gov-uk.atlassian.net/wiki/display/GOVUK/Illustration+of+schema+development+workflow 96 | -------------------------------------------------------------------------------- /rfc-016-how-to-prevent-published-live-frontends-from-reading-from-the-draft-content-store.md: -------------------------------------------------------------------------------- 1 | --- 2 | status: superseded 3 | implementation: superseded 4 | status_last_reviewed: 2024-03-06 5 | status_notes: We no longer use this infrastructure. 6 | --- 7 | 8 | # How to prevent published frontends from reading from the draft content store 9 | 10 | ## Problem 11 | 12 | The old draft stack used firewall rules to ensure that instances of frontend applications intended to serve published content were prevented from inadvertently reading from the instance of the content-store application used to serve draft (unpublished) content. 13 | 14 | This is an operational risk for GOV.UK, since unpublished content may be [embargoed](https://en.wikipedia.org/wiki/News_embargo) for publication or otherwise contain sensitive information that should not become public until published. 15 | 16 | Now that applications serving draft content are hosted in the same vCloud organisation as their counterpart instances used for published content (see ), we must identify any vectors through which the published frontends could read from the draft content store. 17 | 18 | ## Proposal 19 | 20 | ### Risk vectors 21 | 22 | We have identified two ways that a published frontend application might inadvertently read from the draft instance of the content-store application: 23 | 24 | 1. The instance of the content-store used for published content could inadvertently read from the draft instance of the content-store database if its database settings were misconfigured. 25 | 2. Published instances of frontend applications could inadvertently read from the draft instance of the content-store application if their configuration was incorrect. 26 | 27 | ### Mitigation for vector (1) 28 | 29 | Vector (1) has been mitigated by [avoiding the use of a default](https://github.gds/gds/puppet/commit/3fa80cdceb7138dc2f1a7e4aba90976274a3ce65#diff-317c81f58cd20afec981e1e6c339703f) for the name of the content-store database when configuring the content-store application, meaning that each machine class (i.e. [content\_store](https://github.gds/gds/puppet/blob/effc4c0cab1/hieradata/class/content_store.yaml#L3) or [draft\_content\_store](https://github.gds/gds/puppet/blob/effc4c0cab1/hieradata/class/draft_content_store.yaml#L3)) must explicitly set the database name. 30 | 31 | ### Mitigation for vector (2) 32 | 33 | The draft instance of the content-store application is fronted by Nginx the existing api\_lb machines, much the same way that the published instance of content-store is fronted by Nginx. The decision to share the api\_lb for requests for both draft and published content was made to avoid the additional expense for creating two new draft\_api\_lb machines for serving draft content. 34 | 35 | We initially explored access limiting by IP address within the Nginx virtual host for requests going to `draft-content-store.production.alphagov.co.uk` on the api\_lb machines so as to block requests coming from published instances of frontend applications, however this was not possible because the client IP address in the HTTP request is presented as originating from the API vDC network's gateway IP address. This makes it impossible, in this configuration, to differentiate between requests from published and draft instances of frontend applications. 36 | 37 | The only other possibility for implementing access control by IP address would be in the vShield Edge Gateway's firewall rules. Applying a firewall rule for the [existing API vSE load balancer](https://github.gds/gds/govuk-provisioning/blob/c33df9b/vcloud-edge_gateway/rules/lb.yaml.mustache#L169-L175) would not work as both published and draft applications would make requests through the same path and there would be no way to distinguish between requests intended for the published versus draft instances of content-store.  38 | 39 | To make that distinction possible, we propose to: 40 | 41 | - Instantiate a new vShield Edge Gateway (vSE) load balancer named 'DraftAPI', which will use the existing api\_lb machines as its pool members on port 8443. 42 | - Add a new virtual host to Nginx on the api\_lb machines that listens on port 8443 and have it serve requests destined for `draft-content-store.production.alphagov.co.uk` 43 | - Add a firewall rule to the vShield Edge Gateway that prevents IP addresses other than those used by the draft\_frontend machines from connecting to the new 'DraftAPI' vSE load balancer. 44 | 45 |   46 | 47 |   48 | 49 |   50 | 51 | -------------------------------------------------------------------------------- /rfc-087-dealing-with-errors.md: -------------------------------------------------------------------------------- 1 | # Dealing with errors 2 | 3 | ## Summary 4 | 5 | This describes how we treat errors on GOV.UK 6 | 7 | ## Problem 8 | 9 | We've recently migrated to a new error tracking service, Sentry. This provides us an opportunity to rethink how we treat errors. 10 | 11 | ## Proposal 12 | 13 | There are 2 principles: 14 | 15 | ### 1. When something goes wrong, we should be notified 16 | 17 | Applications should report exceptions to Sentry. Applications must not swallow errors. 18 | 19 | ### 2. Notifications should be actionable 20 | 21 | Sentry notifications should be something that requires a developer of the app to do something about it. Not just a piece of information. 22 | 23 | ### 3. Applications should not error 24 | 25 | The goal of GOV.UK is that applications should not error. When something goes wrong it should be fixed. 26 | 27 | ## Classifying errors 28 | 29 | ### Bug 30 | 31 | A code change makes the application crash. 32 | 33 | Desired behaviour: error is sent to Sentry, developers are notified and fix the error. Developers mark the error in Sentry as `Resolved`. This means a recurrence of the error will alert developers again. 34 | 35 | ### Intermittent errors without user impact 36 | 37 | Frontend applications often see timeouts when talking to the content-store. 38 | 39 | There's no or little user impact because the request will be answered by the caching layer. 40 | 41 | Example: 42 | 43 | Desired behaviour: error is not sent to Sentry. Instead, we rely on Smokey and Icinga checks to make sure we the site functions. 44 | 45 | ### Intermittent errors with user impact 46 | 47 | Publishing applications sometimes see timeouts when talking to publishing-api. This results in the publisher seeing an error page and possibly losing data. 48 | 49 | Example: 50 | 51 | Desired behaviour: apps handle these errors better, for example by offloading the work to a Sidekiq worker. Since these errors aren't actionable, they should not be reported to Sentry. They should be tracked in Graphite. 52 | 53 | ### Intermittent retryable errors 54 | 55 | Sidekiq worker sends something to the publishing-api, which times out. Sidekiq retries, the next time it works. 56 | 57 | Desired behaviour: errors are not reported to Sentry until retries are exhausted. See [this PR for an example](https://github.com/alphagov/content-performance-manager/pull/353). 58 | 59 | Relevant: https://github.com/getsentry/raven-ruby/pull/784 60 | 61 | ### Expected environment-based errors 62 | 63 | MySQL errors on staging while data sync happens. 64 | 65 | Example: 66 | 67 | Desired behaviour: our environment is set up such that these errors do not occur. 68 | 69 | ### Bad request errors 70 | 71 | User makes a request the application can't handle ([example][bad-request]). 72 | 73 | Often happens in [security checks](https://sentry.io/govuk/app-frontend/issues/400074979). 74 | 75 | Example: 76 | 77 | Desired behaviour: user gets feedback, error is not reported to Sentry 78 | 79 | [bad-request]: https://sentry.io/govuk/app-service-manual-frontend/issues/400074003 80 | 81 | ### Incorrect bubbling up of errors 82 | 83 | Rummager crashes on date parsing, returns `422`, which raises an error in finder-frontend. 84 | 85 | Example: 86 | 87 | Desired behaviour: a 4XX reponse is returned to the browser, including an error message. Nothing is ever sent to Sentry. 88 | 89 | ### Manually logged errors 90 | 91 | Something goes wrong and we need to let developers know. 92 | 93 | Example: [Slimmer's old behaviour](https://github.com/alphagov/slimmer/pull/203/files#diff-e5615a250f587cf4e2147f6163616a1a) 94 | 95 | Desired behaviour: developers do not use Sentry for logging. The app either raises the actual error (which causes the user to see the error) or logs the error to Kibana. 96 | 97 | ### IP spoof errors 98 | 99 | Rails reports `ActionDispatch::RemoteIp::IpSpoofAttackError`. 100 | 101 | Example: 102 | 103 | Desired behaviour: HTTP 400 is returned, error is not reported to Sentry. 104 | 105 | ### Database entry not found 106 | 107 | Often a controller will do something like `Thing.find(params[:id])` and rely on Rails to show a 404 page for the `ActiveRecord::RecordNotFound` it raises ([context](https://stackoverflow.com/questions/27925282/activerecordrecordnotfound-raises-404-instead-of-500)). 108 | 109 | Desired behaviour: errors are not reported to Sentry 110 | -------------------------------------------------------------------------------- /rfc-027-supporting-slug-changes-in-the-publishing-api.md: -------------------------------------------------------------------------------- 1 | --- 2 | status: accepted 3 | implementation: done 4 | status_last_reviewed: 2024-03-06 5 | --- 6 | 7 | # Supporting slug changes in the Publishing API 8 | 9 | ## Problem 10 | 11 | - slug changes are **costly and increasingly common** for live content 12 | - we also need to handle slug changes for **draft content** 13 | 14 | ### Slug changes in live content 15 | 16 | GOV.UK's existing publishing tools were built upon the assumption that document slugs do not change. This was a helpful simplifying assumption in the early days of GOV.UK and mostly held true. In the rare cases where a published slug did need to change, the cost of manual intervention from a developer to correct the issue was reasonable. 17 | 18 | However as GOV.UK has matured, the factors in this trade-off have changed: 19 | 20 | - we now have vastly more content on GOV.UK, so although the chance of a particular document needing a slug change remains low, slug change requests are still much more frequent 21 | - our systems have become more complex over time and the effort involved in performing a slug change manually without introducing errors has increased 22 | 23 | If we need evidence of the cost and the frequency of slug changes, we could review the workload of 2nd line support. 24 | 25 | ### Slug changes in draft content 26 | 27 | We have always allowed slugs of draft content to change. This has never been an issue because draft items were contained within a single system, so there was no requirement to maintain consistency between multiple systems representing the same content item. 28 | 29 | With the introduction of the 'content preview' system in Publishing API, we now handle draft items whose slugs can change.  30 | 31 | The primary identifier used for content items in the publishing API is the `base_path`. Using `base_path` as the primary identifier is based on the assumption that it does not change. 32 | 33 | If the slug of a draft content item changes, our only option at present would be to require the publishing application to notify the publishing API of this change so that it can remove the document at the previous slug (publishing API currently does not support deletion of content items). 34 | 35 | ## Proposal 36 | 37 | ### Proposal 1: we should assume that slug changes will happen and incorporate this into the design of the publishing API 38 | 39 | The simplifying assumption that slug changes do not happen is no longer serving us. 40 | 41 | ### Proposal 2: use content\_id as the primary identifier of content items 42 | 43 | In order to cater for the above change in assumptions, we should use a persistent abstract identifier for content items. We already have such an identifier in the systems, in the form of `content_id`.  44 | 45 | Since it's a GUID it can be generated independently and asynchronously by the publishing applications (no need for a central coordinating authority). 46 | 47 | All publishing API endpoints should accept `content_id` rather than `base_path`, ie. instead of: 48 | 49 | This implies that `content_id` would be **required**  for all content items (not an onerous requirement). 50 | 51 | In order to transition to this approach there are a few options: 52 | 53 | - introduce a set of publishing API endpoints which accept content by guid (e.g. `PUT /content_by_guid/,` `PUT /draft_content_by_guid/` or something similar) 54 | - allow the existing endpoints to detect a slug which looks like a guid and treat it as such. Although slightly hacky, the chance that a normal slug would match the pattern for a guid is extremely low. 55 | 56 | #### Benefits of using `content_id` as primary identifier 57 | 58 | This will allow the publishing API to understand when the slug of a content item posted has changed. It will then allow publishing API to either: 59 | 60 | - disallow the change 61 | - gracefully handle the change by propogating the change to any downstream systems (e.g. router, url arbiter etc). It could even put in place a redirect from the old url to the new url. 62 | 63 | Further down the line, if we move to a system where publishing API keeps some kind of 'transaction log' record, this API will allow us to keep a record of the changes of a documents slug over time. Having this data in a single transaction log will mean that we have all the information in one place to verify and enforce consistency in downstream systems. 64 | 65 | ## Status of this RFC 66 | 67 | This is an early draft, there are probably many things I have missed or not thought about. 68 | 69 | - Do you agree with the end goal? 70 | - Do you see any issues with migrating to this? 71 | - Can you see any problems or risks I haven't identified? 72 | - Does anything need fleshing out further? 73 | 74 | Thanks for reading and for your input! 75 | -------------------------------------------------------------------------------- /rfc-145-unarchive-govuk_admin_template.md: -------------------------------------------------------------------------------- 1 | --- 2 | status: accepted 3 | implementation: done 4 | status_last_reviewed: 2024-03-04 5 | status_notes: "We're getting close to being able to archive this again, with the publishing apps removing their dependency on Bootstrap" 6 | --- 7 | 8 | # Unarchive govuk_admin_template Bootstrap project 9 | ## Summary 10 | 11 | The [govuk-admin-template](https://github.com/alphagov/govuk_admin_template) gem has been deprecated for a while, and new GOV.UK admin applications should be built using the [layout component in govuk_publishing_components](https://govuk-publishing-components.herokuapp.com/component-guide/layout_for_admin). Ideally, we would have migrated all our applications to use the new Design System and publishing components, however there are still several in active use which still rely on govuk-admin-template. The GitHub project is archived, and is therefore read-only. 12 | 13 | This RFC proposes un-archiving the project to allow iterative accessibility fixes to be applied to live applications. 14 | 15 | Importantly, this RFC _does not_ propose removing the deprecation notice, or advocate for significant development of govuk_admin_template. It is intended as a stop-gap that acknowledges the slow pace of migration to the Design System, and its intention is to yield accessibility fixes for users prior to those migrations taking place. 16 | 17 | ## Problem 18 | 19 | When [govuk-admin-template](https://github.com/alphagov/govuk_admin_template) was deprecated and made read-only back in July 2018, we anticipated projects would be swiftly migrated to the GOV.UK Design System. There are many benefits to this, and projects should still aim to be migrated. 20 | 21 | Three and a half years on, we still have [18](#projects-using-govuk_admin_template) projects referencing govuk-admin-template. This includes many publishing apps, most prominently [Whitehall](https://github.com/alphagov/whitehall). 22 | 23 | Several accessibility issues have been identified our publishing applications which stem from govuk-admin-template. Some of these are relatively easy fixes, however we can't apply them because the repository is read-only. 24 | 25 | ## Proposal 26 | 27 | With the intention of enabling quick wins for some perennial accessibility issues, we propose: 28 | 29 | 1. We will un-archive [govuk-admin-template](https://github.com/alphagov/govuk_admin_template) on GitHub, making it possible to release new versions of the gem with accessibility fixes. 30 | 31 | 2. GOV.UK developers may release new versions of the gem to address **critical** accessibility, usability, and security issues. 32 | 33 | 3. Teams **MUST** endeavour to migrate pages and applications to the GOV.UK Design System where practical. 34 | 35 | 4. (Unchanged) Teams **MUST NOT** create new applications, user journeys or significant new features using `govuk-admin-template`. 36 | 37 | ## Consequences 38 | 39 | - By improving accessibility in the govuk-admin-template, there may be less impetus to migrate to the GOV.UK Design System. We believe the immediate benefits to users outweighs this concern. All applications are still expected to be migrated, and it's important that we retain senior management buy-in for the migration. 40 | 41 | - By enabling fixes in the underlying gems, we can start iteratively improving some of our oldest and most painful systems. 42 | 43 | ## Appendices 44 | 45 | ### Projects using govuk_admin_template 46 | 47 | > It's not possible to filter archived repositories in GitHub search yet, but manually filtering [this search](https://github.com/search?p=3&q=org%3Aalphagov+%22gem+govuk_admin_template%22&type=Code) yields: 48 | 49 | - [collections-publisher](https://github.com/alphagov/collections-publisher) 50 | - [contacts-admin](https://github.com/alphagov/contacts-admin) 51 | - [content-tagger](https://github.com/alphagov/content-tagger) 52 | - [imminence](https://github.com/alphagov/imminence) 53 | - [local-links-manager](https://github.com/alphagov/local-links-manager) 54 | - [manuals-publisher](https://github.com/alphagov/manuals-publisher) 55 | - [maslow](https://github.com/alphagov/maslow) 56 | - [publisher](https://github.com/alphagov/publisher) 57 | - [search-admin](https://github.com/alphagov/search-admin) 58 | - [search-performance-explorer](https://github.com/alphagov/search-performance-explorer) 59 | - [service-manual-publisher](https://github.com/alphagov/service-manual-publisher) 60 | - [short-url-manager](https://github.com/alphagov/short-url-manager) 61 | - [signon](https://github.com/alphagov/signon) 62 | - [specialist-publisher](https://github.com/alphagov/specialist-publisher) 63 | - [support](https://github.com/alphagov/support) 64 | - [transition](https://github.com/alphagov/transition) 65 | - [travel-advice-publisher](https://github.com/alphagov/travel-advice-publisher) 66 | - [whitehall](https://github.com/alphagov/whitehall) 67 | -------------------------------------------------------------------------------- /rfc-098-csp.md: -------------------------------------------------------------------------------- 1 | --- 2 | status: accepted 3 | implementation: done 4 | status_last_reviewed: 2024-03-04 5 | status_notes: RFC may not be accurate in the implementation details. 6 | --- 7 | 8 | # RFC 98: Implement Content Security Policy 9 | 10 | ## Summary 11 | 12 | This RFC proposes to configure a Content Security Policy for GOV.UK. 13 | 14 | ## Background 15 | 16 | We'd like to implement a Content Security Policy (CSP) on www.gov.uk. 17 | 18 | > The HTTP Content-Security-Policy response header allows web site administrators to control resources the user agent is allowed to load for a given page. With a few exceptions, policies mostly involve specifying server origins and script endpoints. This helps guard against cross-site scripting attacks (XSS). 19 | 20 | https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Security-Policy 21 | 22 | CSP works by sending a header with each HTTP response. It looks something like this: 23 | 24 | ``` 25 | Content-Security-Policy: default-src 'self' assets.publishing.service.gov.uk; 26 | ``` 27 | 28 | The above will cause the browser to reject any scripts that aren't on the current domain (`self`) or `assets.publishing.service.gov.uk`. It will also reject Javascript loaded via `