├── CONTRIBUTING.md ├── SECURITY.md ├── CODE_OF_CONDUCT.md ├── logos ├── CarvelLogo.png ├── vmware.svg └── revng.svg ├── proposals ├── imgpkg │ ├── 002-recursive-bundles │ │ ├── examples │ │ │ ├── bundle-2 │ │ │ │ ├── cli.sh │ │ │ │ ├── .imgpkg │ │ │ │ │ ├── images.yml │ │ │ │ │ └── bundle.yml │ │ │ │ └── Readme.md │ │ │ ├── bundle-1 │ │ │ │ ├── Readme.md │ │ │ │ ├── .imgpkg │ │ │ │ │ ├── bundle.yml │ │ │ │ │ └── images.yml │ │ │ │ └── config.yml │ │ │ └── recursive-bundle │ │ │ │ ├── Readme.md │ │ │ │ ├── overlay.yml │ │ │ │ └── .imgpkg │ │ │ │ ├── bundle.yml │ │ │ │ └── images.yml │ │ └── README.md │ ├── README.md │ └── 001-bundles │ │ ├── bundle-initial.yml │ │ ├── workflow.md │ │ └── README.md ├── ytt │ ├── README.md │ ├── 001-schemas │ │ ├── future-schema-work.md │ │ ├── example-partial-dex.yaml │ │ ├── cf-for-k8s │ │ │ ├── values.yml │ │ │ ├── ytt-schema.yml │ │ │ ├── json-schema.yml │ │ │ └── json-schema.json │ │ └── README.md │ └── 002-raw-data-values │ │ └── README.md ├── README.md ├── carvel │ └── 001-maintain-documentation │ │ └── README.md └── kapp-controller │ └── 003-package-cr-split │ └── README.md ├── images └── backlog-workflow.png ├── README.md ├── .github └── workflows │ └── add-to-project.yml ├── ADOPTERS.md └── GOVERNANCE.md /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | https://carvel.dev/shared/docs/latest/contributing/ 2 | -------------------------------------------------------------------------------- /SECURITY.md: -------------------------------------------------------------------------------- 1 | https://carvel.dev/shared/docs/latest/security-policy/ 2 | -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | https://carvel.dev/shared/docs/latest/code-of-conduct/ 2 | -------------------------------------------------------------------------------- /logos/CarvelLogo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/carvel-dev/carvel-community/HEAD/logos/CarvelLogo.png -------------------------------------------------------------------------------- /proposals/imgpkg/002-recursive-bundles/examples/bundle-2/cli.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | echo "I am a CLI" 4 | -------------------------------------------------------------------------------- /images/backlog-workflow.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/carvel-dev/carvel-community/HEAD/images/backlog-workflow.png -------------------------------------------------------------------------------- /proposals/ytt/README.md: -------------------------------------------------------------------------------- 1 | # Proposals 2 | 3 | Feature proposals for `ytt`. See each proposal for their status. 4 | 5 | - [Schemas](001-schemas/) 6 | -------------------------------------------------------------------------------- /proposals/imgpkg/002-recursive-bundles/examples/bundle-2/.imgpkg/images.yml: -------------------------------------------------------------------------------- 1 | apiVersion: imgpkg.carvel.dev/v1alpha1 2 | kind: ImagesLock 3 | images: 4 | -------------------------------------------------------------------------------- /proposals/imgpkg/002-recursive-bundles/examples/bundle-1/Readme.md: -------------------------------------------------------------------------------- 1 | ## Application bundle 2 | 3 | This bundle contains the application plus some useful CLI 4 | -------------------------------------------------------------------------------- /proposals/imgpkg/002-recursive-bundles/examples/recursive-bundle/Readme.md: -------------------------------------------------------------------------------- 1 | ## Recursive bundle 2 | 3 | This bundle will provide both bundle-1 and bundle-2 4 | -------------------------------------------------------------------------------- /proposals/imgpkg/README.md: -------------------------------------------------------------------------------- 1 | # Proposals 2 | 3 | Feature proposals for `imgpkg`. See each proposal for their status. 4 | 5 | - [Bundles](001-bundles/) 6 | - [Recursive Bundles](002-recursive-bundles/) 7 | -------------------------------------------------------------------------------- /proposals/imgpkg/002-recursive-bundles/examples/bundle-2/Readme.md: -------------------------------------------------------------------------------- 1 | ## Utilities bundle 2 | 3 | This bundle will contain this readme and a CLI 4 | 5 | This image can be found in: ghcr.io/k14s/design-docs/utilities 6 | -------------------------------------------------------------------------------- /proposals/imgpkg/002-recursive-bundles/examples/recursive-bundle/overlay.yml: -------------------------------------------------------------------------------- 1 | #@ load("@ytt:overlay", "overlay") 2 | 3 | #@overlay/match by=overlay.all,expects="1+" 4 | --- 5 | metadata: 6 | namespace: new-namespace 7 | -------------------------------------------------------------------------------- /proposals/imgpkg/002-recursive-bundles/examples/bundle-1/.imgpkg/bundle.yml: -------------------------------------------------------------------------------- 1 | apiVersion: imgpkg.carvel.dev/v1alpha1 2 | kind: Bundle 3 | metadata: 4 | name: bundle-1 5 | authors: 6 | - name: Carvel Team 7 | email: carvel@vmware.com 8 | websites: 9 | - url: carvel.dev/imgpkg 10 | -------------------------------------------------------------------------------- /proposals/imgpkg/002-recursive-bundles/examples/bundle-2/.imgpkg/bundle.yml: -------------------------------------------------------------------------------- 1 | apiVersion: imgpkg.carvel.dev/v1alpha1 2 | kind: Bundle 3 | metadata: 4 | name: bundle-2 5 | authors: 6 | - name: Carvel Team 7 | email: carvel@vmware.com 8 | websites: 9 | - url: carvel.dev/imgpkg 10 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | **DEPRECATED**: This repo has been deprecated in favor of https://github.com/carvel-dev/carvel 2 | 3 | ![logo](logos/CarvelLogo.png) 4 | 5 | Carvel provides a set of reliable, single-purpose, composable tools that aid in your application building, configuration, and deployment to Kubernetes. 6 | -------------------------------------------------------------------------------- /proposals/imgpkg/002-recursive-bundles/examples/recursive-bundle/.imgpkg/bundle.yml: -------------------------------------------------------------------------------- 1 | apiVersion: imgpkg.carvel.dev/v1alpha1 2 | kind: Bundle 3 | metadata: 4 | name: recursive-bundle 5 | authors: 6 | - name: Carvel Team 7 | email: carvel@vmware.com 8 | websites: 9 | - url: carvel.dev/imgpkg 10 | -------------------------------------------------------------------------------- /proposals/imgpkg/001-bundles/bundle-initial.yml: -------------------------------------------------------------------------------- 1 | #! Describes contents and authorship of bundle 2 | --- 3 | apiVersion: pkg.k14s.io/v1alpha1 4 | kind: Bundle 5 | metadata: 6 | name: my-app 7 | authors: 8 | - name: sarah 9 | email: sarah@example.com 10 | websites: 11 | - url: my-app.example.com 12 | contents: 13 | paths: 14 | - contents/** -------------------------------------------------------------------------------- /proposals/imgpkg/002-recursive-bundles/examples/recursive-bundle/.imgpkg/images.yml: -------------------------------------------------------------------------------- 1 | apiVersion: imgpkg.carvel.dev/v1alpha1 2 | kind: ImagesLock 3 | images: 4 | - image: ghcr.io/k14s/design-docs/simple-app-bundle@sha256:d211dd700949154e429d28661d01c99d53a38af0d5275842ccbf0bf6dbef8ca4 5 | - image: ghcr.io/k14s/design-docs/utilities@sha256:47ae428a887c41ba0aedf87d560eb305a8aa522ffb80ac1c96a37b16df038e0f 6 | -------------------------------------------------------------------------------- /proposals/imgpkg/002-recursive-bundles/examples/bundle-1/.imgpkg/images.yml: -------------------------------------------------------------------------------- 1 | apiVersion: imgpkg.carvel.dev/v1alpha1 2 | kind: ImagesLock 3 | images: 4 | - image: ghcr.io/k14s/design-docs/simple-app@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0 5 | annotations: 6 | kbld.carvel.dev/id: my.registry.io/simple-application 7 | - image: ghcr.io/k14s/design-docs/utilities@sha256:47ae428a887c41ba0aedf87d560eb305a8aa522ffb80ac1c96a37b16df038e0f 8 | -------------------------------------------------------------------------------- /.github/workflows/add-to-project.yml: -------------------------------------------------------------------------------- 1 | --- 2 | name: Add issues to project 3 | 4 | on: 5 | issues: 6 | types: ['opened'] 7 | pull_request: 8 | types: ['opened'] 9 | 10 | jobs: 11 | add-to-project: 12 | name: Add issue to project 13 | runs-on: ubuntu-latest 14 | steps: 15 | - uses: actions/add-to-project@v0.3.0 16 | with: 17 | project-url: https://github.com/orgs/vmware-tanzu/projects/16 18 | github-token: ${{ secrets.CARVEL_ADD_TO_PROJECT_TOKEN }} 19 | -------------------------------------------------------------------------------- /proposals/imgpkg/002-recursive-bundles/examples/bundle-1/config.yml: -------------------------------------------------------------------------------- 1 | --- 2 | apiVersion: v1 3 | kind: Service 4 | metadata: 5 | namespace: default 6 | name: simple-app 7 | spec: 8 | ports: 9 | - port: 80 10 | targetPort: 80 11 | selector: 12 | simple-app: "" 13 | --- 14 | apiVersion: apps/v1 15 | kind: Deployment 16 | metadata: 17 | namespace: default 18 | name: simple-app 19 | spec: 20 | selector: 21 | matchLabels: 22 | simple-app: "" 23 | template: 24 | metadata: 25 | labels: 26 | simple-app: "" 27 | spec: 28 | containers: 29 | - name: simple-app 30 | image: my.registry.io/simple-application 31 | env: 32 | - name: HELLO_MSG 33 | value: stranger 34 | -------------------------------------------------------------------------------- /proposals/ytt/001-schemas/future-schema-work.md: -------------------------------------------------------------------------------- 1 | #### Complex schema annotations 2 | 3 | - `@schema/any-of` 4 | Requires the key to satisfy _at least one_ of the provided schemas 5 | ```yaml 6 | #@schema/any-of schema1(), schema2() 7 | foo: "" 8 | ``` 9 | Note, any values passed to the `any-of` call will be interpreted as schema documents. 10 | 11 | - `@schema/all-of` 12 | Requires the key to satisfy _every_ provided schema 13 | ```yaml 14 | #@schema/all-of schema1(), schema2() 15 | foo: "" 16 | ``` 17 | Note, any values passed to the `all-of` call will be interpreted as schema documents. 18 | 19 | - `@schema/one-of` 20 | Requires the key to satisfy _exactly one_ of provided schema 21 | ```yaml 22 | #@schema/one-of schema1(), schema2() 23 | foo: "" 24 | ``` 25 | Note, any values passed to the `one-of` call will be interpreted as schema documents. 26 | -------------------------------------------------------------------------------- /ADOPTERS.md: -------------------------------------------------------------------------------- 1 | # Carvel Adopters 2 | 3 | If you're using Carvel and want to add your organization to this 4 | list, [follow these directions](#adding-your-organization-to-the-list-of-adopters)! 5 | 6 | ## Organizations using Carvel 7 | 8 | VMware 9 | 10 | Revng 11 | 12 | ## Solutions built with Carvel 13 | 14 | Below is a list of solutions where Carvel is being used as a component. 15 | 16 | **[Revng](https://rev.ng/)** 17 | 18 | Revng is a small company with expertise in compilers, emulation and binary analysis. Revng uses ytt as a flexible templating tool to generate the configuration for [orchestra](https://github.com/revng/orchestra), their meta build system/package manager. 19 | 20 | **[VMware](https://www.vmware.com)** 21 | 22 | VMware uses Carvel as their package management tooling for [their Kubernetes offerings](https://tanzu.vmware.com/products), such as [Tanzu Mission Control](https://tanzu.vmware.com/mission-control) (TMC) and [Tanzu Kubernetes Grid](https://tanzu.vmware.com/kubernetes-grid) (TKG). 23 | 24 | ## Adding your organization to the list of adopters 25 | 26 | If you are using Carvel and would like to be included in the list of Carvel Adopters, add an SVG version of your logo to 27 | the `logos` directory in this repo and submit a pull request with your change. Name the image file something that 28 | reflects your company (e.g., if your company is called Acme, name the image acme.svg). 29 | See [this PR](https://github.com/vmware-tanzu/carvel/pull/4) for an example. 30 | -------------------------------------------------------------------------------- /proposals/imgpkg/001-bundles/workflow.md: -------------------------------------------------------------------------------- 1 | # Making a bundle 2 | 3 | ## team builds a bundle 4 | 5 | instead of providing users with tgz with config, we want to serve that artifact off of registry 6 | 7 | ```bash 8 | # for example, in CI 9 | cd app/ 10 | kbld -f <(ytt ...) --lock-output .imgpkg/images.yml # build images.yml as we do today 11 | imgpkg push -b registry.vendor.com/app-staging:v0.5.0 -f . 12 | 13 | # relocate all external depedencies into registry.com 14 | imgpkg copy -b registry.vendor.com/app-staging:v0.5.0 --to-repo registry.vendor.com/app-prod 15 | ``` 16 | 17 | --- 18 | # Consuming bundle 19 | 20 | ## without relocation 21 | 22 | (see below for relocation) 23 | 24 | ```bash 25 | # potentially auth to registry for pulling bundle 26 | export IMGPKG_REGISTRY_HOSTNAME=registry.com 27 | export IMGPKG_REGISTRY_USERNAME=foo 28 | export IMGPKG_REGISTRY_PASSWORD=bar 29 | 30 | imgpkg pull -b registry.vendor.com/app-prod:v0.5.0 -o /tmp/app 31 | 32 | cd /tmp/app 33 | edit /tmp/app-values/values.yml # create specific data values 34 | kapp deploy -a cf -f <(ytt -f config/ -f /tmp/app-values/values.yml | kbld -f .imgpkg/images.yml) 35 | ``` 36 | 37 | 38 | ## ... or with relocation first 39 | 40 | ```bash 41 | # potentially auth to registry for pushing bundle 42 | export IMGPKG_REGISTRY_HOSTNAME=registry.customer.com 43 | export IMGPKG_REGISTRY_USERNAME=foo 44 | export IMGPKG_REGISTRY_PASSWORD=bar 45 | 46 | imgpkg copy -b registry.vendor.com/app-prod:v0.5.0 --to-repo registry.customer.com/app 47 | imgpkg pull -b registry.customer.com/app:v0.5.0 -o /tmp/app 48 | 49 | cd /tmp/app 50 | edit /tmp/app-values/values.yml 51 | kapp deploy -a cf -f <(ytt -f config/ -f /tmp/app-values/values.yml | kbld -f .imgpkg/images.yml) 52 | ``` 53 | 54 | 55 | ## ... or with air-gapped relocation 56 | 57 | ```bash 58 | imgpkg copy -b registry.vendor.com/app-prod:v0.5.0 --to-tar app-v0.5.0.tar 59 | # transport tarball to the bunker ... 60 | 61 | # from the bunker ... 62 | imgpkg copy --tar app-v0.5.0.tar --to-repo registry.customer.com/app 63 | imgpkg pull -b registry.customer.com/app:v0.5.0 -o /tmp/app 64 | 65 | cd /tmp/app 66 | edit /tmp/app-values/values.yml 67 | kapp deploy -a cf -f <(ytt -f config/ -f /tmp/app-values/values.yml | kbld -f .imgpkg/images.yml) 68 | ``` 69 | -------------------------------------------------------------------------------- /proposals/ytt/001-schemas/example-partial-dex.yaml: -------------------------------------------------------------------------------- 1 | #! https://github.com/helm/charts/blob/21e2e1b1f2656c785ece0ec741b047b21539d7b1/stable/dex/values.yaml 2 | #@schema/match data_values=True 3 | --- 4 | config: 5 | #@schema/validate prefix="https://" 6 | #@schema/example "http://dex.example.com:8080" 7 | issuer: "" 8 | storage: 9 | #@schema/validate enum=["kubernetes"] 10 | type: kubernetes 11 | config: 12 | inCluster: true 13 | logger: 14 | #@schema/validate enum=["debug", "info", "error"] 15 | level: debug 16 | 17 | web: 18 | #@schema/doc "port is taken from ports section above" 19 | #@schema/validate format="ipv4" 20 | address: 0.0.0.0 21 | tlsCert: /etc/dex/tls/https/server/tls.crt 22 | tlsKey: /etc/dex/tls/https/server/tls.key 23 | #@schema/default [] 24 | allowedOrigins: 25 | - "" 26 | 27 | grpc: 28 | #@schema/doc "port is taken from ports section above" 29 | #@schema/validate format="ipv4" 30 | address: 127.0.0.1 31 | tlsCert: /etc/dex/tls/grpc/server/tls.crt 32 | tlsKey: /etc/dex/tls/grpc/server/tls.key 33 | tlsClientCA: /etc/dex/tls/grpc/ca/tls.crt 34 | 35 | #@schema/default [] 36 | connectors: 37 | - 38 | #@schema/validate enum=["github", "slack"] 39 | type: github 40 | #@schema/example "github" 41 | id: "" 42 | #@schema/example "Github" 43 | name: "" 44 | 45 | #@ def gh_config_example(): 46 | clientID: xxxxxxxxxxxxxxx 47 | clientSecret: yyyyyyyyyyyyyyyyyyyyy 48 | redirectURI: https://dex.minikube.local:5556/callback 49 | org: kubernetes 50 | #@ end 51 | 52 | #@schema/type "map" 53 | #@schema/example gh_config_example() 54 | config: {} 55 | 56 | oauth2: 57 | alwaysShowLoginScreen: false 58 | skipApprovalScreen: true 59 | 60 | expiry: 61 | #@schema/validate format="duration" 62 | signingKeys: "6h" 63 | #@schema/validate format="duration" 64 | idTokens: "24h" 65 | 66 | #@schema/default [] 67 | staticClients: 68 | - name: "" 69 | #@schema/example "example-app" 70 | id: "" 71 | #@schema/default [] 72 | redirectURIs: 73 | #@schema/example "http://192.168.42.219:31850/oauth2/callback" 74 | - "" 75 | secret: "" 76 | 77 | enablePasswordDB: true 78 | 79 | #@schema/default [] 80 | staticPasswords: 81 | - userID: "" 82 | username: "" 83 | #@schema/validate format="email" 84 | email: "" 85 | #@schema/validate regexp="^\$.+\$.+\$.+$" 86 | #@schema/example ("$2a$10$2b2cU8CPhOTaGrs1HRQuAueS7JTT5ZHsHSzYiFPm1leZck7Mc8T4W", "bcrypt hash of the string 'password'"") 87 | hash: "" 88 | 89 | frontend: 90 | #@schema/example "https://example.com/yourlogo.png" 91 | logoURL: "" 92 | -------------------------------------------------------------------------------- /logos/vmware.svg: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | -------------------------------------------------------------------------------- /logos/revng.svg: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | -------------------------------------------------------------------------------- /GOVERNANCE.md: -------------------------------------------------------------------------------- 1 | :warning: This document is under construction. 2 | 3 | # Backlog Management 4 | We use [ZenHub for backlog 5 | management](https://app.zenhub.com/workspaces/carvel-backlog-6013063a24147d0011410709) 6 | (note: ZenHub requires GitHub authorization). The backlog is configured as a 7 | single, consolidated backlog. If you're interested in only seeing a specific tool's 8 | backlog then we recommend using the filters accordingly. 9 | 10 | ## How the Backlog is Organized 11 | The below graphic is an overview of how issues flow through our backlog: 12 | ![Backlog Workflow](images/backlog-workflow.png) 13 | 14 | | Column (State) | Description | Labels Used | How does an issue progress from this column? Where does it go next? | 15 | | --- | --- | --- | --- | 16 | | New Issues | Issues to be reviewed and labeled by the Carvel maintainer team in order to determine the next steps. | * `carvel-triage` (add)
* any other categorical labels (such as `bug` or `enhancement`) | A Carvel maintainer will triage this issue to determine what’s needed for this issue. This could mean answering a question, clarifying the problem with the requester, determining product viability, or something else. While information is being collected, issues will remain in this column. Once sufficient information is collected and the issue is product viable, then the issue will move to the **Carvel Accepted** column. | 17 | | Carvel Accepted | Issues that are product viable requiring prioritization consideration. | * `carvel-triage` (remove)
* `carvel-accepted`(add) | A subset of Carvel leadership (product and engineering) will review these stories on a weekly basis to determine whether these stories should be placed in the **Prioritized Backlog** (upcoming planned work) or the **Unprioritized Backlog** (unplanned work). | 18 | | Unprioritized Backlog | Issues that have been accepted from a product perspective but are not prioritized for the Carvel maintainer team in the near future. Community members are encouraged to see if any of these issues are something they want to work on. Contributions are welcome. | | Regular reviews of the **Unprioritized Backlog** by Carvel maintainers can prompt the graduation of issues to the **Prioritized Backlog**. | 19 | | Prioritized Epics | A prioritized list of epics that Carvel maintainers are planning to work on. | | When all of the stories within an epic are completed the issue will be moved to **Done**. When all of the stories within an epic are released the issue will be moved to **Closed**. | 20 | | Prioritized Backlog | A prioritized list of issues that are planned to be worked on by Carvel maintainers. If a community member would like work on one of these, please coordinate with the maintainers in the GitHub Issue comments. | | When work on an issue begins, the person working on the issue will assign themselves to the task and move the issue to In **Progress**. | 21 | | In Progress | Issues currently being worked on by Carvel maintainers or a community member. | * `in-progress` (add) | When work on an issue is ready for review, the person working on the issue will move the issue to **Needs Review**. | 22 | | Needs Review | Issues that are ready for review. Code complete, pending feedback. | | After an issue is reviewed, the issue will either be approved and merged or it will require further changes. If the issue is approved and merged, the issue should be moved to **Done**. If the issue requires further changes, it can remain in **Needs Review** until approved. | 23 | | Done | Issues that have been approved and are waiting to be released. | | After this work is included in a published release, the issue can be moved to **Closed**. 24 | | Closed | Issues that have been released, resolved or deemed not product viable. | * `in-progress` (remove) | | 25 | 26 | -------------------------------------------------------------------------------- /proposals/ytt/001-schemas/cf-for-k8s/values.yml: -------------------------------------------------------------------------------- 1 | #@data/values 2 | --- 3 | system_namespace: cf-system 4 | workloads_namespace: cf-workloads 5 | staging_namespace: cf-workloads-staging 6 | 7 | #! your system domain, e.g., "system.cf.example.com" 8 | system_domain: 9 | #! list of app domains, e.g., ["apps.cf.example.com"] 10 | app_domains: [] 11 | 12 | #! password for admin user in plain text 13 | cf_admin_password: 14 | 15 | #! control deployment of a blobstore for CF 16 | cf_blobstore: 17 | access_key: "admin" 18 | secret_key: "" 19 | 20 | #! control optional deployment of a database for CF 21 | cf_db: 22 | admin_password: "" 23 | enabled: true 24 | 25 | #! reserved static ip for istio LoadBalancer 26 | istio_static_ip: "" 27 | 28 | images: 29 | capi: "" 30 | nginx: "" 31 | 32 | system_certificate: 33 | #! Base64-encoded certificate for the wildcard 34 | #! subdomain of the system domain (e.g., CN=*.system.cf.example.com) 35 | crt: "" 36 | #! Base64-encoded private key for the system certificate 37 | key: "" 38 | #! Base64-encoded CA certificate used to sign the system certifcate 39 | ca: "" 40 | 41 | workloads_certificate: 42 | #! Base64-encoded certificate for the wildcard 43 | #! subdomain of the system domain (e.g., CN=*.apps.cf.example.com) 44 | crt: "" 45 | #! Base64-encoded private key for the workloads certificate 46 | key: "" 47 | #! Base64-encoded CA certificate used to sign the workloads certifcate 48 | ca: "" 49 | 50 | internal_certificate: 51 | #! Base64-encoded certificate for the wildcard 52 | #! subdomain of the system domain (e.g., CN=*.cf-system.svc.cluster.local) 53 | crt: "" 54 | #! Base64-encoded private key for the internal certificate 55 | key: "" 56 | #! Base64-encoded CA certificate used to sign the internal certifcate 57 | ca: "" 58 | 59 | capi: 60 | blobstore: 61 | package_directory_key: cc-packages 62 | droplet_directory_key: cc-droplets 63 | resource_directory_key: cc-resources 64 | buildpack_directory_key: cc-buildpacks 65 | region: "''" 66 | endpoint: "http://cf-blobstore-minio.cf-blobstore.svc.cluster.local:9000" 67 | database: 68 | #! or mysql2, as needed 69 | adapter: postgres 70 | host: "" 71 | port: 5432 72 | user: cloud_controller 73 | #! DB user password in plain text 74 | password: "" 75 | name: cloud_controller 76 | 77 | uaa: 78 | #! client secret for uaa admin client in plain text 79 | admin_client_secret: "" 80 | 81 | database: 82 | #! or mysql2, as needed 83 | adapter: postgresql 84 | host: "" 85 | port: 5432 86 | user: uaa 87 | #! DB user password in plain text 88 | password: "" 89 | name: uaa 90 | #! Plain text ca certificate for tls 91 | ca_cert: "" 92 | 93 | jwt_policy: 94 | key_id: "default_jwt_policy_key" 95 | #! Plain text private key 96 | signing_key: "" 97 | 98 | encryption_key: 99 | label: "default_encryption_key" 100 | #! Plain text passphrase 101 | passphrase: "" 102 | 103 | login: 104 | service_provider: 105 | #! Plain text private key 106 | key: "" 107 | #! Plain text password 108 | key_password: "" 109 | #! Plain text certificate 110 | certificate: "" 111 | 112 | login_secret_name: "uaa-login-secret" 113 | 114 | log_cache_ca: 115 | crt: "" #! Base64-encoded ca for the log cache 116 | key: "" #! Base64-encoded private key for the cert above 117 | 118 | log_cache: 119 | crt: "" #! Base64-encoded cert for the log cache requires CN of log-cache 120 | key: "" #! Base64-encoded private key for the cert above 121 | 122 | log_cache_metrics: 123 | crt: "" #! Base64-encoded cert for the log cache metrics requires CN of log-cache-metrics 124 | key: "" #! Base64-encoded private key for the cert above 125 | 126 | log_cache_gateway: 127 | crt: "" #! Base64-encoded cert for the log cache gateway requires CN of log-cache-gateway 128 | key: "" #! Base64-encoded private key for the cert above 129 | 130 | log_cache_syslog: 131 | crt: "" #! Base64-encoded cert for the log cache syslog server requires CN of log-cache-syslog 132 | key: "" #! Base64-encoded private key for the cert above 133 | 134 | app_registry: 135 | hostname: "" 136 | repository: "" 137 | username: "" 138 | password: "" 139 | 140 | metric_proxy: 141 | ca: 142 | secret_name: "metric-proxy-ca" 143 | crt: "" 144 | key: "" 145 | cert: 146 | secret_name: "metric-proxy-cert" 147 | crt: "" 148 | key: "" 149 | -------------------------------------------------------------------------------- /proposals/README.md: -------------------------------------------------------------------------------- 1 | # Carvel Proposals 2 | This directory serves as the home for Carvel proposals. A proposal is a design 3 | document that describes a new feature for a Carvel project. The new feature 4 | can span multiple projects or introduce a new project. A proposal must be 5 | sponsored by at least one Carvel maintainer. Proposals can be worked by anyone 6 | in the community. 7 | 8 | # When to Create a Proposal 9 | The Carvel proposal process is intended for "big" or complex features. _If there 10 | is significant risk with a potential feature or track of work (such as 11 | complexity, cost to implement, product viability, etc.)_, then we recommend 12 | creating a proposal for feedback and approval. _If a potential feature is well 13 | understood and doesn't impose risk_, then we reccomend a standard GH issue to 14 | clarify the details. 15 | 16 | # Submit a Proposal 17 | To create a proposal, submit a PR to this directory under the appropriate 18 | project with a terse name (for example, `ytt/001-schemas/`). If the proposal 19 | concerns multiple projects or is intended for the entire Carvel suite then 20 | please create the proposal at the root of the `proposals` directory (for 21 | example, `./010-carvel-cli/`). 22 | 23 | In that directory, create a `README.md` containing the core proposal. Include 24 | other files (e.g. sets of example artifacts, if necessary) to help support 25 | understanding of the feature. When creating the proposal, please add a `Status: 26 | Draft` line at the top of the proposal to indicate its state. 27 | 28 | ## Proposal Template 29 | The below template is an example. Other than the high-level details (such as 30 | title, proposal status, author, and approvers), please use whichever sections 31 | make the most sense for your proposal. 32 | 33 | ```md 34 | --- 35 | title: "Writing a Proposal" 36 | authors: [ "Aaron Hurley " ] 37 | status: "draft" 38 | approvers: [ "Cari Dean " ] 39 | --- 40 | 41 | # 42 | 43 | ## Problem Statement 44 | _This is a short summary of the problem that exists, why it needs to be 45 | solved: what specific needs are being met. Compelling problem statements 46 | include concrete examples (even if only by reference). How exactly the proposal 47 | would meet those needs should be located in the "Proposal" section, not this one. 48 | The goal of this section is to help readers quickly empathize with the target users' 49 | current experience to motivate the proposed change. 50 | 51 | ## Terminology / Concepts 52 | _Define any terms or concepts that are used throughout this proposal._ 53 | 54 | ## Proposal 55 | _This is the primary content of the proposal explaining how the problem(s) will 56 | be addressed._ 57 | 58 | ### Goals and Non-goals 59 | _A short list of what the goals of this proposal are and are not._ 60 | 61 | ### Specification / Use Cases 62 | _Detailed explanation of the proposal's design._ 63 | 64 | ### Other Approaches Considered 65 | _Mention of other reasonable ways that the problem(s) 66 | could be addressed with rationale for why they were less 67 | desirable than the proposed approach._ 68 | 69 | ## Open Questions 70 | _A list of questions that need to be answered._ 71 | 72 | ## Answered Questions 73 | _A list of questions that have been answered._ 74 | ``` 75 | 76 | # Proposal Review 77 | Once a proposal PR is submitted, project maintainers will review the proposal. 78 | The goal of the review is to gain an understanding of the problem being solved 79 | and the design of the proposed solution. 80 | 81 | # Proposal States 82 | | Status | Definition | 83 | | --- | --- | 84 | | Draft | The proposal is actively being written by the proposer. | 85 | | In Review | The proposal is being reviewed by project maintainers. | 86 | | Accepted | The proposal has been accepted by the project maintainers. | 87 | | Rejected | The proposal has been rejected by the project maintainers. | 88 | 89 | # Lifecycle of a Proposal 90 | 1. Author adds a proposal by creating a PR in draft mode. (Authors can save their work until ready.) 91 | 1. When the author elaborates the proposal sufficiently to withstand critique they: 92 | 1. change the status to `in-review` and 93 | 1. mark the PR as "Ready for Review" 94 | 1. The community critiques the proposal by adding PR reviews in order to mature/converge on the proposal. 95 | 1. When the approvers reach [rough consensus](https://en.wikipedia.org/wiki/Rough_consensus), they: 96 | 1. change the status to `accepted` or `rejected`, 97 | 1. record both majority and dissenting opinions, and 98 | 1. merge the PR. 99 | -------------------------------------------------------------------------------- /proposals/ytt/001-schemas/cf-for-k8s/ytt-schema.yml: -------------------------------------------------------------------------------- 1 | #@schema/match data_values=True 2 | --- 3 | system_namespace: cf-system 4 | workloads_namespace: cf-workloads 5 | staging_namespace: cf-workloads-staging 6 | 7 | #@schema/doc "your system domain" 8 | #@schema/example "system.cf.example.com" 9 | system_domain: "" 10 | 11 | #@schema/doc "list of app domains" 12 | #@schema/example ["apps.cf.example.com"] 13 | #@schema/default [] 14 | app_domains: 15 | - "" 16 | 17 | #@schema/doc "password for admin user in plain text" 18 | cf_admin_password: "" 19 | 20 | #@schema/doc "control deployment of a blobstore for CF" 21 | cf_blobstore: 22 | access_key: "admin" 23 | secret_key: "" 24 | 25 | #@schema/doc "control optional deployment of a database for CF" 26 | #@schema/nullable 27 | #@schema/default {} 28 | cf_db: 29 | admin_password: "" 30 | 31 | #@schema/doc "reserved static ip for istio LoadBalancer" 32 | #@schame/validate min_len=0 33 | istio_static_ip: "" 34 | 35 | images: 36 | #@schame/validate min_len=0 37 | capi: "" 38 | #@schame/validate min_len=0 39 | nginx: "" 40 | 41 | system_certificate: 42 | #@schema/doc "Base64-encoded certificate for the wildcard subdomain of the system domain (e.g., CN=*.system.cf.example.com)" 43 | crt: "" 44 | #@schema/doc "Base64-encoded private key for the system certificate" 45 | key: "" 46 | #@schema/doc "Base64-encoded CA certificate used to sign the system certifcate" 47 | ca: "" 48 | 49 | workloads_certificate: 50 | #@schema/doc "Base64-encoded certificate for the wildcard subdomain of the system domain (e.g., CN=*.apps.cf.example.com)" 51 | crt: "" 52 | #@schema/doc "Base64-encoded private key for the workloads certificate" 53 | key: "" 54 | #@schema/doc "Base64-encoded CA certificate used to sign the workloads certifcate" 55 | ca: "" 56 | 57 | internal_certificate: 58 | #@schema/doc "Base64-encoded certificate for the wildcard subdomain of the system domain (e.g., CN=*.cf-system.svc.cluster.local)" 59 | crt: "" 60 | #@schema/doc "Base64-encoded private key for the internal certificate" 61 | key: "" 62 | #@schema/doc "Base64-encoded CA certificate used to sign the internal certifcate" 63 | ca: "" 64 | 65 | capi: 66 | blobstore: 67 | package_directory_key: cc-packages 68 | droplet_directory_key: cc-droplets 69 | resource_directory_key: cc-resources 70 | buildpack_directory_key: cc-buildpacks 71 | region: "''" 72 | endpoint: "http://cf-blobstore-minio.cf-blobstore.svc.cluster.local:9000" 73 | database: 74 | #@schema/validate enum=["postgres", "mysql2"] 75 | adapter: postgres 76 | #@schema/doc "Only needed to be provided when using external DB for capi" 77 | #@schema/validate min_len=0 78 | host: "" 79 | port: 5432 80 | user: cloud_controller 81 | #@schema/doc "Only needed to be provided when using external DB for capi" 82 | #@schema/validate min_len=0 83 | password: "" 84 | name: cloud_controller 85 | 86 | uaa: 87 | #@schema/doc "client secret for uaa admin client in plain text" 88 | admin_client_secret: "" 89 | 90 | database: 91 | #@schema/validate enum=["postgresql", "mysql2"] 92 | adapter: postgresql 93 | #@schema/doc "Only needed to be provided when using external DB for uaa" 94 | #@schema/validate min_len=0 95 | host: "" 96 | port: 5432 97 | user: uaa 98 | #@schema/doc "Only needed to be provided when using external DB for uaa" 99 | #@schema/validate min_len=0 100 | password: "" 101 | name: uaa 102 | #@schema/doc "Plain text ca certificate for tls" 103 | #@schame/validate min_len=0 104 | ca_cert: "" 105 | 106 | jwt_policy: 107 | key_id: "default_jwt_policy_key" 108 | #@schema/doc "Plain text private key" 109 | signing_key: "" 110 | 111 | encryption_key: 112 | label: "default_encryption_key" 113 | #@schema/doc "Plain text passphrase" 114 | passphrase: "" 115 | 116 | login: 117 | service_provider: 118 | #@schema/doc "Plain text private key" 119 | key: "" 120 | #@schema/doc "Plain text password" 121 | key_password: "" 122 | #@schema/doc "Plain text certificate" 123 | certificate: "" 124 | 125 | login_secret_name: "uaa-login-secret" 126 | 127 | log_cache_ca: 128 | #@schema/doc "Base64-encoded ca for the log cache" 129 | crt: "" 130 | #@schema/doc "Base64-encoded private key for the cert above" 131 | key: "" 132 | 133 | log_cache: 134 | #@schema/doc "Base64-encoded cert for the log cache requires CN of log-cache" 135 | crt: "" 136 | #@schema/doc "Base64-encoded private key for the cert above" 137 | key: "" 138 | 139 | log_cache_metrics: 140 | #@schema/doc "Base64-encoded cert for the log cache metrics requires CN of log-cache-metrics" 141 | crt: "" 142 | #@schema/doc "Base64-encoded private key for the cert above" 143 | key: "" 144 | 145 | log_cache_gateway: 146 | #@schema/doc "Base64-encoded cert for the log cache gateway requires CN of log-cache-gateway" 147 | crt: "" 148 | #@schema/doc "Base64-encoded private key for the cert above" 149 | key: "" 150 | 151 | log_cache_syslog: 152 | #@schema/doc "Base64-encoded cert for the log cache syslog server requires CN of log-cache-syslog" 153 | crt: "" 154 | #@schema/doc "Base64-encoded private key for the cert above" 155 | key: "" 156 | 157 | app_registry: 158 | hostname: "" 159 | repository: "" 160 | username: "" 161 | password: "" 162 | 163 | metric_proxy: 164 | ca: 165 | secret_name: "metric-proxy-ca" 166 | crt: "" 167 | key: "" 168 | cert: 169 | secret_name: "metric-proxy-cert" 170 | crt: "" 171 | key: "" 172 | -------------------------------------------------------------------------------- /proposals/carvel/001-maintain-documentation/README.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: "Maintaining Carvel Documentation" 3 | authors: [ "John Ryan " ] 4 | status: "accepted" 5 | approvers: [ "Aaron Hurley ", "Helen George ", "Vibhas Kumar "] 6 | --- 7 | 8 | # Maintaining Carvel Documentation 9 | 10 | ## Problem Statement 11 | 12 | As we evolve our tools, we want to keep our documentation up-to-date. That said, 13 | not all users will be using the latest version of our tools. 14 | 15 | Currently, the Carvel teams maintain a single version of documentation for each tool. With this scheme, 16 | - PRs to docs that describe pre-release features are held back adding complexity to that contribution process. 17 | - to determine if the description/advice is applicable to their situation, users must "caveat" parts of what they read with mentions of when the feature was introduced or changed in what version (so-called "version caveats"). 18 | - digesting version caveats risks additional cognitive load for the reader: 19 | - https://carvel.dev/kapp/docs/latest/apply-ordering/#overview 20 | - https://carvel.dev/kapp/docs/latest/config/ 21 | - https://carvel.dev/ytt/docs/latest/lang-ref-ytt-library/ 22 | - at the time of writing, there are 101 version caveats across the current document set in the Hugo-based site. 23 | 24 | **What's needed is a consistent scheme for maintaining our set of documentation that is easy to navigate, easy to search, and easy to maintain — across a range of versions of our tools.** 25 | 26 | 27 | ## Terminology / Concepts 28 | 29 | - version caveat — a note or parenthetical that indicates that the behavior of some feature changed around a specific version. 30 | 31 | ## Proposal 32 | 33 | - [Goals and Non-goals](#goals-and-non-goals) 34 | - [Specification / Use Cases](#specification--use-cases) 35 | - [Freezing a Documentation Set](#freezing-a-documentation-set) 36 | - [Use Case: Carvel Team Deprecates a Feature](#use-case-carvel-team-deprecates-a-feature) 37 | - [Use Case: Carvel Team Releases a Patch of an Existing Release](#use-case-carvel-team-releases-a-patch-of-an-existing-release) 38 | - [Use Case: User Navigates into Documentation (current version)](#use-case-user-navigates-into-documentation-current-version) 39 | - [Use Case: User Follows a Deep Link into Documentation (previous version)](#use-case-user-follows-a-deep-link-into-documentation-previous-version) 40 | - [Use Case: User Searches within Documentation](#use-case-user-searches-within-documentation) 41 | - [Consideration: Playgrounds](#consideration-playgrounds) 42 | - [Other Approaches Considered](#other-approaches-considered) 43 | - [Two Versions of Documentation](#two-versions-of-documentation) 44 | - [Multiple Live Versions of Documentation](#multiple-live-versions-of-documentation) 45 | 46 | ### Goals and Non-goals 47 | 48 | **Goals:** 49 | - make it easy for users to know what set of features are available 50 | to them, given the version of the tool they are using 51 | - optimize for the user using the current version of the tools 52 | - provide a consistent experience across all Carvel tools 53 | - keep it easy for contributors to update documentation 54 | 55 | **Non-Goals** 56 | - make it easy to determine which version a given feature was introduced. 57 | 58 | ### Specification / Use Cases 59 | 60 | Carry multiple versions of documentation, one for each [major/minor release](#use-case-carvel-team-releases-a-patch-of-an-existing-release) of the tool: 61 | - only the "latest" version and the "pre-release" versions are actively maintained 62 | - all other versions are [frozen](#freezing-a-documentation-set). 63 | 64 | Maintenance Flow: 65 | 66 | 1. **Development** — as new features are developed, the corresponding documentation goes in the 67 | `develop` directory. 68 | 2. **Release** — when a new version of the tool is released: 69 | 1. the `develop` version is renamed with the new version number (e.g. `v0.32.0`) 70 | 2. this version of the docs are configured as the "latest" version 71 | 3. this version is copied to a new `develop`. 72 | 73 | #### Freezing a Documentation Set 74 | 75 | When a version of documentation is "frozen" it means: 76 | - adorned atop every page in the docs is a highly-visible banner: 77 | 78 | ``` 79 | You are viewing documentation for [TOOL] version: v[VERSION] 80 | [TOOL] v[VERSION] documentation is no longer actively maintained. 81 | The version you are currently viewing is a static snapshot. For 82 | up-to-date documentation, see the latest version. 83 | ``` 84 | - once a version is frozen, as a breakable "rule" no edits are made to it. 85 | 86 | 87 | #### Use Case: Carvel Team Deprecates a Feature 88 | 89 | Features that are found to be made redundant or a new approach has made its presence encourage "bad" patterns of usage, they are deprecated. 90 | 91 | To indicate that a feature is marked for removal, 92 | - with the canonical description of the feature, a noticeable note should say so indicating in which version deprecation started, the rationale, and what replaces it. 93 | - in mentions / uses of the feature, a simple note (this feature was deprecated) with a link to the rationale in the canonical description. 94 | 95 | 96 | #### Use Case: Carvel Team Releases a Patch of an Existing Release 97 | 98 | On occasion — and so far, rare at that — a bugfix results in a patch release. 99 | (e.g. ytt v0.27.x). 100 | 101 | In these cases as well, we'll cut a new version of the docs. Doing so keeps the overall process of maintaining documentation straightforward: there are no special cases. 102 | 103 | 104 | #### Use Case: User Navigates into Documentation (current version) 105 | 106 | - by default the doc site displays the latest version (e.g. if `v0.32.0` is configured as "latest", that version is the default) 107 | - the doc site's search engine indexes _only_ the latest version of the docs. 108 | 109 | #### Use Case: User Follows a Deep Link into Documentation (previous version) 110 | 111 | It is often a great experience to offer a user a deep link into documentation: it places their attention directly at the relevant piece of information. 112 | 113 | However, if a deep link is baked to a specific version, it will inevitably will become out-of-date. As a result, a user may find themselves looking at stale information or now, wrong advice. 114 | 115 | To mitigate this outcome, document sets other than the latest should get full-page styling that signals its aged state (e.g. a very light red background to the whole page; or the version warning is implemented such that it never scrolls out of view) 116 | 117 | 118 | #### Use Case: User Searches within Documentation 119 | 120 | In order to avoid a whole class of undesired user experiences (such as being given search results for a different version of the tool, or the search results being cluttered with repeated results from multiple versions), it is crucial that the search engine index be faceted by version. 121 | 122 | 123 | #### Consideration: Playgrounds 124 | 125 | There will be only one version of playground(s): the latest released. 126 | 127 | 128 | ### Other Approaches Considered 129 | 130 | #### Two Versions of Documentation 131 | 132 | Another approach is to maintain exactly two versions of documentation: 133 | 1. `latest` — corresponding to the latest released version of the tool. 134 | 2. `develop` — corresponding to the version under development. 135 | 136 | In effect, this is the "One Version" approach with a tweak: 137 | - that all new work is done in the `develop` version; 138 | - when a new version of the tool is released, `latest` is replaced with `develop` 139 | 140 | This approach continues to use "version caveat"s in order to convey what features are available in what version of the tool. It's this exact attendant cognitive load that we wish to reduce for readers/users. 141 | 142 | 143 | #### Multiple Live Versions of Documentation 144 | 145 | We considered an approach where not only would we maintain the `latest`, but all previous versions of documentation to keep up their utility. 146 | 147 | This approach was quickly rejected as the maintenance effort significantly outweighs the value of servicing multiple versions. 148 | 149 | ## Open Questions 150 | 151 | **Q2** 152 | 153 | ## Answered Questions 154 | 155 | **Q1** _Can our website search engine be configured to only index the latest documentation?_ \ 156 | **A1** Yes. For example, a tag facet can be added for each version in addition to the tool name. 157 | -------------------------------------------------------------------------------- /proposals/imgpkg/001-bundles/README.md: -------------------------------------------------------------------------------- 1 | # Bundles 2 | 3 | - Status: Being written | **Being implemented** | Included in release | Rejected 4 | 5 | # Summary 6 | 7 | Support creating, relocating, and pulling images. 8 | # Concepts 9 | 10 | ## Bundle 11 | 12 | A bundle abstracts away the difference between configuration and the images 13 | referenced by it. This abstraction allows people to treat both their config 14 | and the images it depends on as a single artifact, removing the overhead 15 | associated with operations such as relocation. 16 | 17 | Bundles are simply images stored in a registry. They include: 18 | 19 | - bundle metadata (e.g. name, authors) 20 | - bundle contents - a set of files (e.g. kubernetes manifests) 21 | - list of image references that are considered to be part of a bundle (can be 22 | empty, but must be present) 23 | 24 | Key constraint: bundle image must always retain its digest when copied around. 25 | 26 | # Resources 27 | 28 | ## Bundle Directory 29 | 30 | ```yaml 31 | my-app/ 32 | .imgpkg/ <-- .imgpkg is what makes this a bundle and a max of 1 can be provided to imgpkg push 33 | bundle.yml <-- describes bundle contents and misc info 34 | images.yml <-- list of referenced images in this bundle 35 | contents/ <-- configuration files or directories referencing images in images.yml; but could be anything 36 | ``` 37 | 38 | ## Bundle YAML 39 | 40 | The Bundle YAML file is meant to contain metadata associated with the bundle. In 41 | the future, the intention is to expand this file to contain a `paths` key which 42 | will allow users to specify paths that are included in the bundle. 43 | 44 | ```yaml 45 | apiVersion: imgpkg.k14s.io/v1alpha1 46 | kind: Bundle 47 | metadata: 48 | name: my-app 49 | authors: 50 | - name: blah 51 | email: blah@blah.com 52 | websites: 53 | - url: blah.com 54 | ``` 55 | 56 | ## BundleLock 57 | 58 | A BundleLock file serves as a deterministic reference to a bundle. It will 59 | contain the original tag, as well as a url that references the bundle image by digest. 60 | 61 | ```yaml 62 | apiVersion: imgpkg.k14s.io/v1alpha1 63 | kind: BundleLock 64 | spec: 65 | image: 66 | url: docker.io/my-app@sha256: 67 | tag: v1.0 68 | ``` 69 | 70 | ## ImagesLock 71 | 72 | An ImagesLock file acts as a way to reference multiple images deterministically. 73 | When included in a bundle image, it will cause imgpkg to relocate all referenced 74 | images along with the bundle image during a copy. Users are also able to 75 | provide just an ImagesLock file when copying to support relocation of multiple 76 | generic images. Initially, the ImagesLock file should only contain references to 77 | generic images, but in the future we plan to allow it to reference bundles, 78 | enabling users to recursively relocate bundles or relocate a list of bundles. 79 | 80 | ```yaml 81 | apiVersion: imgpkg.k14s.io/v1alpha1 82 | kind: ImagesLock 83 | spec: 84 | images: 85 | - image: docker.io/my-app@sha256: 86 | annotations: # <--------- This field is to be populated by other tools 87 | 88 | - image: docker.io/another-app@sha256: 89 | annotations: 90 | 91 | ``` 92 | 93 | Note: imgpkg will require all images to be in digest reference form 94 | 95 | --- 96 | # Initial Commands 97 | 98 | ## imgpkg push ( Create a bundle or config image) 99 | 100 | This command will package a section of the local file system in to an OCI 101 | image and push it to the specified image repository. The push command will be 102 | able to push two types of images, a bundle and a generic image. The presence of 103 | a `.imgpkg` directory as a direct child of one of the arguments to `-f` will 104 | cause imgpkg to label the image's config, denoting it is a bundle. If the imgpkg 105 | directory is in any other location, if there are multiple `.imgpkg` directories, or 106 | if a `.imgpkg` exists when a `-i` flag was used, `imgpkg push` will error. 107 | 108 | In the near future, we would like to allow users to push a bundle using the `-b` 109 | flag, even if a `.imgpkg` directory does not exist, by automatically creating an 110 | empty one. 111 | 112 | Flags: 113 | * `-f` - bundle directory to create bundle from # Can be used multiple times 114 | * `-b, --bundle` - reference to push bundle to 115 | * `--lock-output` - location to write a BundleLock file to 116 | 117 | Examples: 118 | - `imgpkg push -f ... -b docker.io/klt/foo:v123 # simple case; just pack up dir and push` 119 | - `imgpkg push -f ... -b docker.io/klt/foo:v123 --lock-output bundle.lock.yml # with BundleLock output` 120 | - `imgpkg push -f ... -b docker.io/klt/foo --lock-output bundle.lock.yml # tag gets auto-incremented?` 121 | 122 | Notes: 123 | * Adds Label to image config to denote it is a bundle 124 | * Requires `.imgpkg` directory to push a bundle 125 | * If present, requires `.imgpkg` to: 126 | * be a direct child of an argument to `-f` 127 | * be a singleton 128 | * contain an images.yml file 129 | 130 | --- 131 | ## imgpkg pull ( Download and unpack the contents of a bundle to your local filesystem ) 132 | 133 | Flags: 134 | * `-o` - location to unpack the bundle directory 135 | * `-b` - reference to the bundle to unpack 136 | * `--lock` - BundleLock with a reference to a bundle to unpack (Error is ImagesLock?) 137 | * `--image` - An image to unpack 138 | 139 | Examples: 140 | - `imgpkg pull -o /tmp -b foo:v123` 141 | - `imgpkg pull -o /tmp --lock bundle.lock.yml` 142 | - `imgpkg pull -o /tmp -b foo:v123@digest` 143 | - `imgpkg pull -o /tmp -b foo@digest` 144 | - `imgpkg pull -o /tmp --image foov123` 145 | 146 | Notes: 147 | * Will rewrite bundle's images.lock.yml if images are in same repo as bundle 148 | * can be determined by a get to the repo with the digest 149 | 150 | --- 151 | ## imgpkg copy ( Copy bundles and images to various locations ) 152 | 153 | Copy is responsible for relocating artifacts. It is able to consume a variety of 154 | inputs and copy all required images to either an image repository or a tar file 155 | on the local filesystem. When relocating bundles, imgpkg will also relocate any 156 | images references by the bundle's ImagesLock file. 157 | 158 | Flags: 159 | * `--bundle` - the bundle reference we are copying (happens thickly, i.e. bundle image + all referenced images) 160 | * `--from-tar` - Tar file which contains assets to be copied to a registry 161 | * `--lock` - either an ImageLock file or BundleLock file with asset references to copy to destination 162 | * `--image` - image reference for copying generic images 163 | * `--to-repo` - the location to upload assets 164 | * `--to-tar` - location to write a tar file containing assets 165 | * `--to-tag` - the tag to upload with (if not present either existing tag will be used or random will be generated) 166 | * `--lock-output` - location to output updated lockfile. If BundleLock in, BundleLock out. If ImagesLock in, ImagesLock out. 167 | 168 | Examples: 169 | - `imgpkg copy --bundle docker.io/foo:v123 --to-repo gcr.io/foo # repo to repo thick copy without outputting an update lockfile` 170 | - `imgpkg copy --bundle docker.io/foo:v123 --to-repo gcr.io/foo --lock-output bundle.lock.yml --to-tag v124 # repo to repo copy with updated lock output and tag override` 171 | - `imgpkg copy --bundle docker.io/foo:v123 --to-tar foo.tar # write bundle contents (thickly) to a tar on the local file system` 172 | - `imgpkg copy --from-tar foo.tar --to-repo gcr.io/foo # upload bundle assets from foo.tar to remote repo foo` 173 | - `imgpkg copy --lock bundle.lock.yml --to-repo gcr.io/foo --lock-output bundle.lock.yml # thickly copy the bundle referenced by bundle.lock.yml to the repo foo (tags will be preserved)` 174 | - `imgpkg copy --image docker.io/foo:v123 --to-repo gcr.io/foo # relocate a generic image to the foo repo -- Do we want to preserve tags? It could result in collisions` 175 | 176 | Notes: 177 | * Source lock file may contain bundle or images lock contents 178 | 179 | --- 180 | # Potential Extensions 181 | 182 | ## imgpkg list ( list bundles in a repo or registry ) 183 | 184 | --- 185 | ## imgpkg init ( initialize a directory with the bundle format ) 186 | 187 | --- 188 | # Use Cases 189 | 190 | See [workflow](workflow.md) for an example E2E workflow. 191 | 192 | ## Use Case: No relocate bundle consumption 193 | 194 | Developer wants to provide a no-surprises install of a "K8s-native" app, leveraging a number of publicly available images. 195 | 196 | "no-surprises" means: 197 | 198 | * by simple inspection, user knows all images that will be used; 199 | * user knows the exact version of each image (i.e. version tag and digest); 200 | 201 | ### Bundle creator: 202 | 1. Create a bundle directory 203 | 2. `imgpkg push -f -b docker.io/klt/some-bundle:1.0.0` 204 | 205 | ### Bundle consumer: 206 | 1. `imgpkg pull -b docker.io/klt/some-bundle:1.0.0` 207 | 2. `ytt -f contents/ | kbld -f ./.imgpkg/images.yml | kapp deploy -a some-bundle -f-` 208 | 209 | **Notes:** 210 | * Producer could distribute a BundleLock file to give consumers a stronger 211 | guarantee the tag is the correct bundle 212 | 213 | --- 214 | ## Use Case: Thickly Relocated Bundles 215 | 216 | ### Bundle creator: 217 | Same as above 218 | 219 | ### Bundle consumer: 220 | 1. `imgpkg copy --bundle docker.io/klt/some-bundle:1.0.0 --to-repo internal.reg/some-bundle` (or using --bundle + --to-tar and --tar + --to-repo for air-gapped environments, but outcome is the same) 221 | 2. `imgpkg pull -b internal.reg/some-bundle:1.0.0` 222 | 3. `ytt -f contents | kbld -f ./.imgpkg/images.yml | kapp deploy -a some-bundle -f-` 223 | 224 | --- 225 | ## Use Case: Generic Relocation 226 | 227 | ### A Single Image 228 | 1. `imgpkg copy --image gcr.io/my-image --to-repo docker.io/klt --lock-output image.lock.yml` 229 | 230 | or 231 | 232 | ### Multiple Images 233 | 1. `imgpkg copy --lock images.lock.yml --to-repo docker.io/klt --lock-output relocated-images.lock.yml` 234 | -------------------------------------------------------------------------------- /proposals/kapp-controller/003-package-cr-split/README.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: "Splitting the Package CR" 3 | authors: [ "Eli Wrenn " ] 4 | status: "accepted" 5 | approvers: [ "Dmitriy Kalinin " ] 6 | --- 7 | 8 | # Splitting the Package CR 9 | 10 | ## Problem Statement 11 | Currently, the means of discovering which packages are available on a cluster 12 | can be overwhelming. Because the Package CR is versioned uniquely, when users 13 | list packages, there will be an entry for every version of software shipped. 14 | This can quickly grow to be thousands via a few mature repositories, which 15 | provides an undesirable experience of sifting through thousands of package 16 | versions to discover which packages are available on the cluster. 17 | 18 | Also, because each Package CR contains, and is versioned along with, all of the 19 | metadata fields, there will be massive amount of duplication present in 20 | sufficiently large package offerings. For example, every version of a particular 21 | package will likely have the same logo, but because logo is a field of a 22 | versioned Package CR, it will need to be duplicated for every version. This 23 | duplication will lead to unnecessary storage consumption by kapp-controller. 24 | 25 | These two issues combined provide a sub par experience for consumers of the 26 | packaging APIs. 27 | 28 | ## Terminology / Concepts 29 | 30 | Definitions and details about packaging APIs and concepts can be found in the 31 | [documentation](https://carvel.dev/kapp-controller/docs/latest/packaging/). 32 | 33 | ## Proposal 34 | 35 | ### Goals and Non-goals 36 | 37 | #### Goals 38 | - Provide a better experience for consumers trying to discover packages 39 | available on their cluster 40 | - Provide a way to reduce duplication of metadata fields 41 | 42 | ### Specification 43 | In order to achieve the above goals, we will split what is the current Package 44 | CR in to two CR's, Package and PackageVersion. 45 | 46 | #### Package CR 47 | As mentioned, in the current state, a Package CR contains both version-specific 48 | data, such as the spec section, as well as version agnostic data, such as 49 | description. For example, 50 | 51 | ```yaml 52 | apiVersion: package.carvel.dev/v1alpha1 53 | kind: Package 54 | metadata: 55 | name: fluent-bit.vmware.com.v1.5.3 56 | spec: 57 | publicName: fluent-bit.vmware.com # aka package name 58 | version: v1.5.3 # aka package version 59 | displayName: "Fluent Bit" 60 | description: "Fluent Bit is an open source and multi-platform..." 61 | template: # type of App CR 62 | spec: 63 | fetch: 64 | - imgpkgBundle: 65 | image: registry.vmware.run/tkg-fluent-bit@sha256:... 66 | template: 67 | - ytt: 68 | paths: 69 | - config 70 | - kbld: 71 | paths: 72 | - - 73 | - .imgpkg/images.yml 74 | deploy: 75 | - kapp: {} 76 | valuesSchema: 77 | # generated from templates (like ytt, helm chart, etc.) 78 | openAPIV3Schema: 79 | ... 80 | ``` 81 | 82 | In the proposed state, the Package CR will continue to exist, but will instead 83 | act as the place for authors to specify any unversioned metadata about their 84 | package, such as a logo, description, etc. A Package will remain a unique 85 | cluster scoped resource that states a package is available. For example, 86 | 87 | ```yaml 88 | apiVersion: package.carvel.dev/v1alpha1 89 | kind: Package 90 | metadata: 91 | name: fluent-bit.vmware.com 92 | # cluster scoped 93 | spec: 94 | displayName: "Fluent Bit" 95 | icon: "" 96 | shortDescription: "Fluent Bit is an open source and multi-platform..." 97 | longDescription: "..." 98 | provider: VMware 99 | maintainers: ... 100 | category: logging 101 | support: ... 102 | ``` 103 | 104 | will show users that a package named `fluent-bit.vmware.com` is available and 105 | tells them, at a high-level, any information they would need to know about it. 106 | 107 | By extracting this unversioned information, we are able to reduce the 108 | duplication within the cluster as well as provide a more straight-forward 109 | discovery experience, which will be explored more in the Use Cases section. 110 | 111 | #### PackageVersion CR 112 | The new PackageVersion CR will contain any version-specific information, such as 113 | particular resources to install, how to install them, release notes, etc. It 114 | must also be associated with a high-level package definition via a reference to 115 | the Package CR. An example of the PackageVersion CR: 116 | 117 | ```yaml 118 | apiVersion: package.carvel.dev 119 | kind: PackageVersion 120 | metadata: 121 | name: fluent-bit.vmware.com.1.5.3 122 | spec: 123 | packageName: fluent-bit.vmware.com 124 | version: 1.5.3 125 | template: # type of App CR 126 | spec: 127 | fetch: 128 | - imgpkgBundle: 129 | image: registry.vmware.run/fluent-bit@sha256:... 130 | template: 131 | - ytt: 132 | paths: 133 | - config 134 | - kbld: 135 | paths: 136 | - - 137 | - .imgpkg/images.yml 138 | deploy: 139 | - kapp: {} 140 | valuesSchema: 141 | # generated from templates (like ytt, helm chart, etc.) 142 | openAPIV3Schema: 143 | releaseNotes: | 144 | ... 145 | systemRequirements: ... 146 | license: ... 147 | ``` 148 | 149 | Because PackageVersions must reference a higher level package definition, if 150 | that package does not exist, kapp-controller will create an empty Package CR 151 | with the correct name. 152 | 153 | #### InstalledPackage CR 154 | This CR will remain largely unchanged, except for a slight update to how 155 | the desired Package is referenced. 156 | 157 | ```yaml 158 | apiVersion: install.package.carvel.dev/v1alpha1 159 | kind: InstalledPackage 160 | metadata: 161 | name: fluent-bit 162 | namespace: my-namespace 163 | spec: 164 | serviceAccountName: default-cluster-admin 165 | packageName: fluent-bit.vmware.com # No longer under packageRef key 166 | versionSelection: 167 | constraint: "1.5.3" 168 | values: 169 | - secretRef: 170 | name: ... 171 | status: 172 | PackageVersionRef: 173 | name: fluent-bit.vmware.com.1.5.3 174 | conditions: 175 | - type: ValuesSchemaCheckFailed 176 | - type: ReconcileSucceeded 177 | - type: ReconcileFailed 178 | - type: Reconciling 179 | ``` 180 | 181 | #### Use Case: Discovering Packages 182 | 183 | With the unversioned metadata split from the versioned, discovering packages 184 | becomes simpler. Users will be able to list the packages installed in the 185 | cluster and see a table consisting of a single entry for each unique package in 186 | the cluster. The table will include columns to provide some information, such as 187 | category, short descriptions, etc.: 188 | 189 | ```bash 190 | $ kubectl get packages 191 | Name Display Name Description Category 192 | fluent-bit.vmware.com Fluent Bit ... logging 193 | ... 194 | ``` 195 | 196 | To find out more details, users can run `kubectl get 197 | packages/fluent-bit.vmware.com` as they would with any other kubernetes 198 | resources. 199 | 200 | Once the desired package has been found, users can then discover versions via 201 | `kubectl get packageversions --field-selector 202 | packageName=fluent-bit.vmware.com`, which will display a table of information 203 | about the versions available: 204 | 205 | ```bash 206 | $ kubectl get packageversions --field-selector packageName=fluent-bit.vmware.com 207 | Name Version Notes Preview 208 | fluent-bit.vmware.com.1.5.3 1.5.3 ... 209 | ... 210 | ``` 211 | 212 | As with packages, users will be able to see more details about a specific 213 | PackageVersion via `kubectl get packageversions/fluent-bit.vmware.com.1.5.3 214 | -oyaml`. 215 | 216 | After the user has discovered the package version they'd like to install, the 217 | flow is the same, and they will create an InstalledPackage with the correct 218 | packageName and version constraints. 219 | 220 | #### Use Case: Installing an Instance of a Package 221 | This consumer use case remains unchanged beyond a minor restructuring of the 222 | InstalledPackage CR. 223 | 224 | #### Use Case: Authoring a Package 225 | In the proposed state, authoring a Package would also remain largely what it is 226 | today. Authors would first create their Package CR, such that it contains all 227 | the desired information about their package, and then author new PackageVersions 228 | any time a new version is ready to be shipped. Authors are also able to iterate 229 | on the Package CR as they see fit. 230 | 231 | In the case an author wants to provide a repo that adds versions, without 232 | redefining the package, they will be able to create a repository without a 233 | corresponding Package CR. Once the consumers add this repository, if the package 234 | is already available on the cluster, the new versions will simply be added, but 235 | if the Package CR is not present, kapp-controller will automatically create an 236 | empty Package CR with the correct name. 237 | 238 | Note: There are some open questions related to how these CRs will be 239 | incorporated into a repository, but these problems exist in the current state 240 | and should not block this proposal. 241 | 242 | 243 | ### Other Approaches Considered 244 | 1. Splitting metdata that is common in Package CRs, but can be unversioned, in to a 245 | PackageMetadata CR, which pacakges then reference. 246 | 247 | This approach seemed largely similar to the one described above, but would be 248 | less natural to consumers. 249 | 250 | ## Open Questions 251 | 1. How do we handle the case of two repositories both defining the same Package? 252 | 253 | ## Answered Questions 254 | 255 | 1. How do we handle the case of a repository containing PackageVersion CRs, but 256 | not the required Package CR? 257 | 258 | - The current thinking here is to have kapp-controller automatically add the 259 | needed Package CR. The CR will be empty and only created if one does not 260 | already exist on the cluster. 261 | -------------------------------------------------------------------------------- /proposals/ytt/001-schemas/cf-for-k8s/json-schema.yml: -------------------------------------------------------------------------------- 1 | $schema: http://json-schema.org/draft-07/schema 2 | type: object 3 | title: The Root Schema 4 | properties: 5 | app_domains: 6 | type: array 7 | title: The App_domains Schema 8 | default: [] 9 | capi: 10 | type: object 11 | title: The Capi Schema 12 | default: {} 13 | properties: 14 | blobstore: 15 | type: object 16 | title: The Blobstore Schema 17 | default: {} 18 | properties: 19 | endpoint: 20 | type: string 21 | title: The Endpoint Schema 22 | default: "" 23 | database: 24 | type: object 25 | title: The Database Schema 26 | default: {} 27 | properties: 28 | adapter: 29 | type: string 30 | title: The Adapter Schema 31 | default: "" 32 | host: 33 | type: string 34 | title: The Host Schema 35 | default: "" 36 | name: 37 | type: string 38 | title: The Name Schema 39 | default: "" 40 | password: 41 | type: string 42 | title: The Password Schema 43 | default: "" 44 | port: 45 | type: integer 46 | title: The Port Schema 47 | default: 0 48 | user: 49 | type: string 50 | title: The User Schema 51 | default: "" 52 | cf_admin_password: 53 | title: The Cf_admin_password Schema 54 | default: null 55 | cf_blobstore: 56 | type: object 57 | title: The Cf_blobstore Schema 58 | default: {} 59 | properties: 60 | access_key: 61 | type: string 62 | title: The Access_key Schema 63 | default: "" 64 | secret_key: 65 | type: string 66 | title: The Secret_key Schema 67 | default: "" 68 | cf_db: 69 | type: object 70 | title: The Cf_db Schema 71 | default: {} 72 | properties: 73 | admin_password: 74 | type: string 75 | title: The Admin_password Schema 76 | default: "" 77 | enabled: 78 | type: boolean 79 | title: The Enabled Schema 80 | default: false 81 | docker_registry: 82 | type: object 83 | title: The Docker_registry Schema 84 | default: {} 85 | properties: 86 | http_secret: 87 | type: string 88 | title: The Http_secret Schema 89 | default: "" 90 | doppler: 91 | type: object 92 | title: The Doppler Schema 93 | default: {} 94 | properties: 95 | tls: 96 | type: object 97 | title: The Tls Schema 98 | default: {} 99 | properties: 100 | crt: 101 | type: string 102 | title: The Crt Schema 103 | default: "" 104 | key: 105 | type: string 106 | title: The Key Schema 107 | default: "" 108 | eirini: 109 | type: object 110 | title: The Eirini Schema 111 | default: {} 112 | properties: 113 | tls: 114 | type: object 115 | title: The Tls Schema 116 | default: {} 117 | properties: 118 | crt: 119 | type: string 120 | title: The Crt Schema 121 | default: "" 122 | key: 123 | type: string 124 | title: The Key Schema 125 | default: "" 126 | images: 127 | type: object 128 | title: The Images Schema 129 | default: {} 130 | properties: 131 | capi: 132 | type: string 133 | title: The Capi Schema 134 | default: "" 135 | cfroutesync: 136 | type: string 137 | title: The Cfroutesync Schema 138 | default: "" 139 | fluent: 140 | type: string 141 | title: The Fluent Schema 142 | default: "" 143 | log_cache: 144 | type: string 145 | title: The Log_cache Schema 146 | default: "" 147 | log_cache_gateway: 148 | type: string 149 | title: The Log_cache_gateway Schema 150 | default: "" 151 | nginx: 152 | type: string 153 | title: The Nginx Schema 154 | default: "" 155 | syslog_server: 156 | type: string 157 | title: The Syslog_server Schema 158 | default: "" 159 | log_cache: 160 | type: object 161 | title: The Log_cache Schema 162 | default: {} 163 | properties: 164 | crt: 165 | type: string 166 | title: The Crt Schema 167 | default: "" 168 | key: 169 | type: string 170 | title: The Key Schema 171 | default: "" 172 | log_cache_ca: 173 | type: object 174 | title: The Log_cache_ca Schema 175 | default: {} 176 | properties: 177 | crt: 178 | type: string 179 | title: The Crt Schema 180 | default: "" 181 | key: 182 | type: string 183 | title: The Key Schema 184 | default: "" 185 | log_cache_client: 186 | type: object 187 | title: The Log_cache_client Schema 188 | default: {} 189 | properties: 190 | id: 191 | type: string 192 | title: The Id Schema 193 | default: "" 194 | secret: 195 | type: string 196 | title: The Secret Schema 197 | default: "" 198 | log_cache_gateway: 199 | type: object 200 | title: The Log_cache_gateway Schema 201 | default: {} 202 | properties: 203 | crt: 204 | type: string 205 | title: The Crt Schema 206 | default: "" 207 | key: 208 | type: string 209 | title: The Key Schema 210 | default: "" 211 | log_cache_metrics: 212 | type: object 213 | title: The Log_cache_metrics Schema 214 | default: {} 215 | properties: 216 | crt: 217 | type: string 218 | title: The Crt Schema 219 | default: "" 220 | key: 221 | type: string 222 | title: The Key Schema 223 | default: "" 224 | log_cache_syslog: 225 | type: object 226 | title: The Log_cache_syslog Schema 227 | default: {} 228 | properties: 229 | crt: 230 | type: string 231 | title: The Crt Schema 232 | default: "" 233 | key: 234 | type: string 235 | title: The Key Schema 236 | default: "" 237 | system_certificate: 238 | type: object 239 | title: The System_certificate Schema 240 | default: {} 241 | properties: 242 | ca: 243 | type: string 244 | title: The Ca Schema 245 | default: "" 246 | crt: 247 | type: string 248 | title: The Crt Schema 249 | default: "" 250 | key: 251 | type: string 252 | title: The Key Schema 253 | default: "" 254 | system_domain: 255 | title: The System_domain Schema 256 | default: null 257 | system_namespace: 258 | type: string 259 | title: The System_namespace Schema 260 | default: "" 261 | uaa: 262 | type: object 263 | title: The Uaa Schema 264 | default: {} 265 | properties: 266 | admin_client_secret: 267 | type: string 268 | title: The Admin_client_secret Schema 269 | default: "" 270 | certificate: 271 | type: object 272 | title: The Certificate Schema 273 | default: {} 274 | properties: 275 | crt: 276 | type: string 277 | title: The Crt Schema 278 | default: "" 279 | key: 280 | type: string 281 | title: The Key Schema 282 | default: "" 283 | database: 284 | type: object 285 | title: The Database Schema 286 | default: {} 287 | properties: 288 | adapter: 289 | type: string 290 | title: The Adapter Schema 291 | default: "" 292 | host: 293 | type: string 294 | title: The Host Schema 295 | default: "" 296 | name: 297 | type: string 298 | title: The Name Schema 299 | default: "" 300 | password: 301 | type: string 302 | title: The Password Schema 303 | default: "" 304 | port: 305 | type: integer 306 | title: The Port Schema 307 | default: 0 308 | user: 309 | type: string 310 | title: The User Schema 311 | default: "" 312 | encryption_key: 313 | type: object 314 | title: The Encryption_key Schema 315 | default: {} 316 | properties: 317 | label: 318 | type: string 319 | title: The Label Schema 320 | default: "" 321 | passphrase: 322 | type: string 323 | title: The Passphrase Schema 324 | default: "" 325 | jwt_policy: 326 | type: object 327 | title: The Jwt_policy Schema 328 | default: {} 329 | properties: 330 | key_id: 331 | type: string 332 | title: The Key_id Schema 333 | default: "" 334 | signing_key: 335 | type: string 336 | title: The Signing_key Schema 337 | default: "" 338 | login: 339 | type: object 340 | title: The Login Schema 341 | default: {} 342 | properties: 343 | service_provider: 344 | type: object 345 | title: The Service_provider Schema 346 | default: {} 347 | properties: 348 | certificate: 349 | type: string 350 | title: The Certificate Schema 351 | default: "" 352 | key: 353 | type: string 354 | title: The Key Schema 355 | default: "" 356 | key_password: 357 | type: string 358 | title: The Key_password Schema 359 | default: "" 360 | login_secret_name: 361 | type: string 362 | title: The Login_secret_name Schema 363 | default: "" 364 | workloads_namespace: 365 | type: string 366 | title: The Workloads_namespace Schema 367 | default: "" 368 | -------------------------------------------------------------------------------- /proposals/imgpkg/002-recursive-bundles/README.md: -------------------------------------------------------------------------------- 1 | --- 2 | tags: recursive bundles, imgpkg 3 | --- 4 | # imgpkg Recursive Bundles Proposal 5 | 6 | - Proposal Status: Draft, **In Review**, Accepted, Rejected 7 | 8 | 9 | # Table of Contents 10 | 11 | [TOC] 12 | 13 | --- 14 | 15 | # Recursive Bundles Proposal 16 | 17 | ## Use Case 18 | 19 | **As** A Software Packager 20 | 21 | **I want to** Create a bundle to distribute the multiple applications that are part of a deployment 22 | 23 | **So that** Application consumers can retrieve all needed software from a single place 24 | 25 | ### Goals 26 | - Create a UX that is suitable to be used by other higher level applications 27 | - Create a solution that could be scalable 28 | 29 | ### Anti Goals 30 | - Be prescriptive about the implementation details 31 | 32 | 33 | ### Current Pain Points 34 | 35 | With today's approach, how might an Application consumer get multiple applications that are part of a single deployment? 36 | - Single Bundle with all the applications 37 | 1. The Packager creates an `images.yml` with all the individual OCI Images for each application 38 | 2. The Packager collects from the Application Teams all the needed configuration for each application 39 | 3. The Packager is responsible for organizing the configuration for all the Applications 40 | 4. The Packager creates a single bundle. 41 | 42 | This approach is complicated to maintain in the long run for the Packager. In case an Application Team creates a new update on the application, this process would have to be repeated by the Packager. This process can become a bottleneck because the Packager would need to have a deep understanding of each application and the configuration needs. 43 | - Multiple bundles: 44 | 1. Each Application Team would generate their Bundle with their application 45 | 2. The Packager collects all the Bundles SHA 46 | 3. The Packager creates a document with all the Bundle SHA 47 | 4. The Packager creates overlay's and adds some utilities to use during installation 48 | 5. The Packager generates a Bundle with these artifacts 49 | 6. The Consumer retrieves the Bundle generated by the Packager 50 | 7. The Consumer retrieves every other Bundle that is required 51 | 52 | This approach has challenges for both Packager and Consumer 53 | The Consumer would have to download multiple separate bundles and follow the instructions from the Packager to install all the applications 54 | The Packager would have to collect the information about the applications and generate an installation guide for the Consumer. 55 | 56 | ### Mitigate Pain Points 57 | 58 | In this proposal, we adapt `imgpkg` to try to mitigate some of the current pains. 59 | Given that `imgpkg` already supports [bundles](https://carvel.dev/imgpkg/docs/latest/resources/#bundle) with a simple extension that allows bundles to contain other bundles can be created to help with the problem. 60 | 61 | The workflow that this change enables is the following: 62 | 1. Each Application Team generates a bundle with all the needed images and the configuration 63 | 2. The Packager would collect the applications Image Bundle SHA's for all applications 64 | 3. The Packager can provide some overlay's for the configuration and add some utilities that are used during installation 65 | 4. Create a Bundle that contains all application Image Bundles artifacts generated in the previous point 66 | 5. The Application Consumer would be given a single Bundle Image that can be downloaded and consumed 67 | 68 | The benefits of this approach versus the current state are 69 | - Each Application Team manages their application's Bundle. The team that builds the application will manage all the requirements 70 | - The Packager just needs to collect the Bundle SHA for each application 71 | - The Consumer can get all the needed resources from a single Bundle to install all the applications needed 72 | 73 | # Proposed changes 74 | 75 | In this section, the proposal will only talk about parts that will be changed based on the Recursive Bundle solution 76 | 77 | ## Pushing a bundle 78 | 79 | The major proposed change in this section is to remove the validation done today that checks if the bundle being pushed contains bundles in the `.imgpkg/images.yml` Lock file. 80 | 81 | ### Required changes 82 | 83 | #### Validation removal 84 | 85 | Today we validate to ensure that when we are pushing a bundle, the images associated with it are not bundles. 86 | The error message that is shown read `Error: Expected image lock to not contain bundle reference:` 87 | 88 | As part of this proposal, this check no longer will be done. 89 | 90 | #### Ignore bundle specific folder 91 | 92 | When a bundle is pulled, and contain recursive bundles, the folder `bundles` is created inside the `.imgpkg/` folder. 93 | As part of this proposal, we should ensure that if a user tries to create a bundle and it contains a folder called `.imgpkg/bundles` 94 | `imgpkg` should ignore the folder and provide the following message 95 | 96 | ``` 97 | Warning: Ignoring ".imgpkg/bundles" directory (directory is only used for keeping pulled dependent bundles) 98 | ``` 99 | 100 | ### Example 101 | 102 | Assuming the provided example in [here](https://github.com/k14s/design-docs/tree/002-recursive-bundles/imgpkg/002-recursive-bundles/examples) 103 | 104 | ```= 105 | $ imgpkg push -b ghcr.io/k14s/design-docs/simple-app-bundle -f examples/bundle-1 106 | dir: . 107 | dir: .imgpkg 108 | file: .imgpkg/bundle.yml 109 | file: .imgpkg/images.yml 110 | file: Readme.md 111 | file: config.yml 112 | Pushed 'ghcr.io/k14s/design-docs/simple-app-bundle@sha256:d211dd700949154e429d28661d01c99d53a38af0d5275842ccbf0bf6dbef8ca4' 113 | Succeeded 114 | ``` 115 | 116 | ## Copy a bundle 117 | 118 | ### From Repository to Tar 119 | 120 | #### Required changes 121 | 122 | ##### Retrieve all images for all bundles 123 | 124 | The most significant change in this operation is that when we copy the images `imgpkg` will have to traverse all the bundles to collect all the images to copy. 125 | 126 | ##### Layer deduped on disk 127 | 128 | When creating the tar file in disk `imgpkg` need to ensure that we do not store duplicated layers in disk 129 | 130 | #### Example 131 | 132 | Given that we create a bundle using the command 133 | 134 | `imgpkg push -b ghcr.io/k14s/design-docs/simple-app-install-package -f imgpkg/002-recursive-bundles/examples/recursive-bundle` 135 | 136 | ```= 137 | imgpkg copy -b ghcr.io/k14s/design-docs/simple-app-install-package --to-tar recursive-bundle.tar 138 | copy | exporting 4 images... 139 | copy | will export ghcr.io/k14s/design-docs/simple-app-bundle@sha256:d211dd700949154e429d28661d01c99d53a38af0d5275842ccbf0bf6dbef8ca4 140 | copy | will export ghcr.io/k14s/design-docs/simple-app@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0 141 | copy | will export ghcr.io/k14s/design-docs/simple-app-install-package@sha256:77c97e82306fb2a616da0a78796db039aa76e5bac1508954d40c0c10c073e035 142 | copy | will export ghcr.io/k14s/design-docs/utilities@sha256:47ae428a887c41ba0aedf87d560eb305a8aa522ffb80ac1c96a37b16df038e0f 143 | copy | exported 4 images 144 | copy | writing layers... 145 | copy | done: file 'manifest.json' (35.107µs) 146 | copy | done: file 'sha256-81fc6f37c9774541136e6113d899c215151496f4cf91c89c056783d2feb5ae0d.tar.gz' (46.817µs) 147 | copy | done: file 'sha256-9abb11371e7e53b5c33da086ea50dabb5d4cdd280be7d489169374b0188feab1.tar.gz' (99.194µs) 148 | copy | done: file 'sha256-87bf2c587b3315143cd05df7bd24d4e608ddb59f8c62110fe1b579fb817a2917.tar.gz' (119.416µs) 149 | copy | done: file 'sha256-8ece9ac45f2b7228b2ed95e9f407b4f0dc2ac74f93c62ff1156f24c53042ba54.tar.gz' (370.872834ms) 150 | Succeeded 151 | 152 | ``` 153 | 154 | ### From Tar to Repository 155 | 156 | #### Required changes 157 | 158 | ##### Retrieve all images for all bundles 159 | 160 | The most significant change in this operation is that when we copy the images `imgpkg` will have to traverse all the bundles to collect all the images to copy. 161 | 162 | #### Example 163 | 164 | Given that we create a bundle using the command 165 | `imgpkg push -b ghcr.io/k14s/design-docs/simple-app-install-package -f imgpkg/002-recursive-bundles/examples/recursive-bundle` 166 | 167 | Followed by 168 | `imgpkg copy -b ghcr.io/k14s/design-docs/simple-app-install-package --to-tar recursive-bundle.tar` 169 | 170 | ```= 171 | imgpkg copy --tar recursive-bundle.tar --to-repo some-other.registry.io/recursive-bundle 172 | copy | importing 4 images... 173 | copy | importing ghcr.io/k14s/design-docs/simple-app@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0 -> some-other.registry.io/recursive-bundle@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0... 174 | copy | importing ghcr.io/k14s/design-docs/utilities@sha256:47ae428a887c41ba0aedf87d560eb305a8aa522ffb80ac1c96a37b16df038e0f -> some-other.registry.io/recursive-bundle@sha256:47ae428a887c41ba0aedf87d560eb305a8aa522ffb80ac1c96a37b16df038e0f... 175 | copy | importing ghcr.io/k14s/design-docs/simple-app-bundle@sha256:d211dd700949154e429d28661d01c99d53a38af0d5275842ccbf0bf6dbef8ca4 -> some-other.registry.io/recursive-bundle@sha256:d211dd700949154e429d28661d01c99d53a38af0d5275842ccbf0bf6dbef8ca4... 176 | copy | importing ghcr.io/k14s/design-docs/simple-app-install-package@sha256:77c97e82306fb2a616da0a78796db039aa76e5bac1508954d40c0c10c073e035 -> some-other.registry.io/recursive-bundle@sha256:77c97e82306fb2a616da0a78796db039aa76e5bac1508954d40c0c10c073e035... 177 | copy | imported 4 images 178 | Succeeded 179 | ``` 180 | 181 | ### From Repository to Repository 182 | 183 | #### Required changes 184 | 185 | ##### Retrieve all images for all bundles 186 | 187 | The most significant change in this operation is that when we copy the images `imgpkg` will have to traverse all the bundles to collect all the images to copy. 188 | 189 | ##### Bundles/Images deduped 190 | 191 | When copying the images/bundles ensure that `imgpkg` does not try to copy the same image multiple times 192 | 193 | #### Example 194 | 195 | Given that we create a bundle using the command 196 | 197 | `imgpkg push -b ghcr.io/k14s/design-docs/simple-app-install-package -f imgpkg/002-recursive-bundles/examples/recursive-bundle` 198 | 199 | ```= 200 | imgpkg copy -b ghcr.io/k14s/design-docs/simple-app-install-package --to-repo some-other.registry.io/recursive-bundle 201 | copy | exporting 4 images... 202 | copy | will export ghcr.io/k14s/design-docs/simple-app@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0 203 | copy | will export ghcr.io/k14s/design-docs/utilities@sha256:47ae428a887c41ba0aedf87d560eb305a8aa522ffb80ac1c96a37b16df038e0f 204 | copy | will export ghcr.io/k14s/design-docs/simple-app-bundle@sha256:d211dd700949154e429d28661d01c99d53a38af0d5275842ccbf0bf6dbef8ca4 205 | copy | will export ghcr.io/k14s/design-docs/simple-app-install-package@sha256:77c97e82306fb2a616da0a78796db039aa76e5bac1508954d40c0c10c073e035 206 | copy | exported 4 images 207 | copy | importing 4 images... 208 | copy | importing ghcr.io/k14s/design-docs/simple-app@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0 -> some-other.registry.io/recursive-bundle@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0... 209 | copy | importing ghcr.io/k14s/design-docs/utilities@sha256:ghcr.io/k14s/design-docs/utilities@sha256:47ae428a887c41ba0aedf87d560eb305a8aa522ffb80ac1c96a37b16df038e0f -> some-other.registry.io/recursive-bundle@sha256:ghcr.io/k14s/design-docs/utilities@sha256:47ae428a887c41ba0aedf87d560eb305a8aa522ffb80ac1c96a37b16df038e0f... 210 | copy | importing ghcr.io/k14s/design-docs/simple-app-bundle@sha256:d211dd700949154e429d28661d01c99d53a38af0d5275842ccbf0bf6dbef8ca4 -> some-other.registry.io/recursive-bundle@sha256:d211dd700949154e429d28661d01c99d53a38af0d5275842ccbf0bf6dbef8ca4... 211 | copy | importing ghcr.io/k14s/design-docs/simple-app-install-package@sha256:77c97e82306fb2a616da0a78796db039aa76e5bac1508954d40c0c10c073e035 -> some-other.registry.io/recursive-bundle@sha256:77c97e82306fb2a616da0a78796db039aa76e5bac1508954d40c0c10c073e035... 212 | copy | imported 4 images 213 | Succeeded 214 | ``` 215 | 216 | ## Pull a bundle 217 | 218 | ### Required changes 219 | 220 | #### Change folder structure 221 | 222 | When using the `pull` command without the `-r` flag the behavior is not changed. Nothing will be dowloaded from the nested bundles. 223 | 224 | When downloading a bundle that contains other bundles to disk using the `pull -r` command will work as currently, except for that it will download the nested bundles into a hidden folder called `.imgpkg/bundles`. 225 | 226 | ##### Folder structure 227 | 228 | The structure of the output folder will be 229 | ``` 230 | $ tree -a recursive-bundle 231 | . 232 | ├── .imgpkg 233 | | ├── bundles 234 | | │ ├── sha256-{SHA Of the First Nested Bundle} 235 | | │   │   ├── .imgpkg 236 | | │   │   │   ├── bundle.yml 237 | | │   │   │   └── images.yml 238 | | │   │   └── config2.yml 239 | | │   └── sha256-{SHA Of the Second Nested Bundle} 240 | | │      ├── .imgpkg 241 | | │      │   ├── bundle.yml 242 | | │      │   └── images.yml 243 | | │   └── config1.yml 244 | │   ├── bundle.yml 245 | │   └── images.yml 246 | └── config.yml 247 | 248 | 7 directories, 9 files 249 | ``` 250 | 251 | The folder name will be the sha256-{SHA} where SHA is the SHA256 of the bundle OCI Image. The mapping between the folder names and the origin images can be found in the `.imgpkg/images.yml` of the bundle that included this bundle 252 | 253 | The decision to have this folder inside the `.imgpkg` folder was taken to minimize the footprint that `imgpkg` leaves in the file system. 254 | 255 | ###### Cyclic nesting 256 | 257 | To ensure that there is not cyclic nesting in the disk `imgpkg` will flatten the bundle structure to 1 level. 258 | ![](https://i.imgur.com/zSlnzg7.png) 259 | In the image above Bundle 1, Bundle 2 and, Bundle 3 will be all in a single level inside the `bundles` folder. 260 | 261 | **Caveat:** As per our goals on this document, this proposal tries to cater to the UX experience for other applications to use the output of the `pull` command. In this iteration, it might be a little bit more complicated for a human to easily follow the structure, but in the future, there might so other features or tooling that could help humans better visualize the bundle contents. 262 | 263 | 264 | ### Example with -r 265 | 266 | Given that we create a bundle using the command 267 | `imgpkg push -r -b ghcr.io/k14s/design-docs/simple-app-install-package -f imgpkg/002-recursive-bundles/examples/recursive-bundle` 268 | 269 | ```= 270 | $ imgpkg pull -r -b ghcr.io/k14s/design-docs/simple-app-install-package -o /tmp/recursive-bundle 271 | 272 | Pulling bundle 'ghcr.io/k14s/design-docs/simple-app-install-package@sha256:77c97e82306fb2a616da0a78796db039aa76e5bac1508954d40c0c10c073e035' 273 | Bundle Layers 274 | Extracting layer 'sha256:87bf2c587b3315143cd05df7bd24d4e608ddb59f8c62110fe1b579fb817a2917' (1/1) 275 | 276 | Nested bundles 277 | Pulling Bundle 'ghcr.io/k14s/design-docs/simple-app-install-package@sha256:d211dd700949154e429d28661d01c99d53a38af0d5275842ccbf0bf6dbef8ca4' (1/2) 278 | Extracting layer 'sha256:81fc6f37c9774541136e6113d899c215151496f4cf91c89c056783d2feb5ae0d' (1/1) 279 | Found 1 Bundle packaged 280 | 281 | Pulling Nested Bundle 'ghcr.io/k14s/design-docs/simple-app-install-package@sha256:47ae428a887c41ba0aedf87d560eb305a8aa522ffb80ac1c96a37b16df038e0f' (1/1) 282 | Extracting layer 'sha256:9abb11371e7e53b5c33da086ea50dabb5d4cdd280be7d489169374b0188feab1' (1/1) 283 | 284 | Pulling Nested Bundle 'ghcr.io/k14s/design-docs/simple-app-install-package@sha256:47ae428a887c41ba0aedf87d560eb305a8aa522ffb80ac1c96a37b16df038e0f' (2/2) 285 | Skipped, already downloaded 286 | 287 | Locating image lock file images... 288 | The bundle repo (ghcr.io/k14s/design-docs/simple-app-install-package) is hosting every image specified in the bundle's Images Lock file (.imgpkg/images.yml) 289 | 290 | Updating all images in the ImagesLock file: pull-tmp/.imgpkg/images.yml 291 | + Changing all image registry/repository references in pull-tmp/.imgpkg/images.yml to ghcr.io/k14s/design-docs/simple-app-install-package 292 | ``` 293 | 294 | ### Example without -r 295 | 296 | Given that we create a bundle using the command 297 | `imgpkg push -b ghcr.io/k14s/design-docs/simple-app-install-package -f imgpkg/002-recursive-bundles/examples/recursive-bundle` 298 | 299 | ```= 300 | $ imgpkg pull -b ghcr.io/k14s/design-docs/simple-app-install-package -o /tmp/recursive-bundle 301 | 302 | Pulling bundle 'ghcr.io/k14s/design-docs/simple-app-install-package@sha256:77c97e82306fb2a616da0a78796db039aa76e5bac1508954d40c0c10c073e035' 303 | Bundle Layers 304 | Extracting layer 'sha256:87bf2c587b3315143cd05df7bd24d4e608ddb59f8c62110fe1b579fb817a2917' (1/1) 305 | 306 | Nested bundles 307 | Bundle 'ghcr.io/k14s/design-docs/simple-app-install-package@sha256:d211dd700949154e429d28661d01c99d53a38af0d5275842ccbf0bf6dbef8ca4' (1/2) 308 | Bundle 'ghcr.io/k14s/design-docs/simple-app-install-package@sha256:47ae428a887c41ba0aedf87d560eb305a8aa522ffb80ac1c96a37b16df038e0f' (2/2) 309 | 310 | Locating image lock file images... 311 | The bundle repo (ghcr.io/k14s/design-docs/simple-app-install-package) is hosting every image specified in the bundle's Images Lock file (.imgpkg/images.yml) 312 | 313 | Updating all images in the ImagesLock file: pull-tmp/.imgpkg/images.yml 314 | + Changing all image registry/repository references in pull-tmp/.imgpkg/images.yml to ghcr.io/k14s/design-docs/simple-app-install-package 315 | ``` 316 | 317 | ## List Images in Bundle 318 | 319 | *This feature is a nice to have* 320 | 321 | Enable the users of `imgpkg` to understand, without pulling the bundle, what images are part of the bundle. 322 | 323 | ### Proposed change 324 | 325 | Create a new command that could provide the user with information about the contents of a bundle 326 | 327 | ```= 328 | $ imgpkg info -b ghcr.io/k14s/design-docs/simple-app-install-package 329 | 330 | Images: 331 | - ghcr.io/k14s/design-docs/simple-app-install-package@sha256:d211dd700949154e429d28661d01c99d53a38af0d5275842ccbf0bf6dbef8ca4 (Bundle) 332 | Images: 333 | - ghcr.io/k14s/design-docs/simple-app-install-package@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0 334 | Annotations: 335 | kbld.carvel.dev/id: my.registry.io/simple-application 336 | 337 | - ghcr.io/k14s/design-docs/simple-app-install-package@sha256:47ae428a887c41ba0aedf87d560eb305a8aa522ffb80ac1c96a37b16df038e0f 338 | - ghcr.io/k14s/design-docs/simple-app-install-package@sha256:47ae428a887c41ba0aedf87d560eb305a8aa522ffb80ac1c96a37b16df038e0f 339 | ``` 340 | -------------------------------------------------------------------------------- /proposals/ytt/001-schemas/cf-for-k8s/json-schema.json: -------------------------------------------------------------------------------- 1 | { 2 | "$schema": "http://json-schema.org/draft-07/schema", 3 | "type": "object", 4 | "title": "The Root Schema", 5 | "properties": { 6 | "app_domains": { 7 | "type": "array", 8 | "title": "The App_domains Schema", 9 | "default": [] 10 | }, 11 | "capi": { 12 | "type": "object", 13 | "title": "The Capi Schema", 14 | "default": {}, 15 | "properties": { 16 | "blobstore": { 17 | "type": "object", 18 | "title": "The Blobstore Schema", 19 | "default": {}, 20 | "properties": { 21 | "endpoint": { 22 | "type": "string", 23 | "title": "The Endpoint Schema", 24 | "default": "" 25 | } 26 | } 27 | }, 28 | "database": { 29 | "type": "object", 30 | "title": "The Database Schema", 31 | "default": {}, 32 | "properties": { 33 | "adapter": { 34 | "type": "string", 35 | "title": "The Adapter Schema", 36 | "default": "" 37 | }, 38 | "host": { 39 | "type": "string", 40 | "title": "The Host Schema", 41 | "default": "" 42 | }, 43 | "name": { 44 | "type": "string", 45 | "title": "The Name Schema", 46 | "default": "" 47 | }, 48 | "password": { 49 | "type": "string", 50 | "title": "The Password Schema", 51 | "default": "" 52 | }, 53 | "port": { 54 | "type": "integer", 55 | "title": "The Port Schema", 56 | "default": 0 57 | }, 58 | "user": { 59 | "type": "string", 60 | "title": "The User Schema", 61 | "default": "" 62 | } 63 | } 64 | } 65 | } 66 | }, 67 | "cf_admin_password": { 68 | "title": "The Cf_admin_password Schema", 69 | "default": null 70 | }, 71 | "cf_blobstore": { 72 | "type": "object", 73 | "title": "The Cf_blobstore Schema", 74 | "default": {}, 75 | "properties": { 76 | "access_key": { 77 | "type": "string", 78 | "title": "The Access_key Schema", 79 | "default": "" 80 | }, 81 | "secret_key": { 82 | "type": "string", 83 | "title": "The Secret_key Schema", 84 | "default": "" 85 | } 86 | } 87 | }, 88 | "cf_db": { 89 | "type": "object", 90 | "title": "The Cf_db Schema", 91 | "default": {}, 92 | "properties": { 93 | "admin_password": { 94 | "type": "string", 95 | "title": "The Admin_password Schema", 96 | "default": "" 97 | }, 98 | "enabled": { 99 | "type": "boolean", 100 | "title": "The Enabled Schema", 101 | "default": false 102 | } 103 | } 104 | }, 105 | "docker_registry": { 106 | "type": "object", 107 | "title": "The Docker_registry Schema", 108 | "default": {}, 109 | "properties": { 110 | "http_secret": { 111 | "type": "string", 112 | "title": "The Http_secret Schema", 113 | "default": "" 114 | } 115 | } 116 | }, 117 | "doppler": { 118 | "type": "object", 119 | "title": "The Doppler Schema", 120 | "default": {}, 121 | "properties": { 122 | "tls": { 123 | "type": "object", 124 | "title": "The Tls Schema", 125 | "default": {}, 126 | "properties": { 127 | "crt": { 128 | "type": "string", 129 | "title": "The Crt Schema", 130 | "default": "" 131 | }, 132 | "key": { 133 | "type": "string", 134 | "title": "The Key Schema", 135 | "default": "" 136 | } 137 | } 138 | } 139 | } 140 | }, 141 | "eirini": { 142 | "type": "object", 143 | "title": "The Eirini Schema", 144 | "default": {}, 145 | "properties": { 146 | "tls": { 147 | "type": "object", 148 | "title": "The Tls Schema", 149 | "default": {}, 150 | "properties": { 151 | "crt": { 152 | "type": "string", 153 | "title": "The Crt Schema", 154 | "default": "" 155 | }, 156 | "key": { 157 | "type": "string", 158 | "title": "The Key Schema", 159 | "default": "" 160 | } 161 | } 162 | } 163 | } 164 | }, 165 | "images": { 166 | "type": "object", 167 | "title": "The Images Schema", 168 | "default": {}, 169 | "properties": { 170 | "capi": { 171 | "type": "string", 172 | "title": "The Capi Schema", 173 | "default": "" 174 | }, 175 | "cfroutesync": { 176 | "type": "string", 177 | "title": "The Cfroutesync Schema", 178 | "default": "" 179 | }, 180 | "fluent": { 181 | "type": "string", 182 | "title": "The Fluent Schema", 183 | "default": "" 184 | }, 185 | "log_cache": { 186 | "type": "string", 187 | "title": "The Log_cache Schema", 188 | "default": "" 189 | }, 190 | "log_cache_gateway": { 191 | "type": "string", 192 | "title": "The Log_cache_gateway Schema", 193 | "default": "" 194 | }, 195 | "nginx": { 196 | "type": "string", 197 | "title": "The Nginx Schema", 198 | "default": "" 199 | }, 200 | "syslog_server": { 201 | "type": "string", 202 | "title": "The Syslog_server Schema", 203 | "default": "" 204 | } 205 | } 206 | }, 207 | "log_cache": { 208 | "type": "object", 209 | "title": "The Log_cache Schema", 210 | "default": {}, 211 | "properties": { 212 | "crt": { 213 | "type": "string", 214 | "title": "The Crt Schema", 215 | "default": "" 216 | }, 217 | "key": { 218 | "type": "string", 219 | "title": "The Key Schema", 220 | "default": "" 221 | } 222 | } 223 | }, 224 | "log_cache_ca": { 225 | "type": "object", 226 | "title": "The Log_cache_ca Schema", 227 | "default": {}, 228 | "properties": { 229 | "crt": { 230 | "type": "string", 231 | "title": "The Crt Schema", 232 | "default": "" 233 | }, 234 | "key": { 235 | "type": "string", 236 | "title": "The Key Schema", 237 | "default": "" 238 | } 239 | } 240 | }, 241 | "log_cache_client": { 242 | "type": "object", 243 | "title": "The Log_cache_client Schema", 244 | "default": {}, 245 | "properties": { 246 | "id": { 247 | "type": "string", 248 | "title": "The Id Schema", 249 | "default": "" 250 | }, 251 | "secret": { 252 | "type": "string", 253 | "title": "The Secret Schema", 254 | "default": "" 255 | } 256 | } 257 | }, 258 | "log_cache_gateway": { 259 | "type": "object", 260 | "title": "The Log_cache_gateway Schema", 261 | "default": {}, 262 | "properties": { 263 | "crt": { 264 | "type": "string", 265 | "title": "The Crt Schema", 266 | "default": "" 267 | }, 268 | "key": { 269 | "type": "string", 270 | "title": "The Key Schema", 271 | "default": "" 272 | } 273 | } 274 | }, 275 | "log_cache_metrics": { 276 | "type": "object", 277 | "title": "The Log_cache_metrics Schema", 278 | "default": {}, 279 | "properties": { 280 | "crt": { 281 | "type": "string", 282 | "title": "The Crt Schema", 283 | "default": "" 284 | }, 285 | "key": { 286 | "type": "string", 287 | "title": "The Key Schema", 288 | "default": "" 289 | } 290 | } 291 | }, 292 | "log_cache_syslog": { 293 | "type": "object", 294 | "title": "The Log_cache_syslog Schema", 295 | "default": {}, 296 | "properties": { 297 | "crt": { 298 | "type": "string", 299 | "title": "The Crt Schema", 300 | "default": "" 301 | }, 302 | "key": { 303 | "type": "string", 304 | "title": "The Key Schema", 305 | "default": "" 306 | } 307 | } 308 | }, 309 | "system_certificate": { 310 | "type": "object", 311 | "title": "The System_certificate Schema", 312 | "default": {}, 313 | "properties": { 314 | "ca": { 315 | "type": "string", 316 | "title": "The Ca Schema", 317 | "default": "" 318 | }, 319 | "crt": { 320 | "type": "string", 321 | "title": "The Crt Schema", 322 | "default": "" 323 | }, 324 | "key": { 325 | "type": "string", 326 | "title": "The Key Schema", 327 | "default": "" 328 | } 329 | } 330 | }, 331 | "system_domain": { 332 | "title": "The System_domain Schema", 333 | "default": null 334 | }, 335 | "system_namespace": { 336 | "type": "string", 337 | "title": "The System_namespace Schema", 338 | "default": "" 339 | }, 340 | "uaa": { 341 | "type": "object", 342 | "title": "The Uaa Schema", 343 | "default": {}, 344 | "properties": { 345 | "admin_client_secret": { 346 | "type": "string", 347 | "title": "The Admin_client_secret Schema", 348 | "default": "" 349 | }, 350 | "certificate": { 351 | "type": "object", 352 | "title": "The Certificate Schema", 353 | "default": {}, 354 | "properties": { 355 | "crt": { 356 | "type": "string", 357 | "title": "The Crt Schema", 358 | "default": "" 359 | }, 360 | "key": { 361 | "type": "string", 362 | "title": "The Key Schema", 363 | "default": "" 364 | } 365 | } 366 | }, 367 | "database": { 368 | "type": "object", 369 | "title": "The Database Schema", 370 | "default": {}, 371 | "properties": { 372 | "adapter": { 373 | "type": "string", 374 | "title": "The Adapter Schema", 375 | "default": "" 376 | }, 377 | "host": { 378 | "type": "string", 379 | "title": "The Host Schema", 380 | "default": "" 381 | }, 382 | "name": { 383 | "type": "string", 384 | "title": "The Name Schema", 385 | "default": "" 386 | }, 387 | "password": { 388 | "type": "string", 389 | "title": "The Password Schema", 390 | "default": "" 391 | }, 392 | "port": { 393 | "type": "integer", 394 | "title": "The Port Schema", 395 | "default": 0 396 | }, 397 | "user": { 398 | "type": "string", 399 | "title": "The User Schema", 400 | "default": "" 401 | } 402 | } 403 | }, 404 | "encryption_key": { 405 | "type": "object", 406 | "title": "The Encryption_key Schema", 407 | "default": {}, 408 | "properties": { 409 | "label": { 410 | "type": "string", 411 | "title": "The Label Schema", 412 | "default": "" 413 | }, 414 | "passphrase": { 415 | "type": "string", 416 | "title": "The Passphrase Schema", 417 | "default": "" 418 | } 419 | } 420 | }, 421 | "jwt_policy": { 422 | "type": "object", 423 | "title": "The Jwt_policy Schema", 424 | "default": {}, 425 | "properties": { 426 | "key_id": { 427 | "type": "string", 428 | "title": "The Key_id Schema", 429 | "default": "" 430 | }, 431 | "signing_key": { 432 | "type": "string", 433 | "title": "The Signing_key Schema", 434 | "default": "" 435 | } 436 | } 437 | }, 438 | "login": { 439 | "type": "object", 440 | "title": "The Login Schema", 441 | "default": {}, 442 | "properties": { 443 | "service_provider": { 444 | "type": "object", 445 | "title": "The Service_provider Schema", 446 | "default": {}, 447 | "properties": { 448 | "certificate": { 449 | "type": "string", 450 | "title": "The Certificate Schema", 451 | "default": "" 452 | }, 453 | "key": { 454 | "type": "string", 455 | "title": "The Key Schema", 456 | "default": "" 457 | }, 458 | "key_password": { 459 | "type": "string", 460 | "title": "The Key_password Schema", 461 | "default": "" 462 | } 463 | } 464 | } 465 | } 466 | }, 467 | "login_secret_name": { 468 | "type": "string", 469 | "title": "The Login_secret_name Schema", 470 | "default": "" 471 | } 472 | } 473 | }, 474 | "workloads_namespace": { 475 | "type": "string", 476 | "title": "The Workloads_namespace Schema", 477 | "default": "" 478 | } 479 | } 480 | } -------------------------------------------------------------------------------- /proposals/ytt/002-raw-data-values/README.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: Data Values as Plain YAML 3 | authors: 4 | - John Ryan 5 | - Steven Locke 6 | status: accepted 7 | approvers: 8 | - Dmitriy Kalinin 9 | - Eli Wrenn 10 | --- 11 | 12 | # Data Values as Plain YAML 13 | 14 | - [Problem Statement](#problem-statement) 15 | - [Terminology / Concepts](#terminology--concepts) 16 | - [Proposal](#proposal) 17 | - [Specification](#specification) 18 | - [Analysis](#analysis) 19 | - [Examples](#examples) 20 | - [Open Questions](#open-questions) 21 | 22 | ## Problem Statement 23 | 24 | Today, users primarily supply Data Values using Data Values Overlays. 25 | 26 | Configuration Consumers providing configuration data to a `ytt` invocation is required to do so by capturing those as a YAML document and annotating it _as_ a Data Values Overlay (i.e. `@data/values`). Further, they must resolve any merge ambiguities by annotating the document with `@overlay/...`. 27 | 28 | Examples: 29 | - https://github.com/vmware-tanzu/carvel-ytt/issues/81 30 | - https://github.com/vmware-tanzu/carvel-ytt/issues/51 31 | 32 | These requirements also make integrating `ytt` into automation tooling less than desirable: the `ytt` concepts of "data values" and "overlays" are foisted on the hapless end user of that automation tooling when all they wanted to do was customize their use of some higher-level feature. 33 | 34 | Examples: 35 | - the `values` section of an `InstalledPackage` custom resource: https://carvel.dev/kapp-controller/docs/latest/package-consumption/#installing-a-package 36 | - in GitLab, [generating child pipelines](https://docs.gitlab.com/ee/ci/parent_child_pipelines.html#dynamic-child-pipelines) based on plain YAML configuration input. 37 | 38 | Likewise, some Configuration Consumers (i.e. direct users of `ytt`) may be supplying Data Values via upstream tooling. These users shoulder the glue work required to integrate `ytt` into their toolchain. 39 | 40 | **What's needed is a mode of accepting Data Values into a `ytt` invocation stripped of any requirements to understand overlay syntax or concepts.** 41 | 42 | ## Terminology / Concepts 43 | 44 | These terms are used in this proposal: 45 | 46 | - **`data` module** : the [`ytt` module](https://carvel.dev/ytt/docs/latest/lang-ref-ytt/#data) that supplies Data Values to templates (typically via `load("@ytt:data", "data")`). 47 | - **Data Value** : an input into `ytt` (typically a key/value pair in a YAML document) that ultimately is presented to templates via the `data` module. 48 | - **Data Values** : an overloaded term, loosely meaning a bunch of "Data Value"s. In practice it can mean: 49 | 1. the final set of values resulting from merging all the Data Value inputs of a given `ytt` run (i.e. the output of `--data-values-inspect`). 50 | 2. a YAML file containing a set of "Data Value"s. 51 | - **Data Values File** : a plain YAML file that contains Data Values. 52 | - this is the primary object of this proposal. 53 | - **Data Values Overlay** : a YAML document annotated with `@data/values`. 54 | - optionally contains `@overlay` annotations to resolve matching ambiguities and indicate the action/edit desired. 55 | - separated from all _other_ overlays (e.g. those applied on top the output of templates) and evaluated as a group to calculate the Data Values" for the given `ytt` invocation. 56 | - "**plain YAML file**" : short-hand for a file whose contents are in YAML format and contain no `ytt` annotations. 57 | - **private library** : a `ytt` library acting as a dependency on the primary (aka "Root") library. (see also [`ytt` docs > Library Module](https://carvel.dev/ytt/docs/latest/lang-ref-ytt-library/#library-module)) 58 | 59 | 60 | ## Proposal 61 | 62 | In short: 63 | - introduce a new flag: `--data-values-file`; 64 | - that flag accepts only plain YAML; 65 | - it behaves like a set of `--data-value-yaml` parameters; 66 | - continue to support Data Values Overlays, but background them as an "advanced feature." 67 | 68 | Details: 69 | 70 | - [Goals](#goals) 71 | - [Non-Goals](#non-goals) 72 | - [Specification](#specification) 73 | - [Analysis](#analysis) 74 | - [Examples](#examples) 75 | 76 | 77 | ### Goals 78 | 79 | - describe a mechanism by which plain YAML can be supplied as Data Values. 80 | - preserve the full functionality provided by (now known as) Data Values Overlays. 81 | 82 | ### Non-Goals 83 | 84 | - achieve feature parity with Data Values Overlays. 85 | 86 | ### Specification 87 | 88 | `ytt` has a family of flags that provide an interface by which Data Values can be supplied. 89 | 90 | Today, Data Values... 91 | - can be plucked from OS environment variables (`--data-values-env[-yaml]`) or provided explicitly on the command-line (`--data-value[-yaml]`). 92 | - can be accepted as strings `--data-values{-env,}` or parsed as YAML expressions `--data-values{-env,}-yaml` 93 | 94 | We propose to extend this family of flags to be _the_ first-class means of supplying Data Values, including a full set of values via a plain YAML file. This will be done via a new flag: `--data-values-file` 95 | 96 | Syntax: 97 | 98 | ```console 99 | ytt ... --data-values-file [] 100 | ``` 101 | where: 102 | - `libref` directs the values to the specified private library 103 | - format: `@( | ~):` (i.e. identical to the syntax for other `--data-value...` flags) 104 | - `filepath` refers to a file that contains plain YAML. Data Values will be extracted and set from its contents. 105 | - there are no restrictions on the name of the file, but the `.yaml` or `.yml` extension is recommended. 106 | 107 | Given how common this flag will be used, it also has a shorthand form: 108 | 109 | ```console 110 | ytt ... -d [] 111 | ``` 112 | 113 | (see [Example: Giving a Single YAML Document](#example-giving-a-single-yaml-document) to illustrate the simplest case.) 114 | 115 | We consider a number of factors in turn: 116 | 117 | - [Multiple Documents](#multiple-documents) 118 | - [Interaction with Schema](#interaction-with-schema) 119 | - [Interactions with Data Values Overlays](#interactions-with-data-values-overlays) 120 | - [Interactions with Private Libraries](#interactions-with-private-libraries) 121 | - [Clear Consistent Messaging Around Data Values](#clear-consistent-messaging-around-data-values) 122 | - [Miscellaneous](#miscellaneous) 123 | 124 | #### Multiple Documents 125 | 126 | It is not uncommon that users will want to supply Data Values in not just one YAML document, but many. 127 | 128 | This use-case is supported: 129 | 130 | - the `--data-values-file` flag can be specified multiple times. 131 | - left-most instance applied first. 132 | - the file given to the flag can contain zero or more documents. 133 | - top-most document applied first. 134 | - each instance of a Data Value _replaces_ the previous value (this is already the behavior of the `--data-value[-yaml]` flag. 135 | - if a given Data Value had not been previously named, it is _added_ (the same behavior as if the value was supplied by `--data-value[-yaml] (key)+=(value)`, i.e. with the `+` operator as implemented at [data_values_flags.go](https://github.com/vmware-tanzu/carvel-ytt/blob/906bfe07cf1aded44f21fbdb501c76f911406fc8/pkg/cmd/template/data_values_flags.go#L234)) 136 | 137 | See also: 138 | - [Example: Giving Multiple YAML Files](#example-giving-multiple-yaml-files) 139 | - [Example: Giving Multiple YAML Documents in a Single File](#example-giving-multiple-yaml-documents-in-a-single-file) 140 | 141 | 142 | #### Interaction with Schema 143 | 144 | There are no new interactions between Data Values supplied with this new flag and ytt Schema. 145 | 146 | Data Values supplied with this new flag are typed, checked, and validated against schema in exactly the same way as Data Values specified via the other `--data-value` flags. 147 | 148 | 149 | #### Interactions with Data Values Overlays 150 | 151 | The file specified with the `--data-values-file` flag are referred to as "Data Values Files" 152 | 153 | There are no new interactions between Data Values supplied with this new flag and Data Values Overlays: 154 | 155 | - just as values supplied via other `--data-value...` flags have higher precedence over any Data Values Overlays, so too with the values specified by this new flag. 156 | - users will not be _required_ to supply data values via the `--data-values-file` flag; users are free to continue specifying these values via Data Values Overlays. 157 | - documentation and examples need adjusting to clearly message that _this_ mechanism is preferred for most use-cases 158 | 159 | 160 | #### Interactions with Private Libraries 161 | 162 | A user may optionally indicate the exact library that should receive the Data Values. This is done using the `libref` notation described at the start of [Specification](#specification). 163 | 164 | See also: [Example: Targeting a Data Values to a Library](#example-targeting-a-data-values-to-a-library). 165 | 166 | #### Clear Consistent Messaging Around Data Values 167 | 168 | For the features described in this proposal to be well adopted and not create confusion, clear and consistent messaging needs to be crafted. 169 | 170 | This includes (but is not limited to): 171 | - the terms "Data Values File" and "Data Values Overlay" be consistently used throughout documentation, examples, and output from the tool itself 172 | - when Data Values are mentioned, the "Data Values File" feature is referenced (rather than the "Data Values Overlay" feature) 173 | - examples use Data Values Files (instead of Data Values Overlays), unless overlay-specific features are being employed 174 | 175 | It's also crucial to keep in mind that there are two fundamental proto-personas affected by the introduction of another way to supply Data Values: 176 | - Integrators -- the folks who are incorporating `ytt` as an implementation detail of their tooling 177 | - Configuration Consumer -- people who are using `ytt` as an "end user": using the CLI directly. 178 | 179 | Integrators ought to find it easy to discover this new flag as it relates to the other `--data-value...` flags, as those are most suitable for integrators, already. 180 | 181 | Configuration Consumers should only find it easy to discover this new flag when all of the overall messaging (i.e. docs, examples, Playground) support it as a first class feature. 182 | 183 | _(see also: [Consideration: Implies a Significant Effort to Implement](#consideration-implies-a-significant-effort-to-implement))_ 184 | 185 | 186 | #### Contents of Data Values Files 187 | 188 | Data Values Files are plain YAML files that happen to contain Data Values. 189 | 190 | As such: 191 | - YAML comments (i.e. strings prefixed with `#`) are permitted (as well as other features enjoyed by plain YAML) 192 | - `ytt` annotations (i.e. strings prefixed with `#@`) are _not_ permitted 193 | - this helps catch cases where a user submits a `ytt` template (e.g. a Data Values Overlay, containing at least the `@data/values` document annotation), and it quietly ignores all directives because the file is treated as a plain YAML. 194 | - one can also understand this constraint as a way that this flag behaves exactly like its `--data-value...` siblings: that one can't include "directives" in those other flags, nor can they with this `--data-value-file` flag. 195 | - when such features are desired, users can employ / integrators can allow Data Values Overlays and direct those inputs to the `--file` flag. 196 | 197 | ### Analysis 198 | 199 | The following are commentary regarding the above specification: 200 | 201 | - [Consideration: Confusingly similar to `--data-value-file`](#consideration-confusingly-similar-to---data-value-file) 202 | - [Consideration: Does not support merging arrays](#consideration-does-not-support-merging-arrays) 203 | - [Other Approach Considered: Introduce a new File Mark Type](#other-approach-considered-introduce-a-new-file-mark-type) 204 | - [Consideration: Potentially Confusing that Data Values Overlays are not Data Values files](#consideration-potentially-confusing-that-data-values-overlays-are-not-data-values-files) 205 | 206 | #### Consideration: Confusingly similar to `--data-value-file` 207 | 208 | One observation is that `--data-values-file` is easily confused or even mistyped to `--data-value-file`. 209 | 210 | We believe this similarity is acceptable because: 211 | 1. it is relatively easy to detect when the argument given was intended for the other flag; and 212 | 2. the actual difference (i.e. the former is the plural of "data value") can help in remembering which flag is applicable when. 213 | 214 | Examining the "signature" of each flag: 215 | - `--data-values-file` — has an argument in the format: `` (e.g. `../dev/values.yml`) 216 | - `--data-value-file` — has an argument in the format: `=` (e.g. `server.certificate=../server.cert`) 217 | 218 | We note that `--data-value-file` _requires_ the equal sign in the parameter — a character that is highly unlikely to appear in a `filepath`. We expect it trivial to detect when one flag was used with the parameter intended for the other and can provide the needed hint to get the user back on track. 219 | 220 | Further, that the newly proposed flag refers to `data-values` (i.e. the plural), it's a mnemonic that the flag is specifying _multiple_ values. In contrast, the existing flag refers to `data-value` (i.e. the singular), a reminder that the value specified is headed for the data value at `keypath`. 221 | 222 | 223 | #### Consideration: Does not support merging arrays 224 | 225 | The careful reader will note that values specified through the `--data-values-file` flag _replaces_ any previously set value. This is most noticeable for values that are arrays. 226 | 227 | We observe: 228 | - in the community managing Kubernetes configuration, almost all libraries that declare a configuration value as an array set the default to an empty array (`[]`). 229 | - most users think of supplying data values as "setting" the value (rather than merging). 230 | - in the rare case where the user intends to append values to an array, they can accomplish this through a Data Value Overlay. 231 | 232 | Given those conditions, we conclude that the most common cases are best served by _replacing_ the specified values. 233 | 234 | 235 | #### Consideration: Implies a Significant Effort to Implement 236 | 237 | Introducing a second way to supply input files _and_ presenting this new approach as the _primary_ means of supplying Data Values implies a significant effort. This is in _addition_ to the effort required to implement the specified feature, itself (i.e. the `--data-values-file` flag and related plumbing), 238 | 239 | This includes (but is not limited to): 240 | - extending the bulk-in/bulk-out interface to also accept data values in a separate key. 241 | - extending the `ytt` Playground to separately accept these data values files and use the updated bulk-in/bulk-out feature. 242 | - provide some means of populating the Playground with a gist that includes one or more Data Values files. 243 | - rework a non-trivial amount of documentation that describes the use of Data Values (the scope of which is sketched in [Clear Consistent Messaging Around Data Values](#clear-consistent-messaging-around-data-values), above). 244 | 245 | There's a risk that if this work is _not_ included, users can become quite confused by the multiple ways to specify their Data Values. 246 | 247 | Possible mitigation: 248 | - quietly introduce the specified flag, making it available for tools integrators, specifically. Spread the investment describe in the previous paragraph over time. Make the documentation/messaging change as the final step. 249 | 250 | #### Consideration: Potentially Confusing that Data Values Overlays are not Data Values files 251 | 252 | It is probable that a user submits a file with document(s) annotated with `@data/values` to the `--data-values-file` flag. If they did, they would get an error given that no annotations would be permitted in the such input. 253 | 254 | Beyond this being surprising (i.e. a failure when a success seems guaranteed), it also risks creating cognitive dissonance regarding Data Values: if a "Data Values" is not acceptable input to `--data-values-file` then, what _is_ a "Data Value"? 255 | 256 | This amounts to a kind of [accidental complexity](http://www.catb.org/~esr/writings/taoup/html/ch13s01.html#id2961759) from two features being somewhat non-orthogonal. 257 | 258 | 259 | To mitigate this complexity: 260 | - include a page in the docs dedicated to singularly covering these two file types. 261 | - differentiate them clearly 262 | - make plain when each would be used (use-case based) and by what mechanisms 263 | - when a user supplies a Data Values Overlay as the input for the `--data-values-file` flag, let it be that the error message is clear about this difference. 264 | - the error message references the previously mentioned page in the docs(?) 265 | 266 | With those measures in place, we believe the added complexity is worth it. 267 | 268 | 269 | #### Other Approach Considered: Introduce a new File Mark Type 270 | 271 | [A previous version of this proposal](https://github.com/vmware-tanzu/carvel-community/blob/2a817298f3e263438ff3767b2bc4ccfcb9dbbc1b/proposals/ytt/002-raw-data-values/README.md) centered around the idea of being able to mark an input file (i.e. a file implied by a `--file` argument) as a "Plain YAML Data Value" file. 272 | 273 | **Pros:** 274 | - maintains a consistent interface: all input files continue to be provided through one interface: the `--file` flag; 275 | - preserves orthogonality: it keeps the two concerns of "which order to process" and "how to process" separate. 276 | 277 | **Cons:** 278 | - it does nothing to help make the use of Data Values simpler; in fact, it complicates the use by requiring an additional (and rather verbose) flag. 279 | - there would be two flavors of Data Values being included with rather different behaviors: \ 280 | _(this specific point was not recognized in the previous version of this proposal, but came into view during some design discussions which prompted the pivot to this approach.)_ 281 | - files annotated with `@data/values`... 282 | - _append_ its array items to any existing array, by default 283 | - will report overlay error when a key in the file is not already defined as a Data Value 284 | - files marked with `type=yaml-plain-data-values`... 285 | - _replace_ any existing array values with its own 286 | - when a key is not yet a Data Value, declares it as a new one 287 | 288 | As it stands, today, it's more clear that: 289 | - the approach outlined in this revision of this proposal is not mutually exclusive to providing other means of including Data Values Files (although care should be taken in creating deliberately overlapping features) 290 | - the design tension pulling from the family of `--data-value...` flags feels stronger than the need to keep all file-based inputs together (another way of stating the pros, listed above) 291 | - the behavior of this flag (intentionally) follows that of the other `--data-value...` flags and therefore a natural extension. 292 | - in practice, Data Values that actually configure/personalize the `ytt` invocation are typically located in a place elsewhere from the author supplied templates, overlays, and _declaring_ Data Values (i.e. those that establish _what are_ the configuration variables). 293 | 294 | ### Examples 295 | 296 | 1. [Giving a Single YAML Document](#example-giving-a-single-yaml-document) 297 | 1. [Giving Multiple YAML Files](#example-giving-multiple-yaml-files) 298 | 1. [Giving Multiple YAML Documents in a Single File](#example-giving-multiple-yaml-documents-in-a-single-file) 299 | 1. [Giving an Empty File Does Nothing](#example-giving-an-empty-file-does-nothing) 300 | 1. [Declaring Additional Data Values](#example-declaring-additional-data-values) 301 | 1. [Using Process Substitution to Supply Data Values](#example-using-process-substitution-to-supply-data-values) 302 | 1. [Targeting a Data Values to a Library](#example-targeting-a-data-values-to-a-library) 303 | 1. [Specifying a Scalar Value](#example-specifying-a-scalar-value) 304 | 305 | 306 | #### Example: Giving a Single YAML Document 307 | 308 | _Absent schema, the first file declares the set of Data Values._ 309 | 310 | `values.yml` 311 | ```yaml 312 | foo: 13 313 | bar: 314 | - name: alpha 315 | - name: beta 316 | ``` 317 | 318 | ```console 319 | $ ytt --data-values-file values.yml --data-values-inspect 320 | foo: 13 321 | bar: 322 | - name: alpha 323 | - name: beta 324 | ``` 325 | 326 | #### Example: Giving Multiple YAML Files 327 | 328 | _Each file overwrites previously specified values._ 329 | 330 | `values.yml` 331 | ```yaml 332 | foo: 13 333 | bar: 334 | - alpha 335 | - beta 336 | ``` 337 | 338 | `values2.yml` 339 | ```yaml 340 | bar: 341 | - first 342 | - second 343 | ``` 344 | 345 | ```console 346 | $ ytt --data-values-file values.yml --data-values-file values2.yml --data-values-inspect 347 | foo: 13 348 | bar: 349 | - first 350 | - second 351 | ``` 352 | 353 | #### Example: Giving Multiple YAML Documents in a Single File 354 | 355 | _Each document overwrites previously specified values._ 356 | 357 | `values.yml` 358 | ```yaml 359 | foo: 13 360 | bar: 361 | - alpha 362 | - beta 363 | --- 364 | bar: 365 | - first 366 | - second 367 | ``` 368 | 369 | ```console 370 | $ ytt --data-values-file values.yml --data-values-inspect 371 | foo: 13 372 | bar: 373 | - first 374 | - second 375 | ``` 376 | 377 | #### Example: Giving an Empty File Does Nothing 378 | 379 | _An empty file is accepted; and is a no-op._ 380 | 381 | `values.yml` 382 | ```yaml 383 | foo: 13 384 | bar: 385 | - alpha 386 | - beta 387 | ``` 388 | 389 | `values2.yml` 390 | ```yaml 391 | ``` 392 | 393 | ```console 394 | $ ytt --data-values-file values.yml --data-values-file values2.yml --data-values-inspect 395 | foo: 13 396 | bar: 397 | - alpha 398 | - beta 399 | ``` 400 | 401 | #### Example: Declaring Additional Data Values 402 | 403 | _Absent schema, additional data values can be declared (if schema is present, new data values must be declared by overlaying schema)._ 404 | 405 | `values.yml` 406 | ```yaml 407 | foo: 13 408 | bar: 409 | - alpha 410 | - beta 411 | ``` 412 | 413 | `values2.yml` 414 | ```yaml 415 | bar: 416 | - first 417 | - second 418 | ree: true 419 | ``` 420 | 421 | ```console 422 | $ ytt --data-values-file values.yml --data-values-file values2.yml --data-values-inspect 423 | foo: 13 424 | bar: 425 | - first 426 | - second 427 | ree: true 428 | ``` 429 | 430 | #### Example: Using Process Substitution to Supply Data Values 431 | 432 | _In Unix shells, input can be provided via process substitution. Since there are no restrictions on the name of the file, no custom filename is required._ 433 | 434 | ```console 435 | $ ytt --data-values-file <(echo -e "foo: 13\nbar: [first, second]") --data-values-inspect 436 | foo: 13 437 | bar: 438 | - first 439 | - second 440 | ``` 441 | 442 | _or, given a set of files..._ 443 | 444 | `values.yml` 445 | ```yaml 446 | foo: 13 447 | bar: 448 | - alpha 449 | - beta 450 | ``` 451 | 452 | `values2.yml` 453 | ```yaml 454 | bar: 455 | - first 456 | - second 457 | ``` 458 | 459 | ```console 460 | $ ytt --data-values-file <(cat values.yml values2.yml) --data-values-inspect 461 | foo: 13 462 | bar: 463 | - first 464 | - second 465 | ``` 466 | 467 | _or, further could be used to merge a set files (as is the case when a large set of Data Values is split into several files)..._ 468 | 469 | `values.yml` 470 | ```yaml 471 | foo: 13 472 | bar: 473 | - alpha 474 | - beta 475 | ``` 476 | 477 | `values2.yml` 478 | ```yaml 479 | ree: 480 | - first 481 | - second 482 | ``` 483 | 484 | ```console 485 | $ ytt --data-values-file +:<(cat values.yml values2.yml) --data-values-inspect 486 | foo: 13 487 | bar: 488 | - alpha 489 | - beta 490 | ree: 491 | - first 492 | - second 493 | ``` 494 | 495 | #### Example: Targeting a Data Values to a Library 496 | 497 | _Configuration Consumers can direct a set of Data Values to a specific library._ 498 | 499 | ```console 500 | $ tree . 501 | . 502 | ├── config 503 | │ ├── _ytt_lib 504 | │ │ └── libby 505 | │ │ ├── defaults.yml 506 | │ │ └── template.yml 507 | │ └── main.yml 508 | └── values.yml 509 | ``` 510 | 511 | `libby/defaults.yml` 512 | ```yaml 513 | #@data/values 514 | --- 515 | foo: 0 516 | ``` 517 | 518 | `libby/template.yml` 519 | ```yaml 520 | #@ load("@ytt:data", "data") 521 | --- 522 | foo_in_lib: #@ data.values.foo 523 | ``` 524 | 525 | `main.yml` 526 | ```yaml 527 | #@ load("@ytt:library", "library") 528 | #@ load("@ytt:template", "template") 529 | --- #@ template.replace(library.get("libby").eval()) 530 | ``` 531 | 532 | `values.yml` 533 | ```yaml 534 | --- 535 | foo: 42 536 | ``` 537 | 538 | ```console 539 | ytt -f config/ --data-values-file @libby:values.yml 540 | foo_in_lib: 42 541 | ``` 542 | 543 | #### Example: Specifying a Scalar Value 544 | 545 | _While rare, it is valid to specify a single data value that is a scalar._ 546 | 547 | `values.yml` 548 | ```yaml 549 | 42 550 | ``` 551 | 552 | `template.yml` 553 | ```yaml 554 | #@ load("@ytt:data", "data") 555 | --- 556 | answer: #@ data.values 557 | ``` 558 | 559 | ```console 560 | $ ytt -f template --data-values-file values.yml 561 | answer: 42 562 | ``` 563 | 564 | ## Open Questions 565 | -------------------------------------------------------------------------------- /proposals/ytt/001-schemas/README.md: -------------------------------------------------------------------------------- 1 | --- 2 | tags: schema 3 | --- 4 | # Schemas 5 | 6 | - Status: Scoping | **Pre-Alpha** | In Alpha | In Beta | GA | Rejected 7 | - Originating Issue: [ytt#103](https://github.com/k14s/ytt/issues/103) 8 | - Shared working copy: [![hackmd-github-sync-badge](https://hackmd.io/pODV3wzbT56MbQTxbQOOKQ/badge)](https://hackmd.io/pODV3wzbT56MbQTxbQOOKQ) 9 | 10 | 11 | # Table of Contents 12 | 13 | [TOC] 14 | 15 | --- 16 | 17 | # Problem Statement 18 | 19 | Configuration Authors want to be able to document, constrain, and generate configuration inputs (for example, Data Values) for _their_ users. 20 | 21 | Specifically, they would like to: 22 | - **have a single authoritative definition of possible inputs** (think: API) _to simplify the task of discovering the total set of possible inputs_; 23 | - be able to **attach documentation to those definitions** in a format that tools could extract _to communicate meaning and intent of each input_; 24 | - **provide structure/type checks and value validations** to inputs _so that errors can be detected quickly and reported in terms the user will likely understand_. 25 | 26 | As of v0.26.0, `ytt` technically has a feature set that could satisfy all of these requirements. However, doing so is onerous and error prone. Also, many of the error messages that `ytt` emits today around these topics have been found by many users to be enigmatic (if not frustrating). 27 | 28 | There exist tools that can meet some of these requirements. However, given the need to weave in many of these behaviors to the structure building algorithm that is `ytt`'s main flow, they fall short of meeting the first-order needs and do little to improve the user experience around specifying data values. 29 | 30 | As an opportunistic driver, Configuration Authors who wrestle with non-trivial templatizing and patching of configuration would appreciate the ability to validate the resulting documents, helping ensure that their `ytt` library yields well-formed and valid YAML docs. 31 | 32 | # Proposal 33 | 34 | Implement a light-weight [type system](https://en.wikipedia.org/wiki/Type_system) for YAML documents: `ytt` Schema. 35 | 36 | Focus first on addressing the needs around Data Values documents: making it easy to declare and activate a schema for this input. 37 | 38 | This mechanism ought to: 39 | 40 | - provide a way to articulate a schema for Data Values; 41 | - declare YAML document contents: Map, Array, their contained items by example (i.e. by simply adding the nodes) 42 | - the schema must be YAML — itself — _so that users can leverage all that they learn/know about this format and not have to context switch to read/consider/write this description_; 43 | - schema documents should enjoy all the features of data values files and templates _so that all of the attendant capabilities are available in this context as well_; 44 | - be able to define functions to capture expressions that are evaluated at runtime 45 | - be able to attach metadata (i.e. annotations) to capture information _outside_ of the structure of the document 46 | - provide a means of attaching documentation to nodes; 47 | - validate supplied Data Values against the schema, reporting violations in terms the end-user would understand. 48 | 49 | Keep in mind that we aim for being able to articulate schema for any YAML document `ytt` handles. e.g. [ytt#103@schema-for-templates](https://github.com/k14s/ytt/issues/103#issuecomment-624326922) 50 | 51 | ## Sources of Inspiration 52 | 53 | Be aware of prior art: avoid reinvention; shamelessly "repurpose" great ideas. 54 | 55 | - [JSON Schema](https://json-schema.org/) 56 | - [OpenAPI v3 schema](https://swagger.io/specification/) — _(see also [ytt#103@openapi](https://github.com/k14s/ytt/issues/103#issuecomment-672296576))_ 57 | 58 | Remember the differences in context: 59 | 60 | - we seek simplicity, brevity, and readability. 61 | 62 | 63 | # Specification 64 | ## Glossary 65 | 66 | `ytt` bridges two type spaces: YAML and Starlark. Terms from both spaces are used in this specification 67 | 68 | When referring to elements that are YAML, this specification uses terms defined in the [YAML 1.2 specification](https://yaml.org/spec/1.2/spec.html). 69 | 70 | Commonly used terms: 71 | - **document** — root node with a single value that can be a mapping, sequence, or scalar. 72 | - **mapping** — a map (aka "dictionary") containing zero or more key/value pairs. Each value can be a mapping, sequence, or scalar. 73 | - **node** - the composite of a YAML document; one of: a document, map, map item, array, array item, or scalar. 74 | - **scalar** — single value type: boolean, float, integer, null, string. 75 | - **sequence** — an array (aka "list") containing zero or more elements, in order. 76 | 77 | 78 | Terms as they are meant in this specification: 79 | 80 | - **schema** — (n.) a definition of the structure, types and validation rules for a given YAML document. 81 | - **type** — (n.) a specific shape for a given node: a mapping, sequence, or scalar. 82 | - **type checking** — (v.) comparing the shape of a YAML structure against its schema's types. 83 | - **type check** — (n.) the result of type checking 84 | - **validating** — (v.) determining whether the values contained in a YAML structure are acceptable. 85 | - **validation** — (n.) the result of validating 86 | - **validation rule** — predicate that indicates whether a given (sub-)structure is valid or not. 87 | - **verifying** — (v.) a short-hand way of saying, "type checking and validation." 88 | - **violation** — (n.) an instance of either a failed type check or a failed validation. 89 | 90 | ## Part 1: Declaring Schema 91 | 92 | Configuration Authors establish a Schema by capturing the structure in a YAML document: 93 | 94 | ### @schema/definition 95 | 96 | ```yaml 97 | #@schema/definition data_values=False 98 | --- 99 | #! Schema contents 100 | ``` 101 | - `data_values` (`bool`) — whether this schema is applicable to data values. If this 102 | is not set to `True`, this results in an error. 103 | 104 | Notes: 105 | - Schema must be defined _before_ any other files containing YAML can be processed. Therefore, a file containing a Schema document must not contain other kinds of documents. 106 | - files containing Schema documents will be detected among input files (i.e. those specified by `-f`). 107 | - the first file containing a Schema document establishes the "base". Subsequent files containing Schema documents are overlayed onto the "base" in the order they appear (identically to how Data Values files are processed). 108 | 109 | ## Part 2: Applying Schema 110 | 111 | The Configuration Author can currently apply schema to Data Values. 112 | 113 | _(Review [previous versions of this proposal](https://github.com/k14s/design-docs/commit/abd6dba2dc4bf9e46c0f352d2f2ae7bf17d08ac6#diff-1b42631f7965920fdf750e6ae57eb93df771f053b684d338dc89b5d0eb215bc1R114-R142) including [commentary as to why those ideas were deferred](https://github.com/k14s/design-docs/pull/1#discussion_r520717953) for ideas to more broadly apply schema)_ 114 | 115 | 116 | **TODO:** elaborate on programmatic application of schema (e.g. to the contents of a ConfigMap) 117 | 118 | ## Part 3: Supported Types 119 | 120 | While YAML provides for an extendable range of types, the `ytt` Schema supports a specific set. 121 | 122 | ### Scalar Types 123 | 124 | - `bool` — `true`, `false` (and when not strict, `yes`, `no`, `Y`, `N`, etc.) 125 | - `float` — e.g. `0.4` 126 | - `int` — e.g. `42` 127 | - `null` — `null`, `~`, and when value is omitted. 128 | - `string` — e.g. `""`, `"ConfigMap"`, `"0xbeadcafe"` 129 | 130 | 131 | ### Algebraic Types 132 | 133 | - `any` — any valid YAML value is permitted. 134 | - `oneof` — value can be any one of the the specified types. 135 | 136 | TBD: 137 | - **Q:** given that there is no concrete type `any` (i.e. it represents the union of all types), should "any value" be 138 | - a named type: `any` or 139 | - a condition: `#@schema/type any=True` 140 | - **Q:** is there a need for an `anyof` type? (i.e. where a value if well-formed if it matches any subset of the union of the specified types) (for example [anyOf from Swagger docs](https://swagger.io/docs/specification/data-models/oneof-anyof-allof-not/#anyof-vs-oneof)) 141 | 142 | 143 | ### Collection Types 144 | 145 | A collection type is the pair of itself (either `map` or `array`) and its contained type. 146 | 147 | #### Array type 148 | 149 | `array` — a YAML sequence, each item must be of the specified `type`. 150 | 151 | Examples: 152 | - `array` — a sequence of strings (and only strings) 153 | - `array>` — a sequence containing any combination of integers and strings 154 | - `array>` — a sequence containing only map instances of the same shape. 155 | 156 | 157 | #### Map type 158 | 159 | `map ` — a YAML mapping that must contain exactly one item of each given key, whose value must be of the type specified. 160 | 161 | Examples: 162 | - `map <"enable":bool, "static_ip":oneof` — two items: one must have the key `enable` containing a boolean, the other must have the key `static_ip` and can be either a string or null. 163 | - `map <"cpu":oneof>` — key named `cpu` that can be set to a floating point value, and integer or a string. 164 | 165 | 166 | ## Part 4: Defining Schema (implicitly) 167 | 168 | The schema definition spells out the set of nodes that must make up that kind of YAML document. Documents that are type checked and validated against the schema are guaranteed to be proper. 169 | 170 | For example, in the case of Data Values, the schema guarantees to templates that all values exist and are of the correct type. The aleviates templates from doing existence and type checks, themselves. 171 | 172 | ### Inferring Types 173 | 174 | Structure in Schema is largely expressed by example rather than by description. Types are — by default — inferred based on the values given. The Configuration Author can override these defaults using the [`@schema/type`](#schematype) annotation. 175 | 176 | ```yaml= 177 | #@schema/definition name="data/values" 178 | --- 179 | system_domain: "" 180 | 181 | load_balancer: 182 | enable: true 183 | static_ip: "" 184 | 185 | app_domains: 186 | - "" 187 | 188 | databases: 189 | - name: "" 190 | adapter: postgresql 191 | host: "" 192 | port: 5432 193 | user: admin 194 | secretRef: 195 | name: "" 196 | ``` 197 | 198 | where: 199 | - Data Values must consist of four top-level map items: 200 | - `system_domain` — whose value is a string, 201 | - `load_balancer` — whose value is a map containing two items: 202 | - `enable` — whose value is a boolean 203 | - `static_ip` — whose value is a string 204 | - `app_domains` — whose value is an array of strings (and only strings) 205 | - `databases` — whose values is an array of maps of a specific shape: six (6) items: 206 | - `name` — containing a string 207 | - `adapter` — containing a string 208 | - `host` — also holds a string 209 | - `port` — holds an integer 210 | - `user` — a string 211 | - `secretRef` — containing a map with one key: 212 | - `name` — containing a string 213 | 214 | ### Inferring Default Values 215 | 216 | The exact values that the Configuration Author specifies in Schema are the defaults (with one natable exception — arrays — which are detailed below). This ensures that all nodes are initialized with values, ideally with reasonable defaults. 217 | 218 | Configuration Consumers can enjoy specifying only those values which vary from the default, reducing risk of errors and generally making configuration easier to maintain. 219 | 220 | #### Inferring Defaults for Scalars 221 | 222 | When a scalar value is specified, it is simply the default. From the example, above: 223 | 224 | - `system_domain` is an empty string, by default. 225 | - `load_balancer.enable` is true by default. 226 | - `databases[].adapter` is `"postgres"`, by default. 227 | 228 | #### Inferring Defaults for Mappings 229 | 230 | The set of items of a map are its default: any item missing will be automatically inserted with _its_ defaults. 231 | 232 | - if `load_balancer` is omitted from supplied Data Values, it is inserted: 233 | - `load_balancer.enabled` is `true` 234 | - `load_balancer.static_ip` is `""` 235 | - if `load_balancer` is specified with only `static_ip`, `enabled` is `true`. 236 | 237 | When the Configuration Consumer supplies an empty map (i.e. `{}`), all items are defaulted: 238 | 239 | For example, 240 | 241 | `load_balancer: {}` 242 | 243 | is defaulted to: 244 | 245 | ```yaml 246 | load_balancer: 247 | enable: true 248 | static_ip: "" 249 | ``` 250 | 251 | #### Inferring Defaults for Arrays 252 | 253 | The default value for all arrays is an empty array. Arrays are the only type for which defaults are **not** inferred. 254 | 255 | From the example, above, 256 | 257 | - `app_domains`, if not otherwise specified is `[]` 258 | - `databases`, if not otherwise specified is `[]` 259 | 260 | This behavior can be overriden using the [`@schema/default`](#schemadefault) annotation. 261 | 262 | When values for an array _are_ provided by the Configuration Consumer, each item is defaulted based on the type specified in the schema. 263 | 264 | Focusing on just `databases`, if the Configuration Consumer supplies this Data Value: 265 | 266 | ```yaml= 267 | #@data/values 268 | --- 269 | databases: 270 | - name: uaa 271 | - name: capi 272 | host: capi-db.svc.cluster.local 273 | secretRef: 274 | name: capi-db-credentials 275 | - {} 276 | ``` 277 | 278 | The resulting Data Values (i.e. that supplied to templates), would be: 279 | 280 | ```yaml= 281 | databases: 282 | - name: uaa 283 | adapter: postgresql 284 | host: "" 285 | port: 5432 286 | user: admin 287 | secretRef: 288 | name: "" 289 | - name: capi 290 | adapter: postgresql 291 | host: capi-db.svc.cluster.local 292 | port: 5432 293 | user: admin 294 | secretRef: 295 | name: capi-db-credentials 296 | - name: "" 297 | adapter: postgresql 298 | host: "" 299 | port: 5432 300 | user: admin 301 | secretRef: 302 | name: "" 303 | ``` 304 | 305 | 306 | ## Part 5: Defining Schema (explicitly) 307 | 308 | Configuration Authors may override inferred typing and defaults through annotations. 309 | 310 | ### @schema/type 311 | 312 | Explicitly sets the type of the annotated node. 313 | 314 | ```yaml 315 | @schema/type one_of=typeA [, typeB[, ...]] [or_inferred=True] any=True 316 | ``` 317 | 318 | - `one_of` (`array of string`) — 319 | - `typeX` (`string`) — the name of the type that the value of the annotated node must be. 320 | - Accepted values are: 321 | - "bool" 322 | - "float" 323 | - "int" 324 | - "null" 325 | - "string" 326 | - [Algebraic Types](#Algebraic-Types) explains the meaning of `one_of` types. 327 | - `or_inferred` (`bool`) — whether to include the type inferred from the value of the annotated node (default: `false`). 328 | - if `false` the type of the value is ignored 329 | - if `true` the type of the value is included in the set of possible types. 330 | - `any` (`bool`) — whether or not any and all types are permitted on this node (and its children) 331 | - to avoid confusion, this keyword argument is mutually exclusive to all other arguments; if this argument is 332 | provided along with others, an error results. 333 | 334 | Note: 335 | - it is an error to specify a default value that does not match the specified type. 336 | - a deliberate consequence is that upto one collection type can be specified (through the `or_inferred` keyword argument). 337 | - requiring `one_of` to be a keyword argument (rather than a positional one) leaves room for other kinds of type 338 | combinations (e.g. not of, any of, all of). 339 | 340 | __ 341 | 342 | _Example 1: Integer or String_ 343 | 344 | ```yaml 345 | #@schema/type one_of=["int", "string"] 346 | percentage: 0 347 | ``` 348 | 349 | allows `percentage` to contain either an integer or string. 350 | 351 | __ 352 | 353 | _Example 2: Nullable Collection_ 354 | 355 | ```yaml 356 | #@schema/type one_of="null" or_inferred=True 357 | aws: 358 | access_key: "" 359 | secret_key: "" 360 | ``` 361 | 362 | allows `aws` to either be `null` or the pair of access and secret key values. 363 | 364 | __ 365 | 366 | _Example 3: Type mismatch_ 367 | 368 | ```yaml 369 | #@schema/type one_of=["null", "string"] 370 | aws: 371 | access_key: "" 372 | secret_key: "" 373 | ``` 374 | 375 | will produce an error: with `or_inferred` defaulted to `false`, the given values's type (i.e. `map<...>`) is not permitted. 376 | 377 | __ 378 | 379 | ### @schema/default 380 | 381 | Explicitly sets the default value of the annotated node. 382 | 383 | ```yaml 384 | @schema/default value 385 | ``` 386 | 387 | - `value` (any except `function`) — the overriding default value. 388 | 389 | Notes: 390 | - `value` must be of the type of the annotated node. 391 | - The type of a node is never inferred from the overriding default value. 392 | - this annotation is the _only_ way to set a non-empty default value for arrays. 393 | - Assumption: setting default values for arrays is uncommon. 394 | 395 | TBD: 396 | - in order to encourage readability of `ytt` schemas, we would like to limit use of this annotation only to cases where it is required. However, this gets complicated when the value can also be `null`. Cases where explicitly setting a default value is redundant include: 397 | - a scalar default on a node with a scalar value (that's not nullable) 398 | - a map whose items either contain scalars or maps of scalars. 399 | - `any` 400 | 401 | __ 402 | 403 | _Example 1: Default values for arrays of scalars_ 404 | 405 | ```yaml 406 | #@schema/default ["apps.example.com", "services.example.com"] 407 | app_domains: 408 | - "" 409 | ``` 410 | 411 | __ 412 | 413 | _Example 2: Default values for arrays of maps_ 414 | 415 | ```yaml= 416 | #@def uaa_db(): 417 | name: uaa 418 | host: uaa-db.svc.cluster.local 419 | user: admin 420 | #@end 421 | 422 | #@def capi_db(): 423 | name: capi-embedded 424 | user: admin 425 | #@end 426 | 427 | #@schema/default [uaa_db(), capi_db(), {"name": "null_db"}] 428 | databases: 429 | - name: "" 430 | adapter: postgresql 431 | host: "" 432 | port: 5432 433 | user: admin 434 | secretRef: 435 | name: "" 436 | ``` 437 | 438 | where 439 | - `databases` is type-inferred as an array of maps of the shape shown from L14-20. 440 | - there are three items by default: the `uaa`, `capi-embedded`, and `null_db` 441 | 442 | __ 443 | 444 | _Example 3: Explicitly nullable_ 445 | 446 | ```yaml 447 | #@schema/type one_of="null", or_inferred=True 448 | #@schema/default None 449 | aws: 450 | access_key: "" 451 | secret_key: "" 452 | ``` 453 | 454 | where 455 | - `aws` is typed as a nullable map; the inferred default value are the two items, `access_key` and `secret_key` both empty strings. 456 | - in fact, the default of `aws` will be `null` (`None` is the Starlark keyword for `null`) 457 | 458 | __ 459 | 460 | ### @schema/nullable 461 | 462 | Extends the type of the node to include "null" _and_ to be "null" by default. 463 | 464 | ```yaml 465 | @schema/nullable 466 | ``` 467 | 468 | Notes: 469 | - Assumption: it is common for the Configuration Author to wish to specify that no matter the type, a value is both nullable and null, by default. 470 | - this annotation is equivalent to: 471 | ``` 472 | #@schema/type "null", or_inferred=True 473 | #@schema/default None 474 | ``` 475 | 476 | _Example 1: Null string_ 477 | 478 | ```yaml 479 | #@schema/nullable 480 | name: "" 481 | ``` 482 | 483 | where `name` can hold a string, but also be null. `name` is `null` by default. 484 | 485 | __ 486 | 487 | _Example 2: Null map_ 488 | 489 | ```yaml 490 | #@schema/nullable 491 | aws: 492 | username: "" 493 | password: "" 494 | ``` 495 | 496 | where `aws` is null by default. When set `aws` will be a map with the two items. 497 | 498 | __ 499 | 500 | _Example 3: Nullable map_ 501 | 502 | ```yaml 503 | #@schema/nullable 504 | #@schema/default {} 505 | cf_db: 506 | username: "sa" 507 | admin_password: "" 508 | ``` 509 | 510 | is equivalent to 511 | 512 | ```yaml 513 | #@schema/type one_of="null", or_inferred=True 514 | cf_db: 515 | username: "sa" 516 | admin_password: "" 517 | ``` 518 | 519 | and in both cases, `cf_db` is defaulted to the inlined value (`username`, the string `"sa"`, `admin_password`, an empty string). However, `cf_db` _can_ also be set to `null`. 520 | 521 | 522 | ### @schema/key 523 | 524 | Defines criteria for values permitted for the key of the annotated node. 525 | 526 | ``` 527 | @schema/key [allowed=Regexp] [(expects=Int|String|List|Function | missing_ok=Bool)] 528 | ``` 529 | 530 | - `allowed=Regexp` — (optional) regular expression declaring the required format of the key. 531 | - if this argument is omitted, the key specified in the document must match that in the schema. 532 | - `expects=Int|String|List|Function` — (optional) expected number of items to be `allowed` 533 | (and that are not matched by other schema). 534 | - `Int` — must match this number, exactly 535 | - `String` (e.g. `"1+"`) — must match _at least_ the number 536 | - `Function(found):Bool` — predicate of whether the expected number of key were found 537 | - `found` (`Int`) — number of actual matches 538 | - `List[Int|String|Function]` — must match one of the given criteria 539 | - Default: `1` (`Int`) (i.e. match exactly one (1) item) 540 | - `missing_ok=Bool` — (optional) short-hand for `expects="0+"` 541 | 542 | Notes: 543 | - this annotation is only applicable to map items. If it is specified on any other kind of node, 544 | it is an error. 545 | - only target map items that have not matched with prior schema map items are subject to matching 546 | via `allowed`. That is, map item matching occurs in the order that they appear in the schema. 547 | - the re-use of `expects` from Overlays is deliberate: in both cases, the parameter captures 548 | how many of a given item are expected. 549 | 550 | _Example 1: Required with any key_ 551 | 552 | To require a map item with unspecified key, containing an array of strings: 553 | 554 | ```yaml 555 | connection_options: 556 | #@schema/key allowed="any" 557 | _: 558 | - "" 559 | ``` 560 | 561 | __ 562 | 563 | _Example 2: Optional with any key_ 564 | 565 | To specify an optional map item with unspecified key, containing an array of strings: 566 | 567 | ```yaml 568 | connection_options: 569 | #@schema/key allowed="any" expects=[0,1] 570 | _: 571 | - "" 572 | ``` 573 | 574 | __ 575 | 576 | _Example 3: Any number of keys_ 577 | 578 | To specify any number of map items with unspecified keys, each containing an array of strings: 579 | 580 | ```yaml 581 | connection_options: 582 | #@schema/key allowed="any" expects="1+" 583 | _: 584 | - "" 585 | ``` 586 | 587 | __ 588 | 589 | _Example 4: Specific possible keys_ 590 | 591 | To specify a map item with a key of a specific set, containing an array of strings: 592 | 593 | ```yaml 594 | connection_options: 595 | #@schema/key allowed=lambda key,_: key in ["aws", "gcp", "azure"] 596 | _: 597 | - "" 598 | ``` 599 | 600 | __ 601 | 602 | _Example 5: Required suffix 603 | 604 | To specify a map item whose key must have a specific suffix, each containing an array of strings: 605 | 606 | ```yaml 607 | connection_options: 608 | #@schema/key allowed=lambda key, suffix: key.endswith(suffix) 609 | _db: 610 | - "" 611 | ``` 612 | 613 | __ 614 | 615 | _Example 6: Optional key 616 | 617 | To specify an optional map item. If present, must match the specified key and shape. 618 | 619 | ```yaml 620 | connection_options: 621 | #@schema/key missing_ok=True 622 | bucket: 623 | access_key: "" 624 | secret_key: "" 625 | ``` 626 | 627 | 628 | ## Part 6: Documenting Schema 629 | 630 | Configuration Authors may document individual items in a Schema to clarify their meaning, purpose, and impact. 631 | They may also wish to decorate schema with descriptive information that can generate documentation or a User Interface 632 | designed to ease configuration. 633 | 634 | ### @schema/title 635 | 636 | Sets the user-friendly name or title of the node. 637 | 638 | ```yaml 639 | @schema/title title 640 | ``` 641 | 642 | - `title` (`string`) — the short and human-readable name for the annotated node. This name should be suitable for 643 | the name of a field in a UI or the "property name" column in a table in documentation (< ~20 characters) 644 | 645 | Note: 646 | - the default title for a map item is its key name where non-alphanumeric are replaced with spaces and the first letter is capitalized 647 | similar to [rails humanize functionality](https://apidock.com/rails/String/humanize). 648 | 649 | _Example:_ 650 | 651 | ```yaml 652 | #@schema/title "Account Password" 653 | password: "" 654 | ``` 655 | 656 | __ 657 | 658 | 659 | ### @schema/doc 660 | 661 | Sets the text describing the purpose of the node. 662 | 663 | ```yaml 664 | @schema/doc documentation 665 | ``` 666 | 667 | - `documentation` (`string`) — a more thorough description of the node or section that shares the context 668 | of the node, enumerates the range of possible values, explaining the effects of certain value ranges. 669 | - often these strings include formatting typography (e.g. HTML or Markdown) to support display of these 670 | values in documentation settings. 671 | 672 | _Example 1: Short description_ 673 | 674 | ```yaml 675 | #@schema/doc "The user password used to log in to the system" 676 | user_password: "" 677 | ``` 678 | 679 | __ 680 | 681 | _Example 2: Multi-line description_ 682 | 683 | Take advantage of Starlark's multi-line string syntax to make a full description readable within the 684 | schema document. 685 | 686 | ```yaml 687 | #@ documentation = \ 688 | #@ ''' Optional configuration if using a private registry that has all cf-for-k8s images relocated to 689 | #@ see 690 | #@ ''' 691 | #@schema/doc documentation 692 | system_registry: 693 | ``` 694 | 695 | __ 696 | 697 | _Example 3: HTML-formatted description_ 698 | 699 | When the primary consumption of documentation will be in a formatted form, the Configuration Author 700 | can use that format. 701 | 702 | ```yaml 703 | #@ documentation = \ 704 | #@ '''

Optional configuration if using a private registry that has all cf-for-k8s images relocated to 705 | #@ see docs/platform_operators/system-registry.md 706 | #@ ''' 707 | #@schema/doc documentation 708 | system_registry: 709 | ``` 710 | 711 | 712 | 713 | ### @schema/example 714 | 715 | Captures an exemplary value for the annotated node. 716 | 717 | ```yaml 718 | @schema/example sample_value 719 | ``` 720 | 721 | - `sample_value` (_type matching the node_) — a sample value that illustrates a common use. The value must be of the 722 | effective type of the node; to specify a value of another type is an error. 723 | - due to current limitations within `ytt`'s annotation system, only one example can be given: this annotation cannot 724 | be repeated. To supply multiple examples, use the [`@schema/examples`](#schemaexamples) annotation. 725 | 726 | _Example:_ 727 | 728 | ```yaml 729 | #@schema/example "my_domain.example.com" 730 | system_domain: "" 731 | ``` 732 | 733 | __ 734 | 735 | 736 | ### @schema/examples 737 | 738 | Captures multiple exemplary values for the annotated node. 739 | 740 | ```yaml 741 | @schema/examples (exampleA_description, exampleA_value)[, (exampleB_description, exampleB_value)] 742 | ``` 743 | 744 | - `(exampleX_description, exampleX_value)` ([`tuple`](https://github.com/google/starlark-go/blob/master/doc/spec.md#tuples)) — 745 | the example pair: 746 | - `exampleX_description` (`string`) — naming and describing the purpose or impact of the example value. 747 | - `exampleX_value` (_type matching the node_) — a sample value that illustrates a common use. The value must be of the 748 | effective type of the node; to specify a value of another type is an error. 749 | 750 | 751 | _Example 1: Simple examples_ 752 | 753 | ```yaml 754 | #@schema/examples ("One domain across the foundation", ["apps.example.com"]), ("Multi-tenant foundation", ["pets.com", "nile.com"] ) 755 | apps_domain: [] 756 | ``` 757 | 758 | __ 759 | 760 | _Example 2: Complex examples_ 761 | 762 | Illustrates how different databases require different amounts of configuration. 763 | 764 | ```yaml 765 | #@ def postgres_example(): 766 | name: uaa 767 | host: uaa-db.svc.cluster.local 768 | user: postgres 769 | secretRef: 770 | name: uaa-db-creds 771 | #@ end 772 | 773 | #@ def mysql_example(): 774 | name: uaa 775 | adapter: mysql 776 | host: uaa-db.svc.cluster.local 777 | port: 3306 778 | user: root 779 | secretRef: 780 | name: uaa-db-creds 781 | #@ end 782 | 783 | #@schema/examples ("Postgres", postgres_example()), ("MySQL", mysql_example()) 784 | databases: [] 785 | ``` 786 | 787 | 788 | ### @schema/deprecated 789 | 790 | Marks a node as targeted for removal while still remaining defined and used. 791 | 792 | ```yaml 793 | @schema/deprecated notice 794 | ``` 795 | 796 | - `notice` (`string`) — a message elaborating on the reason for the deprication, how to prepare for 797 | the removal (e.g. migrate to using a newly introduced value), and possibly a version-based timeline 798 | for when the removal will happen. 799 | 800 | Notes: 801 | - when a value is specified for the node outside of the schema, a standardized warning that includes 802 | `notice` will be output to standard error, and evaluation continues. 803 | 804 | _Example:_ 805 | 806 | ```yaml 807 | load_balancer: 808 | #@schema/deprecated "Will be removed in 2.0.0; in that version, specify `null` for `load_balancer` to disable." 809 | enable: true 810 | static_ip: "" 811 | ``` 812 | 813 | __ 814 | 815 | ### @schema/removed 816 | 817 | ```yaml 818 | @schema/removed remedy 819 | ``` 820 | 821 | - `remedy` (`string`) — a message providing next step instructions on how to repair the document/configuration. 822 | 823 | Notes: 824 | - when a value is specified for the node outside of the schema, a standardized warning that includes 825 | `remedy` will be output to standard error, and evaluation stops. 826 | 827 | _Example:_ 828 | 829 | ```yaml 830 | load_balancer: 831 | #@schema/removed "Was removed as of 2.0.0; specify `null` for `load_balancer` to disable." 832 | enable: true 833 | static_ip: "" 834 | ``` 835 | 836 | 837 | ## Part 7: Validating Documents 838 | 839 | Beyond specifying a type for a value, one can specify more dynamic constraints on the value via validations. 840 | 841 | ### @schema/validate 842 | 843 | ```yaml 844 | @schema/validate [...user-provided-funcs,] [...builtin-kwargs] 845 | ``` 846 | 847 | This annotation specifies how to validate value. `@schema/validate` will provide a set of keyword arguments which map to built-in validations, such as `max`, `max_len`, etc., but will also accept user-defined validation functions. Less common validators will also be provided via a `validations` library. For example, 848 | 849 | ```yaml 850 | #@ def number_is_even(num): return (num % 2 == 0, "Number is not even") 851 | 852 | #@schema/validate number_is_even, min=2 853 | replicas: 6 854 | ``` 855 | 856 | will use the `min` predefined validator as well as the user-provided `number_is_even` function for validations. Function arguments should have a signature which matches the signature of custom functions passed to [`@overlay/assert`](https://github.com/k14s/ytt/blob/develop/docs/lang-ref-ytt-overlay.md). 857 | 858 | Full list of builtin validations exposed via kwargs (all have 2nd form: a tuple of value and error message): 859 | 860 | - min=int 861 | - max=int 862 | - min_len=int 863 | - max_len=int 864 | - enum=[string|int|...]: any set of values 865 | - regexp=string (TBD or pattern?) -> golang regexp 866 | - format=string (mostly copied from JSON Schema) 867 | - date-time: Date and time together, for example, 2018-11-13T20:20:39+00:00. 868 | - time: New in draft 7 Time, for example, 20:20:39+00:00 869 | - date: New in draft 7 Date, for example, 2018-11-13. 870 | - email: Internet email address, see RFC 5322, section 3.4.1. 871 | - hostname: Internet host name, see RFC 1034, section 3.1. 872 | - ip: IPv4 or IPv6 873 | - ipv4: IPv4 address, according to dotted-quad ABNF syntax as defined in RFC 2673, section 3.2. 874 | - ipv6: IPv6 address, as defined in RFC 2373, section 2.2. 875 | - uri: A universal resource identifier (URI), according to RFC3986. 876 | - port: >= 0 && <= 65535 877 | - percent: k8s's IsValidPercent e.g. "10%" 878 | - duration: Golang's duration e.g. "10h" 879 | - not_null=bool to verify value is not null 880 | - unique=bool to verify value contains unique elements (TBD?) 881 | - starts_with=string to verify string has prefix 882 | - ends_with=string to verify string has suffix 883 | - TBD 884 | 885 | Full list of builtin validations included in a library: 886 | 887 | - ...kwargs validations as functions... 888 | - base64.decodable() 889 | - json.decodable() 890 | - yaml.decodable() 891 | - regexp.is_valid() 892 | 893 | String values, by default, have `@schema/validate min_len=1` validation added. 894 | 895 | #### Validation Documentation 896 | 897 | Configuration Consumers are more likely to provide valid values if they are aware 898 | of what defines valid. 899 | 900 | For this reason, validations should provide user-friendly documentation that makes 901 | this clear: 902 | - concise phrase that describes a valid value; 903 | - a short list of common example valid values; 904 | - citation to canonical definition. 905 | 906 | TBD. 907 | 908 | Sketch: 909 | - invoke the validation function with no arguments 910 | - if it returns a `string` that is documentation (including the name) 911 | - if it returns an error, discard and replace with default documentation (e.g. "Validation: "+str(validation_func)) 912 | - first class `ytt` Schema functions can include this behavior 913 | - 3rd party functions can be wrapped in a function (or lambda expression) to decorate this behavior: 914 | ```python 915 | def is_valid_ip(val="__yttdoc__"): 916 | if val != "__yttdoc__": 917 | return thirdPartyLib.valid_ip(val) 918 | else: 919 | return "Valid IP: a well-formed IPv4 (RFC 2673, section 3.2) or IPv6 address (RFC 2373, section 2.2)." 920 | end 921 | ``` 922 | 923 | **Implementation considerations:** 924 | 925 | 926 | As part of the implementation of the validation feature we should use some of the knowledge from the Type Checking 927 | implemented before. 928 | - Isolation of packages like yamlmeta from external dependencies 929 | 930 | This package is intended as a place for the yaml internal structures 931 | - Eventually it make sense to implement the validation near the Schemas and also move the Type Check to the same package 932 | 933 | When implementing we should consider to create a walker that iterate over the YAML document to do the Check or the Validation depending on the action we want to perform in a given moment 934 | 935 | ## Part 8: Command-Line Interface Interactions 936 | 937 | :::warning 938 | There is active User Research into how to more effectively communicate with `ytt` users. Findings from that research 939 | should trump whatever "requirements" are specified below. 940 | ::: 941 | 942 | Schema violation messages should be: 943 | - readable — easy/quick to determine the nature and details of the violation. 944 | - actionable — clear what step would need to be taken to fix the violation. 945 | - complete — no other context is required to fix the violation. 946 | - supportive — informs, without blame. 947 | 948 | To get there, consider these principles: 949 | - convey concisely — the fewer words written, the more words read. 950 | - be consistent — use the same word for the same concept; the same layout for the same kind of message. 951 | - prioritize position of information — place more important information in a more prominent spot. 952 | - be empathetic — use words the user must already know; when you cannot, define it or point to a definition. 953 | 954 | ### Type Violations 955 | 956 | When a value given is malformed (wrong structure/type) `ytt` will report these violations and stop processing. 957 | 958 | Type violations can come from: 959 | - an incorrectly formed explicit default value (i.e. the value of `@schema/default`) 960 | - a data value (noted below: all `@data/values` documents are implied overlays) 961 | 962 | `schema.yml` 963 | ```yaml= 964 | #@schema/definition name="data/values" 965 | --- 966 | system_domain: "" 967 | 968 | load_balancer: 969 | enabled: true 970 | external_ip: "" 971 | ``` 972 | 973 | `values.yml` 974 | ```yaml= 975 | #@data/values 976 | --- 977 | system_domain: false 978 | load_balancer: true 979 | ``` 980 | 981 | ``` 982 | $ ytt -f schema.yml -f values.yml 983 | ytt: Error: ... 984 | 985 | Result of values.yml did not pass "data/values" schema: 986 | 987 | values.yml:3 | system_domain: 988 | | ^^^ found: boolean 989 | | expected: string (by schema.yml:3) 990 | ``` 991 | 992 | **Note:** 993 | - the document at `values.yml:2` implies `@overlay/match by=overlay.all` and `@overlay/match-child-defaults missing_ok=True` 994 | - `values.yml` has two overlay operations: merge of `system_domain` and merge of `load_balancer`. 995 | - there are, in fact, two type violations. However, `ytt` stops after the first overlay operation fails. 996 | 997 | ### Validation Violations 998 | 999 | ## Part 9: Interactions with Existing Features 1000 | 1001 | ### Changes to Data Values 1002 | 1003 | 1004 | #### Data Value Overlays Become More Permissive 1005 | 1006 | Prior to Schemas, Data Values protected against mistakes like typo'ed key names through the strict merge of additional data values documents: to successfully merge a new key, it must be marked as `@overlay/match missing_ok=True`. 1007 | 1008 | The responsibility to ensure key names are valid are shifted from Data Values to Schema. 1009 | 1010 | When schema is present, Data Values documents imply an automatic merge of new keys. This is equivalent to all `@data/values` documents being annotated with `@overlay/match-child-defaults missing_ok=True`. 1011 | 1012 | This behavior should not result in an error if a Data Values document _already_ is annotated this way. 1013 | 1014 | 1015 | #### All Data Values Become Overlays 1016 | 1017 | Before Schema, the first YAML document annotated with `@data/values` effectively defined both an implied schema and defaults. 1018 | 1019 | With Schema, the base data values document is produced by collecting all the net default values from the schema, itself. 1020 | 1021 | `ytt` will automatically convert _all_ data values into overlays. 1022 | 1023 | 1024 | ### Overlays 1025 | 1026 | Schema aims to provide the earliest possible and actionable feedback so that Configuration Consumers can successfully complete their task in the shortest possible duration. 1027 | 1028 | When one or more schema violation(s) occur after a Data Value overlay operation, `ytt` should: 1029 | 1. display the current state of the violating portion of the document; (it's quite possible, this data does not appear anywhere else) 1030 | 2. report the violation(s); and 1031 | 3. stop processing. 1032 | 1033 | 1034 | ### Overlays of Schema 1035 | 1036 | TBD 1037 | 1038 | - It must be possible to extend Schema (i.e. add new items) 1039 | - When do Configuration Consumers need to modify Schema? 1040 | - they add templates that require additional data values 1041 | - soften or remove overzealous validations 1042 | - when overlays on templates change the type of the expected data values 1043 | 1044 | ## Open Questions 1045 | 1046 | - Given that Data Values all become overlays, now all array items require a `@overlay/append` 1047 | - this will be a negative UX for brand new users as it lights a number of tool features just so that I can specify an array for my library. :( 1048 | - this has been a pain 1049 | - How could we assign line numbers to YAML nodes that are generated from Starlark code? 1050 | - how common of a situation is this? 1051 | - becomes more important when referencing nodes in schema because our violation messages depend on naming the type/structure declaration by line number. 1052 | - Overlaying Schema implies the need for meta to be copied along 1053 | - Documentation could be enhanced if validation rules were included. How could validation authors provide documentation for their validation? 1054 | - provide a magic string for the input and the function returns documentation? 1055 | - function always returns documentation as part of return value? 1056 | - How should schema work with ytt libraries? 1057 | - How do we support data values that are targeting a dependency library? 1058 | - How will code defined in a schema file make its way into the execution context of the target "template"? 1059 | - e.g. function supporting validation 1060 | - idea: could we capture the "globals" resulting from the execution of the schema "template" and feed those into the target "template"'s execution context? 1061 | - there are expressions/statements to consume during the schema "template" construction... but other expression/statements that need to be made available to target "templates". 1062 | - Should we support generating `ytt` Schema from other popular sources (e.g. JSON Schema, OpenAPI v3 schema)? 1063 | - how to add more things to schema? (via overlays similar to data values) 1064 | - provide programmatic schema.apply() (similar to overlay.apply) 1065 | - respect schema for data values set via cmd line flags/env vars 1066 | - add --data-values-schema-inspect (similar to --data-values-inspect) 1067 | - a component team introduces breaking change… how do we discover that happened? (is that a use case that's in scope for ytt?) 1068 | - schema diff 1069 | - as a Configuration Consumer, what drives me asking for a diff? what am I looking for? 1070 | - what do we get from YAML diffing tools? 1071 | - what is the gap between the need and what we get with diffing tools? 1072 | - using extensions for ytt types. 1073 | - https://github.com/k14s/ytt/issues/51 1074 | 1075 | 1076 | # Sample Usage 1077 | 1078 | Vetting of User eXperience by way of examples. 1079 | 1080 | ## Common Use Cases 1081 | 1082 | Expectedly common situations that invoke more than one feature, illustrating proper interactions. 1083 | 1084 | ### Mutually Exclusive Map Configuration 1085 | 1086 | ## Edge Cases 1087 | 1088 | Situations that stretch the capabilities of the feature set. 1089 | 1090 | ### Prometheus Label Set Rewriting 1091 | 1092 | Prometheus includes a feature allowing operators to articulate label rewritting rules for metrics. 1093 | This feature is used throughout Prometheus' configuration. It is centered around a structure 1094 | known as a "relabeling". 1095 | 1096 | Defining edits that operate on runtime text is frought with errors. 1097 | - a minor typo of a keyword, 1098 | - a regular expression that fails only in certain cases, or 1099 | - a sometimes required configuration being omitted 1100 | 1101 | This example illustrates how Schema would enhance the Prometheus User's configuration experience. 1102 | 1103 | Sources: 1104 | - https://github.com/prometheus-community/helm-charts/blob/451448b53/charts/kube-prometheus-stack/values.yaml 1105 | - https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config 1106 | 1107 | ```yaml= 1108 | #! ==- Relabel Config -== 1109 | #! 1110 | 1111 | #@ def replace_has_target_label(relabel): 1112 | #@ if relabel.action == "replace" and "targetLabel" not in relabel: 1113 | #@ return (False, "'targetLabel' not found on replace relabel rule") 1114 | #@ end 1115 | #@ end 1116 | 1117 | #@ def non_hashmod_has_regex(relabel): 1118 | #@ if relabel.action in ["replace", "keep", "drop", "labelmap", "labeldrop", "labelkeep"] and \ 1119 | #@ "regex" not in relabel: 1120 | #@ return (False, "'regex' not found on {} relabel rule".format(relabel.action)) 1121 | #@ end 1122 | #@ end 1123 | 1124 | #! As of Prometheus 2.22, there are 200+ meta labels 1125 | #@ known_meta_labels = { 1126 | #@ "__meta_azure_machine_id": True, 1127 | #@ "__meta_azure_machine_location": True, 1128 | #@ "__meta_azure_machine_name": True, 1129 | #@ "__meta_azure_machine_os_type": True, 1130 | #@ "__meta_azure_machine_private_ip": True, 1131 | #! ... and many more 1132 | #@ } 1133 | #@ def known_meta_label(label): 1134 | #@ if label.startswith("__meta_"): 1135 | #@ if known_meta_labels[label] == None: 1136 | #@ return (False, 1137 | #@ "{} is not a known meta label (did you mean {}?)"\ 1138 | #@ .format(label, spelling.nearest(label, known_meta_labels.keys()))) 1139 | #@ end 1140 | #@ end 1141 | #@ end 1142 | 1143 | --- 1144 | #@ def relabeling(): 1145 | #@schema/title "Source labels" 1146 | #@ doc = ''' 1147 | #@ The source labels select values from existing labels. Their content is concatenated 1148 | #@ using the configured separator and matched against the configured regular expression 1149 | #@ for the replace, keep, and drop actions. 1150 | #@ ''' 1151 | #@schema/doc doc 1152 | sourceLabels: 1153 | #@schema/validate known_meta_label, regexp=("[a-zA-Z_][a-zA-Z0-9_]*", "letters and numbers, starting with a letter") 1154 | - "" 1155 | 1156 | #@schema/title "Source Label Separator" 1157 | #@schema/doc "Separator placed between concatenated source label values." 1158 | separator: ; 1159 | 1160 | #@schema/title "Target Label" 1161 | #@ doc = ''' 1162 | #@ Label to which the resulting value is written in a replace action. 1163 | #@ It is mandatory for replace actions. Regex capture groups are available. 1164 | #@ ''' 1165 | #@schema/doc 1166 | #@schema/key missing_ok=True 1167 | #@schema/validate regexp=("[a-zA-Z_][a-zA-Z0-9_]*", "letters and numbers, starting with a letter") 1168 | targetLabel: "" 1169 | 1170 | #@schema/title "Extraction regex" 1171 | #@schema/doc "Regular expression against which the extracted value is matched." 1172 | #@schema/key missing_ok=True 1173 | #@schema/validate regexp.is_valid 1174 | regex: ^(.*)$ 1175 | 1176 | #@schema.title "" 1177 | #@schema/key missing_ok=True 1178 | modulus: 0 1179 | 1180 | replacement: $1 1181 | 1182 | #@schema/validate enum=["replace", "keep", "drop", "hashmod", "labelmap", "labeldrop", "labelkeep"] 1183 | action: replace 1184 | #@ end 1185 | 1186 | #@ def relabelings(): 1187 | #@schema/validate replace_has_target_label, non_hashmod_has_regex 1188 | - #@ relabeling() 1189 | #@ end 1190 | 1191 | #! metrics_path is required to match upstream rules and charts 1192 | #@ def metrics_path_relabel(): 1193 | sourceLabels: [__metrics_path__] 1194 | targetLabel: metrics_path 1195 | #@ end 1196 | 1197 | --- 1198 | kubelet: 1199 | serviceMonitor: 1200 | 1201 | #@schema/title "Scrape interval" 1202 | #@schema/doc "If not set, the Prometheus default scrape interval is used." 1203 | #@schema/validate format="duration" 1204 | #@schema/key missing_ok=True 1205 | interval: "" 1206 | 1207 | #@schema/doc "relabel configs to apply to cAdvisor metric samples before ingestion." 1208 | cAdvisorMetricRelabelings: #@ relabelings() 1209 | 1210 | #@schema/doc "relabel configs to apply to probe metric samples before ingestion." 1211 | probesMetricRelabelings: #@ relabelings() 1212 | 1213 | #@schema/doc "relabel configs to apply to cAdvisor samples before ingestion." 1214 | #@schema/default [metrics_path_relabel()] 1215 | cAdvisorRelabelings: #@ relabelings() 1216 | 1217 | #@schema/doc "relabel configs to apply to probe samples before ingestion." 1218 | #@schema/default [metrics_path_relabel()] 1219 | probesRelabelings: #@ relabelings() 1220 | 1221 | #@schema/doc "relabel configs to apply to resource samples before ingestion." 1222 | #@schema/default [metrics_path_relabel()] 1223 | resourceRelabelings: #@ relabelings() 1224 | 1225 | #@schema/doc "relabel configs to apply to metric samples before ingestion." 1226 | metricRelabelings: #@ relabelings() 1227 | 1228 | #@schema/doc "relabel configs to apply to samples before ingestion." 1229 | #@schema/default [metrics_path_relabel()] 1230 | relabelings: #@ relabelings() 1231 | ``` 1232 | 1233 | Pros: 1234 | - demonstrates the leverage functions bring: to extract the concept of a "relabeling", attach defaults and validations, 1235 | and use widely. 1236 | - trickier configuration (e.g. regular expressions, exact meta label names, string formats) are not just caught earlier, 1237 | but with context for more actionable error messages. 1238 | - ... 1239 | 1240 | Cons: 1241 | - validation rules across the map + the map is an array item ==> the definition of "relabeling" had to be extracted and scoped to a separate document. 1242 | - ... 1243 | 1244 | 1245 | ## Migration of Existing Configuration 1246 | ### Cloud Foundry for Kubernetes 1247 | 1248 | Cloud Foundry for Kubernetes — at the time of writing — is the most sophisticated use of `ytt` known. 1249 | In the absence of the Schema feature specified in this document, the CF for K8s project has written 1250 | their own custom validation logic. 1251 | 1252 | With `ytt` Schemas, _all_ existing validation logic ought to be absorbed into the schema definition. 1253 | 1254 | Specifically: 1255 | - "required" parameters : all string values imply a `@schema/validate min_len=1` by default 1256 | - a "non empty array" is `@schema/validate min_len=1` 1257 | - _(others ?)_ 1258 | 1259 | ```yaml= 1260 | #@schema/definition name="data/values" 1261 | --- 1262 | #@schema/doc "your system domain" 1263 | #@schema/example "system.cf.example.com" 1264 | #@schema/validate format="hostname" 1265 | system_domain: "" 1266 | 1267 | #@schema/title "Application Domains" 1268 | #@schema/example ["apps.cf.example.com"] 1269 | #@schema/validate min_len=1 1270 | app_domains: 1271 | - 1272 | #@schema/validate format="hostname" 1273 | "" 1274 | 1275 | #@schema/title "Application Log Distinations" 1276 | app_log_destinations: [] 1277 | 1278 | #@schema/title "Cloud Foundry Administrator Password" 1279 | #@schema/doc "password for admin user (in plain text)" 1280 | cf_admin_password: 1281 | 1282 | load_balancer: 1283 | enable: true 1284 | #@schema/title "Load Balancer Static IP" 1285 | #@schema/doc "Reserved static IP for Load Balancer" 1286 | #@schema/validation format="ip" 1287 | static_ip: "" 1288 | 1289 | system_certificate: 1290 | #@schema/title "System Certificate" 1291 | #@ doc = '''\ 1292 | #@ Certificate for the wildcard subdomain of the system domain. 1293 | #@ For example, CN=*.system.cf.example.com 1294 | #@ ''' 1295 | #@schema/doc doc 1296 | #@schema/validate base64.decodable 1297 | crt: "" 1298 | #@schema/title "Private Key" 1299 | #@schema/doc "Private key for the system certificate" 1300 | #@schema/validate base64.decodable 1301 | key: "" 1302 | #@schema/title "CA Certificate" 1303 | #@schema/doc "CA certificate used to sign the system certificate" 1304 | #@schema/validate base64.decodable 1305 | ca: "" 1306 | 1307 | workloads_certificate: 1308 | #@schema/title "Applications Certificate" 1309 | #@ doc = '''\ 1310 | #@ Certificate for the wildcard subdomain of the applications domain. 1311 | #@ For example, CN=*.apps.cf.example.co 1312 | #@ ''' 1313 | #@schema/doc doc 1314 | #@schema/validate base64.decodable 1315 | crt: "" 1316 | #@schema/title "Private Key" 1317 | #@schema/doc "Private key for the applications certificate" 1318 | #@schema/validate base64.decodable 1319 | key: "" 1320 | #@schema/title "CA Certificate" 1321 | #@schema/doc "CA certificate used to sign the applications certificate" 1322 | #@schema/validate base64.decodable 1323 | ca: "" 1324 | 1325 | gateway: 1326 | #@schema/doc "When true, automatically upgrades incoming HTTP connections to HTTPS" 1327 | https_only: true 1328 | 1329 | capi: 1330 | #@schema/title "CAPI DB" 1331 | #@schema/doc "Database for Cloud API" 1332 | database: 1333 | #@schema/validation enum=["postgres", "mysql2"] 1334 | adapter: postgres 1335 | encryption_key: "" 1336 | host: "cf-db-postgresql.cf-db.svc.cluster.local" 1337 | #@schema/examples [("Postgres", 5432), ("MySQL", 3306)] 1338 | port: 5432 1339 | user: cloud_controller 1340 | #@schema/doc "DB user password in plain text" 1341 | password: "" 1342 | name: cloud_controller 1343 | #@schema/doc "Plain text ca certificate if tls should be used" 1344 | ca_cert: "" 1345 | log_level: info 1346 | 1347 | uaa: 1348 | #@schema/title "UAA DB" 1349 | #@schema/doc "Database for User Authentication and Authorization service" 1350 | database: 1351 | #@schema/validation enum=["postgres", "mysql2"] 1352 | adapter: postgres 1353 | encryption_key: "" 1354 | host: "cf-db-postgresql.cf-db.svc.cluster.local" 1355 | #@schema/examples [("Postgres", 5432), ("MySQL", 3306)] 1356 | port: 5432 1357 | user: uaa 1358 | #@schema/doc "DB user password in plain text" 1359 | password: "" 1360 | name: uaa 1361 | #@schema/doc "Plain text ca certificate if tls should be used" 1362 | ca_cert: "" 1363 | 1364 | app_registry: 1365 | #@schema/validate format="hostname" 1366 | hostname: "" 1367 | repository_prefix: "" 1368 | username: "" 1369 | password: "" 1370 | 1371 | remove_resource_requirements: false 1372 | add_metrics_server_components: false 1373 | allow_prometheus_metrics_access: false 1374 | use_external_dns_for_wildcard: false 1375 | enable_automount_service_account_token: false 1376 | metrics_server_prefer_internal_kubelet_address: false 1377 | use_first_party_jwt_tokens: false 1378 | 1379 | #@schema/title "CF Blobstore" 1380 | #@schema/doc "configuration for the blobstore holding buildpacks, droplets, packages, etc." 1381 | blobstore: 1382 | endpoint: "http://cf-blobstore-minio.cf-blobstore.svc.cluster.local:9000" 1383 | region: "''" 1384 | access_key_id: "admin" 1385 | secret_access_key: "" 1386 | package_directory_key: "cc-packages" 1387 | droplet_directory_key: "cc-droplets" 1388 | resource_directory_key: "cc-resources" 1389 | buildpack_directory_key: "cc-buildpacks" 1390 | aws_signature_version: "2" 1391 | 1392 | #@ doc = '''\ 1393 | #@ optional configuration if using a private registry that has all cf-for-k8s images relocated to 1394 | #@ see 1395 | #@ ''' 1396 | #@schema/doc doc 1397 | system_registry: 1398 | add_image_pull_secrets: false 1399 | hostname: "" 1400 | username: "" 1401 | password: "" 1402 | ``` 1403 | 1404 | ### Minio 1405 | 1406 | Partial minio config (based on https://github.com/helm/charts/blob/21e2e1b1f2656c785ece0ec741b047b21539d7b1/stable/minio/values.yaml#L244), shows how to have one-of available options. 1407 | 1408 | ```yaml 1409 | #@ load("@ytt:json", "json") 1410 | 1411 | #@schema/definition name="data/values" 1412 | --- 1413 | #@schema/validate lambda g: (g.s3 or g.azure or g.gcs or g.oss, "Must specify one gateway: s3, azure, gcs, oss") 1414 | gateway: 1415 | #@schema/nullable 1416 | s3: 1417 | #@schema/validate min=1 1418 | replicas: 4 1419 | #@schema/validate prefix="https://" 1420 | serviceEndpoint: "" 1421 | accessKey: "" 1422 | secretKey: "" 1423 | 1424 | #@schema/doc "Use minio as an azure blob gateway, you should disable data persistence so no volume claim are created. https://docs.minio.io/docs/minio-gateway-for-azure" 1425 | #@schema/nullable 1426 | azure: 1427 | #@schema/doc "Number of parallel instances" 1428 | #@schema/validate min=1 1429 | replicas: 4 1430 | 1431 | #@schema/doc "Use minio as GCS (Google Cloud Storage) gateway, you should disable data persistence so no volume claim are created. https://docs.minio.io/docs/minio-gateway-for-gcs" 1432 | #@schema/nullable 1433 | gcs: 1434 | #@schema/doc "Number of parallel instances" 1435 | #@schema/validate min=1 1436 | replicas: 4 1437 | #@schema/doc "credential json file of service account key" 1438 | #@schema/validate lambda val: (json.decodable(val), "key must be valid JSON") 1439 | gcsKeyJson: "" 1440 | #@schema/doc "Google cloud project-id" 1441 | projectId: "" 1442 | 1443 | #@schema/nullable 1444 | oss: 1445 | #@schema/doc "Number of parallel instances" 1446 | #@schema/validate min=1 1447 | replicas: 4 1448 | #@schema/validate prefix="https://" 1449 | endpointURL: "" 1450 | ``` 1451 | 1452 | 1453 | 1454 | ### Dex 1455 | 1456 | TODO: review and inline [Dex](example-partial-dex.yaml). 1457 | 1458 | 1459 | ## Complex Examples 1460 | 1461 | This snippet will result in an array with a single element, type `bucket`, with `versioning` defaulted to `Enabled`. 1462 | 1463 | ```yaml 1464 | #@ def default_bucket(): 1465 | versioning: "Enabled" 1466 | #@ end 1467 | 1468 | #@schema/default [default_bucket()] 1469 | bucket: 1470 | - name: "" 1471 | versioning: "" 1472 | access: "" 1473 | ``` 1474 | 1475 | 1476 | Array examples: 1477 | 1478 | ```yaml 1479 | service: 1480 | #@schema/doc "List of IP addresses at which the Prometheus server service is available. Ref: https://kubernetes.io/docs/user-guide/services/#external-ips" 1481 | externalIPs: 1482 | #@schema/validate format="ip" 1483 | - "" 1484 | 1485 | imagePullSecrets: 1486 | - name: "" 1487 | 1488 | ingress: 1489 | #@schema/default ["chart-example.local"] 1490 | hosts: 1491 | - "" 1492 | 1493 | tls: 1494 | - secretName: "" 1495 | #@schema/validate min=1 1496 | hosts: 1497 | - "" 1498 | 1499 | tolerations: 1500 | #@schema/type "any" 1501 | - null 1502 | 1503 | #@schema/doc "Create multiple buckets after minio install. Enabling `defaultBucket` will take priority over this list" 1504 | #@schema/example [{"name": "bucket1"}] 1505 | buckets: 1506 | - name: "" 1507 | #@schema/validate enum=["none"] 1508 | policy: none 1509 | purge: false 1510 | ``` 1511 | 1512 | map[string]string examples: 1513 | 1514 | ```yaml 1515 | #@ def validate_map_str_to_str(vals): 1516 | #@ type(vals) == "dict" or assert.fail("Expected key to be map") 1517 | #@ for key in vals: 1518 | #@ type(key) == "string" or assert.fail("Expected key to be string") 1519 | #@ type(vals[key]) == "string" or assert.fail("Expected val to be string") 1520 | #@ end 1521 | #@ end 1522 | 1523 | #@schema/type "any" 1524 | #@schema/validate validate_map_str_to_str 1525 | #@schema/example {"prometheus.io/scrape": "true", "prometheus.io/path": "/minio/prometheus/metrics", "prometheus.io/port": "9000"} 1526 | annotations: {} 1527 | 1528 | #! ... 1529 | 1530 | #@schema/example {"prometheus.io/scrape": "true", "prometheus.io/path": "/minio/prometheus/metrics", "prometheus.io/port": "9000"} 1531 | annotations: 1532 | #@schema/any-key 1533 | _: "" 1534 | ``` 1535 | 1536 | Type any with default (TBD, should we force schema/default?): 1537 | 1538 | ```yaml 1539 | #@schema/type "any" 1540 | annotations: 1541 | foo: bar 1542 | blah: 123 1543 | ``` 1544 | 1545 | Example of additional free-form properties: 1546 | 1547 | ```yaml 1548 | annotations: 1549 | #@schema/any-key 1550 | _: "" 1551 | 1552 | db_opts: 1553 | username: "" 1554 | password: "" 1555 | #@schema/any-key 1556 | _: 1557 | #@schema/type "any" 1558 | value: 1559 | ``` 1560 | 1561 | #### Asserting on higher level structure with optional lower key with #@schema/key-may-be-present 1562 | 1563 | ```yaml 1564 | #@schema 1565 | --- 1566 | foo: "" 1567 | 1568 | #@data/values 1569 | --- 1570 | foo: "val" 1571 | 1572 | config.yml: 1573 | --- 1574 | foo: #@ data.values.foo #! never errors with: no member 'foo' on data.values 1575 | ``` 1576 | 1577 | ```yaml 1578 | #@schema 1579 | --- 1580 | #@schema/key missing_ok=True 1581 | foo: 1582 | 1583 | #@data/values 1584 | --- 1585 | foo: "val" 1586 | 1587 | config.yml: 1588 | --- 1589 | foo: #@ yaml.encode(data.values) #! => {}, {"foo": "val"} 1590 | ``` 1591 | 1592 | ```yaml 1593 | #@schema 1594 | --- 1595 | #@schema/key missing_ok=True 1596 | foo: 1597 | max_connections: 100 1598 | username: "" 1599 | 1600 | #@data/values 1601 | --- 1602 | foo: 1603 | username: val 1604 | 1605 | config.yml: 1606 | --- 1607 | foo: #@ yaml.encode(data.values) #! => {}, {"foo": {"username": "val", "max_connections":100}} 1608 | ``` 1609 | 1610 | 1611 | # Design Notes 1612 | 1613 | ## Observations/Guesses 1614 | 1615 | _(Collection of "insights" that might bring cohesion to the overall design.)_ 1616 | 1617 | - In effect, we're implementing a dynamic type checking so that we can perform a sort of "late binding" — attaching defaulting and validation behavior to individual nodes. 1618 | - [Node](https://github.com/k14s/ytt/blob/cbd0c3959/pkg/yamlmeta/ast.go#L10-L33)s contain "coarse-grain" type information (i.e. what kind of YAML node is it?); 1619 | - this proposal effectively extends that abstraction to include "fine-grain" type information 1620 | - Parser _yields_ Nodes 1621 | - Schema determines Node Type 1622 | - Types _check_ Nodes 1623 | 1624 | ## Core Features 1625 | 1626 | There are six core operations: 1627 | 1. [Parsing](#Parsing) — from text file (in YAML format) to tree of yamlmeta.Node(s). 1628 | 2. [Compilation](#Compilation) — compiling YAML template into Starlark. 1629 | 4. [Typing](#Typing) — calculating "Type" for each yamlmeta.Node 1630 | 5. [Checking](#Type-Checking) — perform type check 1631 | 6. [Validating](#Validating) — execute validations 1632 | 7. [Documenting](#DocumentingSchema) — generate metadata suitable as input for templatized documentation. 1633 | 1634 | 1635 | ### Parsing 1636 | 1637 | ```graphviz 1638 | digraph graphname { 1639 | 1640 | yaml_parser [label="yaml Parser (v2)"] 1641 | yaml_node [label="yaml.v2 Node" color=blue fontcolor=blue] 1642 | ytt_yaml_node [label="yamlmeta.Node" color=purple fontcolor=purple] 1643 | ytt_parser [label="ytt 'Parser'"] 1644 | 1645 | yaml_parser -> yaml_node; 1646 | yaml_node -> ytt_parser; 1647 | ytt_parser -> ytt_yaml_node [label="«maps to»"]; 1648 | 1649 | } 1650 | ``` 1651 | _([graphviz reference](https://www.tonyballantyne.com/graphs.html))_ 1652 | 1653 | 1654 | ### Typing 1655 | 1656 | Involves: 1657 | 1. identifying the concrete type for a given node 1658 | 2. copy in default values where missing 1659 | 3. attach "machinery" required to support validation. 1660 | 1661 | Notes: 1662 | - we do not check for violations while typing. 1663 | - typing is made on "best-effort" basis 1664 | - for Data Values, there ought to be one choice of structural types, and sparse choices of leaf values (e.g. `stringOrInt`) 1665 | - for other Document Types, we _might_ want to support multiple possible types for a given node. 1666 | - Schema maps from yamlmeta.Node(s) to yamlmeta.Node(s) 1667 | 1668 | 1669 | ```graphviz 1670 | digraph graphname { 1671 | ytt_yaml_node [label="yamlmeta.Node" color=purple fontcolor=purple] 1672 | schema [label="Schema"] 1673 | type [label="Type"] 1674 | 1675 | schema -> ytt_yaml_node [label="«finds type for»"] 1676 | ytt_yaml_node -> type [label="«has a»" color=gray]; 1677 | schema -> type [label="«has list of allowed»"] 1678 | } 1679 | ``` 1680 | _([graphviz reference](https://www.tonyballantyne.com/graphs.html))_ 1681 | 1682 | 1683 | ### Type-Checking 1684 | 1685 | ```graphviz 1686 | digraph graphname { 1687 | rankdir=LR 1688 | 1689 | ytt_yaml_node [label="yamlmeta.Node" color=purple fontcolor=purple] 1690 | 1691 | 1692 | ytt_yaml_node -> Type [label="«has a»" color=gray fontcolor=gray]; 1693 | 1694 | Value -> Type [label="«is checked against»"]; 1695 | Type -> "Type Check" [label="«results in a»"]; 1696 | } 1697 | ``` 1698 | _([graphviz reference](https://www.tonyballantyne.com/graphs.html))_ 1699 | 1700 | 1701 | ### Validation 1702 | 1703 | 1704 | ### Schema Documentation 1705 | 1706 | - users generate documentation through the `--schema-inspect` flag (this mirrors other `--...-inspect` flags that also produce information). 1707 | - this can be rendered in HTML when user couples this with `--output html` 1708 | 1709 | `schema.yml` 1710 | ```yaml= 1711 | 1712 | #@schema/title "Application Domains" 1713 | #@schema/doc "List of DNS domains used for deployed applications." 1714 | #@schema/example [“apps.cf.example.com”] 1715 | app_domains: 1716 | - "" 1717 | 1718 | uaa: 1719 | #@schema/title "UAA Login Secret" 1720 | #@schema/doc "Secret for an external login server to authenticate to UAA" 1721 | login_secret: "" 1722 | login: 1723 | service_provider: 1724 | #@schema/title "UAA Service Provider Certificate" 1725 | #@schema/doc "X.509 Certificate for UAA's SAML provider" 1726 | certificate: "" 1727 | #@schema/title "UAA Service Provider Private Key" 1728 | #@schema/doc "Private key paired with the X.509 Certificate for UAA's SAML provider" 1729 | key: "" 1730 | 1731 | 1732 | app_registry: 1733 | hostname: 1734 | username: 1735 | password: 1736 | 1737 | 1738 | ``` 1739 | 1740 | ```shell 1741 | $ ytt -f schema.yml --schema-inspect 1742 | schema: 1743 | name: "data/values" 1744 | fields: 1745 | - name: username 1746 | title: Database username 1747 | documentation: The account in the database system to authenticate as. 1748 | example: 1749 | - value: admin 1750 | description: null 1751 | - name: db_url 1752 | title: Full URL to the external database. 1753 | ``` 1754 | 1755 | ```shell 1756 | $ ytt -f schema.yml --schema-inspect --output html 1757 |

Schema for "data/values"
1758 | 1759 | 1760 | 1761 | 1762 | 1763 | 1764 | 1765 | 1766 | 1767 | 1768 | 1769 | 1770 | 1771 | 1772 | 1773 | 1774 | 1775 | 1776 | 1777 | 1778 | 1779 | 1780 | 1781 | 1782 | 1791 | 1792 | (@ end -@) 1793 |
NameDescriptionDetailsExample
usernameDatabase username
usernameUser Name used to log into the database. 1783 | The account in the database system to authenticate as. 1784 |

1785 | Example: 1786 | 

1787 |           admin
1788 |
1789 |

1790 |
1794 | 1795 | fields: 1796 | - name: username 1797 | title: 1798 | example: 1799 | - value: admin 1800 | description: null 1801 | - name: db_url 1802 | title: Full URL to the external database. 1803 | ``` 1804 | 1805 | `docs.html.txt` 1806 | ```htmlmixed= 1807 | (@ load("@ytt:data", "data") @) 1808 | 1809 | 1810 | ``` 1811 | 1812 | ```shell 1813 | $ ytt --schema-inspect -f schema.yml --output html 1814 | ``` 1815 | 1816 | 1817 | # Additional References 1818 | 1819 | - 🔒 [Cloud Foundry for Kubernetes Configuration and Versioning Scheme](https://docs.google.com/document/d/1q4QyaElEX3KtIeGjwPmZpxsJpgsxin9sx7M0T4qBiW0) 1820 | - Sorbet (a typing system in Ruby): https://sorbet.org/docs/type-assertions 1821 | - Kubernetes users requesting JSON Schema: https://github.com/kubernetes/kubernetes/issues/14987 1822 | - JSON Schema Validation: https://json-schema.org/draft/2019-09/json-schema-validation.html 1823 | 1824 | # Closed Questions 1825 | 1826 | - _Is there a feasible way for a validation rule to get access to its siblings?_ **not now.** [details](https://github.com/k14s/design-docs/pull/1#discussion_r521425869) 1827 | - _Are there needs around versioning schema?_ **Not for the foreseable future.** [details](https://github.com/k14s/design-docs/pull/1#discussion_r521426706) 1828 | - _With the addition of a schema, the first data values file no longer serves that role. Could we improve the UX of adding multiple data value files by not requiring subsequent data values to be overlays?_ **Yes, which should be covered in [Part 8: Interactions with Existing Features](#Part-8-Interactions-with-Existing-Features). 1829 | - _Could a user include both a data/value + schema/amend in the same file?_ **For now, no.... but maybe** [details](https://github.com/k14s/design-docs/pull/1#discussion_r521433218) 1830 | 1831 | --------------------------------------------------------------------------------