├── files ├── qemu │ ├── OVMF.fd │ └── start_qemu.sh ├── clear-mixer-architecture.png ├── nginx │ └── mixer.conf ├── mixer │ └── release-image-config.json └── clr │ └── user-setup.sh ├── SUMMARY.md ├── book.json ├── README.md ├── 07-2-qa-testing.md ├── 03-scope.md ├── 07-3-debugging.md ├── 07-1-telemetry.md ├── 01-index.md ├── 05-deploying.md ├── 02-concepts.md ├── 06-rpms.md └── 04-mixing.md /files/qemu/OVMF.fd: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/clearlinux/how-to-clear/master/files/qemu/OVMF.fd -------------------------------------------------------------------------------- /files/clear-mixer-architecture.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/clearlinux/how-to-clear/master/files/clear-mixer-architecture.png -------------------------------------------------------------------------------- /files/nginx/mixer.conf: -------------------------------------------------------------------------------- 1 | server { 2 | server_name localhost; 3 | location / { 4 | root /home/clear/mix/update/www; 5 | autoindex on; 6 | } 7 | } 8 | -------------------------------------------------------------------------------- /SUMMARY.md: -------------------------------------------------------------------------------- 1 | # Summary 2 | 3 | * [Index](01-index.md) 4 | * [Concepts](02-concepts.md) 5 | * [Scope](03-scope.md) 6 | * [Mixing](04-mixing.md) 7 | * [Deploying](05-deploying.md) 8 | * [RPMs](06-rpms.md) 9 | * [Telemetry](07-1-telemetry.md) 10 | * [QA Testing](07-2-qa-testing.md) 11 | * [Debugging](07-3-debugging.md) 12 | -------------------------------------------------------------------------------- /book.json: -------------------------------------------------------------------------------- 1 | { 2 | "gitbook": ">=3.0.0", 3 | "plugins": ["autocover", "download-pdf-link"], 4 | "title": "How to Clear", 5 | "author": "T", 6 | "pluginsConfig": { 7 | "autocover": { 8 | "font": { 9 | "size": null, 10 | "family": "Impact", 11 | "color": "#FFF" 12 | }, 13 | "size": { 14 | "w": 1800, 15 | "h": 2360 16 | }, 17 | "background": { 18 | "color": "#09F" 19 | } 20 | } 21 | } 22 | } 23 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | 2 | How to Clear 3 | ============ 4 | 5 | ## What is this? 6 | 7 | This project contains documents that describe how to create a Clear 8 | Linux OS derivative, which is a Linux OS that uses most, or all, of 9 | the Clear Linux OS content and organization, but differs in some way. 10 | 11 | This project is for people who want to custom tailor the 12 | Clear Linux OS for their personal or professional purposes, 13 | but plan to update their content regularly with updates from 14 | [https://clearlinux.org/](clearlinux.org). 15 | 16 | You can make unlimited modifications to the Clear Linux OS. You 17 | can change the organization and not the content, or you can remove 18 | content. Adding and modifying content, and modifying the Linux kernel 19 | are also described in this project. 20 | 21 | 22 | ## Issues? 23 | 24 | See [https://github.com/clearlinux/how-to-clear/blob/master/01-index.md#need-help] 25 | -------------------------------------------------------------------------------- /files/mixer/release-image-config.json: -------------------------------------------------------------------------------- 1 | { 2 | "DestinationType" : "virtual", 3 | "PartitionLayout" : [ { "disk" : "release.img", "partition" : 1, "size" : "32M", "type" : "EFI" }, 4 | { "disk" : "release.img", "partition" : 2, "size" : "16M", "type" : "swap" }, 5 | { "disk" : "release.img", "partition" : 3, "size" : "10G", "type" : "linux" } ], 6 | "FilesystemTypes" : [ { "disk" : "release.img", "partition" : 1, "type" : "vfat" }, 7 | { "disk" : "release.img", "partition" : 2, "type" : "swap" }, 8 | { "disk" : "release.img", "partition" : 3, "type" : "ext4" } ], 9 | "PartitionMountPoints" : [ { "disk" : "release.img", "partition" : 1, "mount" : "/boot" }, 10 | { "disk" : "release.img", "partition" : 3, "mount" : "/" } ], 11 | "Version": "latest", 12 | "Bundles": ["kernel-kvm", "os-core", "os-core-update"] 13 | } 14 | -------------------------------------------------------------------------------- /07-2-qa-testing.md: -------------------------------------------------------------------------------- 1 | 2 | How To Clear - QA and testing Clear update content 3 | ================================================== 4 | 5 | ## What you'll learn in this chapter 6 | 7 | * Build time tests concepts 8 | * QA level testing tools 9 | 10 | 11 | ## Concepts 12 | 13 | It is virtually impossible as a human being to continuously monitor 14 | and test the proper functioning of a conglomerate as large as a normal 15 | Linux distribution. For this reason, we need to automate testing as 16 | much as possible and impose strict error checking at every level. 17 | 18 | In Clear Linux OS, we have created several layers of testing that will 19 | be Open Source in the future. At the time of writing this document 20 | they are not Open Source, so this chapter is incomplete. 21 | 22 | The automated package testing that comes with many of our packages 23 | is available at this time. 24 | 25 | 26 | ## `make check` 27 | 28 | Most upstream projects use the `check` framework to perform 29 | package-level unit tests. Clear Linux OS and autospec attempt to use 30 | these tests and enforce error checking, permitting a developer to 31 | spot errors, pause release, and prevent a known error from reaching 32 | customers. These errors are recorded and counted. 33 | 34 | The `%check` section in a `spec` file is used to call the package 35 | tests. If the output conforms to BAT or other standard unit testing 36 | output that programs like `check` generate, these package test results 37 | are recorded. 38 | 39 | You can add custom check commands by creating a `make_check_command` 40 | and inserting the custom tests or test command in there, and autospec 41 | will include them in every subsequent run. The `options.conf` allows 42 | you to either enforce that all tests must pass, or you may disable 43 | that setting and allow failures. 44 | 45 | The results of the tests are stored by the `common` tooling in the 46 | `testresults` file. You should inspect this file after each build 47 | to make sure that testing remains functional and no new errors are 48 | introduced with each package change. 49 | -------------------------------------------------------------------------------- /files/qemu/start_qemu.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # -*- mode: shell-script; indent-tabs-mode: nil; sh-basic-offset: 4; -*- 3 | # ex: ts=8 sw=4 sts=4 et filetype=sh 4 | # 5 | # start_qemu.sh 6 | # 7 | # Copyright (c) 2016-2017 Intel Corporation 8 | # 9 | # Permission is hereby granted, free of charge, to any person obtaining a copy 10 | # of this software and associated documentation files (the "Software"), to deal 11 | # in the Software without restriction, including without limitation the rights 12 | # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 13 | # copies of the Software, and to permit persons to whom the Software is 14 | # furnished to do so, subject to the following conditions: 15 | # 16 | # The above copyright notice and this permission notice shall be included in all 17 | # copies or substantial portions of the Software. 18 | # 19 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 20 | # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 21 | # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 22 | # AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 23 | # LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 24 | # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 25 | # SOFTWARE. 26 | # 27 | 28 | if [ -z "$1" ]; then 29 | IMAGE=clear.img 30 | else 31 | IMAGE="$1" 32 | shift 33 | fi 34 | 35 | if [[ "$IMAGE" =~ .xz$ ]]; then 36 | >&2 echo "File \"$IMAGE\" is still xz compressed. Uncompress it first with \"unxz\"" 37 | exit 1 38 | fi 39 | 40 | if [ ! -f "$IMAGE" ]; then 41 | >&2 echo "Can't find image file \"$IMAGE\"" 42 | exit 1 43 | fi 44 | rm -f debug.log 45 | 46 | VMN=${VMN:=1} 47 | 48 | qemu-system-x86_64 \ 49 | -enable-kvm \ 50 | -bios OVMF.fd \ 51 | -smp sockets=1,cpus=4,cores=2 -cpu host \ 52 | -m 1024 \ 53 | -vga none -nographic \ 54 | -drive file="$IMAGE",if=virtio,aio=threads,format=raw \ 55 | -netdev user,id=mynet0,hostfwd=tcp::${VMN}0022-:22,hostfwd=tcp::${VMN}2375-:2375 \ 56 | -device virtio-net-pci,netdev=mynet0 \ 57 | -debugcon file:debug.log -global isa-debugcon.iobase=0x402 $@ 58 | -------------------------------------------------------------------------------- /03-scope.md: -------------------------------------------------------------------------------- 1 | 2 | How To Clear - Scope of this training 3 | ===================================== 4 | 5 | ## Aim and scope of this training 6 | 7 | The goal of this training is to cover most of the technology used to 8 | create the Clear Linux\* OS. It includes extensive instructions on 9 | how to use the tools and explains how the tools work internally. The 10 | training uses the tools to create a custom Clear Linux OS for your 11 | own purposes, be it testing, development or other. 12 | 13 | A key point to this training is that we want to show users how Clear 14 | Linux OS is made and why it matters. For this reason, the training 15 | goes relatively deep into the methods used to create updates and how 16 | they are deployed to targets. The design is very different than what 17 | a traditional software developer would use, or what is used in the 18 | embedded sector. For that reason, the reality of how Clear Linux OS 19 | is made may be very different from what new users may expect. 20 | 21 | The methodology centers heavily around the concept that the target 22 | devices should be able to update often and quickly. At the opposite 23 | side, on the server that generates the content, the Clear Linux OS 24 | uses more elaborate and time intensive methods to assure all clients 25 | can properly update, which explains some of the major differences 26 | between Clear Linux OS and other Linux distributions. 27 | 28 | ## What isn't covered 29 | 30 | This training document is meant to provide methods and tools for 31 | users in the public. It builds upon Open Source resources and server 32 | content that is available to the general public. 33 | 34 | The training does not cover `koji` specifically. The Clear Linux OS 35 | team uses koji to maintain RPM repositories but it is only one method 36 | to maintain RPM repositories and it is not required. Clear Linux OS 37 | provides `koji` in a bundle for those that wish to use it. 38 | 39 | There are many other aspects related to the maintenance of a good 40 | Linux OS that are not covered in this training, such as legal and 41 | liability aspects, license compliance, and others. Before you start 42 | maintaining a Linux OS yourself, you should be aware that you may 43 | have duties and requirements that you must follow, even if you are 44 | reusing content from the Clear Linux OS. 45 | -------------------------------------------------------------------------------- /07-3-debugging.md: -------------------------------------------------------------------------------- 1 | 2 | How To Clear - Debugging applications in Clear Linux OS 3 | ======================================================= 4 | 5 | ## What you'll learn in this chapter 6 | 7 | * Special debug file system concepts for Clear Linux OS 8 | * Avoiding having to deploy your own fuse debug_fs 9 | 10 | 11 | ## Concepts 12 | 13 | Developers need much more information about binaries, code and 14 | functions when working with compiled code. Debuggers require lots of 15 | meta information about source code line numbers, function parameters, 16 | and memory offsets when developers need to debug applications. 17 | 18 | Clear Linux OS features a unique solution that avoids the high cost 19 | of providing full debug info files to all systems and creates an 20 | on-the-fly virtual file system that only consumes network bandwidth 21 | when applications are being debugged. This creates a work environment 22 | that isn't in the way of developers but allows full debugging of 23 | every application when needed. 24 | 25 | The virtual debuginfo filesystem is implemented using FUSE fs, 26 | and uses a HTTPS transport. Each Clear Linux OS installation mounts 27 | this virtual network filesystem remotely, but no penalty is incurred 28 | until debugging applications such as `gdb` need to access debug info 29 | files. If a backtrace is needed or manual debugging is started, 30 | the debugger will attempt to find the needed debug info files on 31 | the virtual file system, and the required objects will be fetched 32 | and cached. 33 | 34 | Each binary that Clear Linux OS ships is tagged with a specific version 35 | number unique to each compiled binary. This avoids conflicts between 36 | the many different versions of a patched binary program. The Clear 37 | Linux OS tooling generates the needed files on the server side with 38 | each Clear Linux OS release. 39 | 40 | 41 | ## Don't strip custom binaries 42 | 43 | Because the virtual debug file system will not contain any debug info 44 | files about custom compiled or mixed binaries, people who need to debug 45 | applications on the target should not strip their binaries. Otherwise 46 | they will not be able to obtain debuginfo symbols. As an alternative, 47 | you could also disable the virtual debug file system entirely and 48 | ship custom debuginfo bundles. 49 | 50 | 51 | ## What else to try 52 | 53 | * Launch gdb on a running application and obtain a backtrace. 54 | * Compile a custom binary while disabling stripping using the autospec 55 | `nostrip` option in `option.conf`. 56 | -------------------------------------------------------------------------------- /07-1-telemetry.md: -------------------------------------------------------------------------------- 1 | 2 | How To Clear - Clear Linux OS telemetry 3 | ======================================= 4 | 5 | ## What you'll learn in this chapter 6 | 7 | * Ground rules of telemetry, privacy, opt-in 8 | * Basics of Clear Linux OS telemetry 9 | * Creating a custom telemetry event 10 | * Backend collection concepts 11 | 12 | 13 | ## Concepts 14 | 15 | Clear Linux OS uses a lightweight telemetry solution to allow 16 | applications to avoid being concerned with transporting the data and 17 | whether a user has opted in or out. The solution can provide near 18 | real time data to improve applications. 19 | 20 | Because software updates in Clear Linux OS are performed automatically, 21 | it is important that application developers leverage telemetry where 22 | appropriate. Traditional QA testing and functional testing are often 23 | incomplete. The Clear Linux OS telemetry solution bridges that gap 24 | and helps developers monitor their applications. 25 | 26 | By default, telemetry isn't opt-out and the installer will ask people 27 | if telemetry should be enabled, with the default selection being 28 | "not enabled". We do encourage people to enable telemetry to help 29 | out and make Clear Linux OS higher quality. 30 | 31 | Intel's privacy policies are applicable to the Clear Linux OS telemetry 32 | stack, and the Clear Linux OS telemetry does not collect intentionally 33 | identifiable information about the user or system owner. People who 34 | use the telemetry APIs discussed in this chapter must ensure that their 35 | use does not conflict with any Intel policies or local privacy laws. 36 | 37 | The telemetry API revolves around the delivery of telemetry records 38 | as an individual unit to a HTTPS collection service. On the client, 39 | a spooling daemon `telemd` takes care of opt-in/out, throttling, and 40 | encapsulating the telemetry data. Several telemetry probes generate 41 | probe specific payload data and deliver it to the `telemd` service 42 | for delivery. 43 | 44 | 45 | ## How to start 46 | 47 | First, we enable telemetry on the target device. For this, we add 48 | the telemetrics bundle to our mix, and make it available to clients. 49 | 50 | ``` 51 | ~/mix $ mixer bundle add telemetrics 52 | Adding bundle "telemetrics" from upstream bundles 53 | ~/mix $ sudo mixer build all 54 | ``` 55 | 56 | We can modify the image to add telemetry by default as a bundle, 57 | but we would lose our existing system. We add it on our target device 58 | manually for now: 59 | 60 | ``` 61 | ~ # swupd update 62 | ~ # swupd bundle-add telemetrics 63 | ``` 64 | 65 | Essentially, we now have everything to create telemetry events, even 66 | from C programs or Python\* if needed, because the telemetry bundle 67 | provides a simple pipe-based CLI program that can be called trivially: 68 | 69 | ``` 70 | ~ # telem-record-gen --help 71 | Usage: 72 | telem-record-gen [OPTIONS] - create and send a custom telemetry record 73 | 74 | Help Options: 75 | -h, --help Show help options 76 | 77 | Application Options: 78 | -f, --config-file Path to configuration file (not implemented yet) 79 | -V, --version Print the program version 80 | -s, --severity Severity level (1-4) - (default 1) 81 | -c, --class Classification level_1/level_2/level_3 82 | -p, --payload Record body (max size = 8k) 83 | -P, --payload-file File to read payload from 84 | -R, --record-version Version number for format of payload (default 1) 85 | -e, --event-id Event id to use in the record 86 | ``` 87 | 88 | Using the C library (`libtelemetry.so` - `man 3 telemetry`) uses the 89 | exact same API parameters and yields the same effect. 90 | 91 | Let's try generating a simple heartbeat event, similar to the `hprobe` 92 | heartbeat probe that Clear Linux OS includes by default. 93 | 94 | ``` 95 | ~ # telem-record-gen -c org.clearlinux/hello/world -p "hello" 96 | ``` 97 | 98 | We won't see anything happen, but we can track existing and previous 99 | telemetry events with `telemctl`: 100 | 101 | ``` 102 | ~ # telemctl journal 103 | ``` 104 | 105 | A full example of the heartbeat probe in C is documented in the source 106 | code here: 107 | 108 | * [https://github.com/clearlinux/telemetrics-client/blob/master/src/probes/hello.c] 109 | 110 | 111 | ## Pointing telemetry to a custom backend 112 | 113 | If you have a functional custom collector, you can 114 | modify where your telemetry records get sent by modifying 115 | `/etc/telemetrics/telemetrics.conf` and changing the `server` value 116 | to point to the new telemetry collection URL. Setting up a custom 117 | collector is not covered in this training. 118 | 119 | If you are deploying a custom backend for many Clear Linux OS 120 | installations, you may want to patch the system wide default 121 | `/usr/share/defaults/telemetrics/telemetrics.conf` file to include 122 | a custom collection URL built into the binary. 123 | 124 | 125 | ## Further reading 126 | 127 | * [https://clearlinux.org/features/telemetry] 128 | * [https://clearlinux.org/documentation/clear-linux/tutorials/telemetry-backend] 129 | -------------------------------------------------------------------------------- /01-index.md: -------------------------------------------------------------------------------- 1 | 2 | How To Clear 3 | ============ 4 | 5 | ## Table of Contents 6 | 7 | 1. [Index](01-index.md) 8 | 2. [Concepts](02-concepts.md) 9 | 3. [Extent of this training](03-scope.md) 10 | 4. [Basic mixing](04-mixing.md) 11 | 5. [Deploying a mix](05-deploying.md) 12 | 6. [Adding in custom RPMs](06-rpms.md) 13 | 7. Developer topics: 14 | 1. [Telemetry](07-1-telemetry.md) 15 | 2. [QA and testing](07-2-qa-testing.md) 16 | 3. [Debugging](07-3-debugging.md) 17 | 18 | ``` 19 | ~ $ git clone https://github.com/clearlinux/how-to-clear 20 | ~ $ sudo swupd update 21 | ``` 22 | 23 | ## Foreword 24 | 25 | Clear Linux OS for Intel® Architecture was designed a long time ago 26 | to fill a major gap in the existing Linux OS ecosystem. The problem 27 | identified was that Linux OS distributions were either updated once 28 | or twice a year, or the distro turned your system into a dependency 29 | hell because it allowed your package management system to combine 30 | incompatible components. 31 | 32 | Clear Linux OS attempts to solve this problem by preventing you from 33 | combining fundamentally incompatible components, and allowing you to 34 | update with a frequency of several times a day, if needed. This allows 35 | users to receive smaller updates and maintain functional software 36 | without the overhead of dealing with software incompatibilities. 37 | 38 | Over the last few years, the Clear Linux OS team has consistently 39 | delivered this in a usable form and we've used Clear Linux OS as 40 | a vehicle to deliver much more technological advancement to users. 41 | We've also allowed people to use our tooling to do exactly the same, 42 | without recreating Clear Linux OS from scratch. Our tooling, concepts, 43 | and content are modular and Free (as in beer and speech), and we have 44 | published everything. 45 | 46 | The tooling covered in this training reproduces what the Clear Linux 47 | OS team develops and uses to create and maintain the OS. The tooling 48 | allows users to create a derivative OS that is based on Clear Linux 49 | OS in various degrees of similarity. Either the derivative can be 50 | entirely new and use only the tooling provided, or the derivative 51 | can contain only minor changes. 52 | 53 | The training revolves around mixer from start to finish, and makes 54 | several segues into more advanced topics that are relevant for the 55 | whole picture. In all details, we show in precise ways how the actual 56 | deployment of the results of the advanced topics can be pushed to 57 | actual target OS installations. 58 | 59 | ## About this document 60 | 61 | This training document is a living training that will be adopted in 62 | the future to adjust for changes in the tooling and options, as well 63 | as any fixes that must be included. 64 | 65 | All of the covered tools have been documented in full in separate 66 | official documentation projects that can be found here: 67 | 68 | [ClearLinux Documentation](https://clearlinux.org/documentation) 69 | 70 | This training document is not designed as a reference documentation 71 | for `mixer`, `autospec`, `common`, `swupd` or any of the referenced 72 | programs, instead, it focuses on demonstrating the fundamental 73 | functionality and integration of these tools and what they can 74 | accomplish for the user in a nutshell. 75 | 76 | The content itself is designed to be self-contained and allow someone 77 | to use a clean Clear Linux OS installation to do the training in its 78 | entirety. There should not be a need to install third party software, 79 | and the training material should explain all the concepts without 80 | the need for external reference documentation. It is expected that 81 | people will consult manual pages where appropriate. 82 | 83 | Due to the nature of the Clear Linux OS tools, a functional network 84 | connection is required to use many of the tools. While it is possible 85 | to create a set of trainings that would function entirely offline, 86 | this would be time consuming and out of date almost immediately. 87 | 88 | This training is hosted on github. We appreciate any feedback and 89 | comments, especially in the form of Pull Requests. Please visit the 90 | project page to open a ticket or clone/branch the training and help us. 91 | 92 | [how-to-clear](https://github.com/clearlinux/how-to-clear/) 93 | 94 | For the convenience of students, we've included a folder called `files` 95 | and added most of the files that need to be created or downloaded 96 | during the training exercises. This allows students to skip through 97 | some of the steps and get the proper files in place quickly and stay 98 | focused on the topic without spending time on meta-problems. 99 | 100 | ## Need help? 101 | 102 | The Clear Linux OS team can be reached for generic questions about 103 | Clear Linux OS, bugs, feedback, and any relevant discussion through 104 | the following methods: 105 | 106 | * [Mailinglist](https://lists.clearlinux.org/mailman/listinfo/dev) 107 | * [IRC](http://webchat.freenode.net?channels=%23clearlinux) 108 | * [Github](https://github.com/clearlinux/distribution) 109 | -------------------------------------------------------------------------------- /files/clr/user-setup.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | SCRIPT=$(/usr/bin/basename $0) 4 | PEM="" 5 | SERVERCA="" 6 | CLIENTCA="" 7 | WORKSPACE="clearlinux" 8 | PACKAGE_REPOS= 9 | NEEDS_KVM_GROUP= 10 | 11 | help() { 12 | printf "%s\n" >&2 "Usage: $SCRIPT [options]" \ 13 | "" \ 14 | "Options:" \ 15 | "" \ 16 | "-d --directory NAME: Set up workspace in the given directory." \ 17 | "-a --clone-packages: Clone all package repos." \ 18 | "-j --jobs [NUM]: Clone repos with NUM jobs. If NUM is not given, it is set to the available CPU count." \ 19 | "" \ 20 | "-k --client-cert PEM_FILE: Enable client user cert for koji configuration; requires a PEM file argument" \ 21 | "-s --server-ca PEM_FILE: Enable server CA cert for koji configuration; requires a PEM file argument" \ 22 | "-c --client-ca PEM_FILE: Enable client CA cert for koji configuration; requires a PEM file argument" \ 23 | "" 24 | } 25 | 26 | error() { 27 | echo -e "Error: $1\n" >&2 28 | help 29 | exit 1 30 | } 31 | 32 | while [ $# -gt 0 ]; do 33 | case "$1" in 34 | "--help"|"-h") 35 | help 36 | exit 0 37 | ;; 38 | "--client-cert"|"-k") 39 | shift 40 | PEM="$(realpath $1)" 41 | ;; 42 | "--server-ca"|"-s") 43 | shift 44 | SERVERCA="$(realpath $1)" 45 | ;; 46 | "--client-ca"|"-c") 47 | shift 48 | CLIENTCA="$(realpath $1)" 49 | ;; 50 | "--jobs"|"-j") 51 | if echo "$2" | grep -qx "[1-9][0-9]*"; then 52 | shift 53 | JOBS="$1" 54 | elif [ -f /proc/cpuinfo ]; then 55 | JOBS=$(grep -Ec '^processor.*:.*[0-9]+$' /proc/cpuinfo) 56 | fi 57 | ;; 58 | "--directory"|"-d") 59 | [ -z "$2" ] && error "Must supply a directory name to the -d option" 60 | [ "${2:0:1}" = "-" ] && error "Directory name cannot begin with \"-\"" 61 | shift 62 | WORKSPACE="$1" 63 | ;; 64 | "--clone-packages"|"-a") 65 | PACKAGE_REPOS=1 66 | ;; 67 | *) 68 | help 69 | exit 1 70 | ;; 71 | esac 72 | shift 73 | done 74 | 75 | 76 | if [ -z "$PEM" ] && [ -z "$SERVERCA" ] && [ -z "$CLIENTCA" ]; then 77 | USE_KOJI= 78 | else 79 | if [ -z "$PEM" ] || [ -z "$SERVERCA" ] || [ -z "$CLIENTCA" ]; then 80 | error "Must specify all three command line options (or none)" 81 | fi 82 | if [ ! -f "$PEM" ]; then 83 | error "Missing koji client PEM key file" 84 | fi 85 | if [ ! -f "$SERVERCA" ]; then 86 | error "Missing koji server CA PEM file" 87 | fi 88 | if [ ! -f "$CLIENTCA" ]; then 89 | error "Missing koji client CA PEM file" 90 | fi 91 | USE_KOJI="yes" 92 | fi 93 | 94 | if [ -n "$JOBS" ]; then 95 | JOBS_ARG="-j $JOBS" 96 | fi 97 | 98 | if [ -d "$WORKSPACE" ]; then 99 | error "Directory \"$WORKSPACE\" already exists. \ 100 | Either remove this workspace, or use a different workspace name." 101 | fi 102 | 103 | required_progs() { 104 | local bindir="/usr/bin" 105 | for f in git mock rpm rpmbuild ; do 106 | [ ! -x "${bindir}/${f}" ] && missing+="${f} " 107 | done 108 | [ "$PEM" ] && [ ! -x /usr/bin/koji ] && missing+="koji " 109 | if [ -n "$missing" ]; then 110 | echo "Install the following programs and re-run this script:" >&2 111 | echo $missing >&2 112 | echo 'All programs should be provided in the "os-clr-on-clr" bundle.' >&2 113 | exit 1 114 | fi 115 | } 116 | 117 | required_progs 118 | 119 | if ! groups | grep -qw kvm; then 120 | NEEDS_KVM_GROUP=1 121 | fi 122 | 123 | echo "Initializing development workspace in \"$WORKSPACE\" . . ." 124 | 125 | mkdir "$WORKSPACE" 126 | cd "$WORKSPACE" 127 | 128 | echo "Setting up common repo . . ." 129 | mkdir projects 130 | git clone https://github.com/clearlinux/common projects/common 131 | if [ $? -ne 0 ]; then 132 | echo "Failed to clone common repo." >&2 133 | exit 1 134 | fi 135 | 136 | # Finish setup for packages/projects hierarchy 137 | ln -sf projects/common/Makefile.toplevel Makefile 138 | mkdir -p packages/common 139 | ln -sf ../../projects/common/Makefile.common packages/common/Makefile.common 140 | 141 | if [ "$USE_KOJI" ]; then 142 | echo "Setting up koji certs . . ." 143 | mkdir -p ~/.koji 144 | cp "$PEM" ~/.koji/client.crt 145 | cp "$CLIENTCA" ~/.koji/clientca.crt 146 | cp "$SERVERCA" ~/.koji/serverca.crt 147 | 148 | if [ ! -f /etc/koji.conf ]; then 149 | echo "Setting up koji config . . ." 150 | sudo cp projects/common/conf/koji.conf /etc 151 | fi 152 | fi 153 | 154 | if [ -n "$NEEDS_KVM_GROUP" ]; then 155 | echo "Adding user to kvm group . . ." 156 | sudo usermod -a -G kvm $USER 157 | fi 158 | 159 | echo "Cloning special project repositories . . ." 160 | make ${JOBS_ARG} clone-projects 161 | 162 | if [ -n "$PACKAGE_REPOS" ]; then 163 | echo "Cloning all package repositories . . ." 164 | make ${JOBS_ARG} clone-packages 165 | fi 166 | 167 | echo "Creating mix workspace . . ." 168 | mkdir -p mix 169 | 170 | if [ "$USE_KOJI" ]; then 171 | echo "Testing koji installation . . ." 172 | if koji moshimoshi; then 173 | echo -en "\n************************\n\n" 174 | echo "Koji installed and configured successfully" 175 | else 176 | echo -en "\n************************\n\n" 177 | echo "Error with koji installation or configuration" >&2 178 | exit 1 179 | fi 180 | fi 181 | 182 | echo -en "\n************************\n" 183 | 184 | echo "Workspace has been set up in \"$WORKSPACE\"" 185 | if [ -z "$PACKAGE_REPOS" ]; then 186 | echo "NOTE: To clone all package repos, run \"cd $WORKSPACE; make [-j NUM] clone-packages\"" 187 | echo "NOTE: To clone a single package repo with NAME, run \"cd $WORKSPACE; make clone_NAME\"" 188 | fi 189 | if [ -n "$NEEDS_KVM_GROUP" ]; then 190 | echo 'NOTE: logout and log back in to finalize the setup process' 191 | fi 192 | 193 | 194 | # vi: ft=sh sw=2 et sts=2 195 | -------------------------------------------------------------------------------- /05-deploying.md: -------------------------------------------------------------------------------- 1 | 2 | How To Clear - Deploying updates to a target 3 | ============================================ 4 | 5 | ## What you'll learn in this chapter 6 | 7 | * Creating an update content server 8 | * Creating images 9 | * Deploying an initial image to a target 10 | * Starting a Clear Linux OS based Virtual Machine (VM) 11 | * Switching the Clear Linux machine to use the custom mix update content 12 | 13 | 14 | ## Conveying the content to a target 15 | 16 | For the purpose of this training, we chose a setup that can be 17 | reproduced easily using a stock Clear Linux OS. The simplest setup is 18 | to use a simple VM that will be updating against our locally produced 19 | mix content. 20 | 21 | We will set up a HTTP service for the client using `nginx` because 22 | it comes with Clear Linux OS in a bundle. `nginx` needs some minor 23 | configuration but we won't cover more complex `nginx` setup issues 24 | here. This training does not cover topics like `certbot` which will 25 | be different for most setups. 26 | 27 | 28 | ## nginx 29 | 30 | ``` 31 | ~/mix $ sudo swupd bundle-add web-server-basic 32 | ``` 33 | 34 | This installs the `nginx` service onto your device. It doesn't start 35 | by default and we need to configure it. 36 | 37 | ``` 38 | ~/mix $ sudo mkdir -p /etc/nginx/conf.d 39 | ~/mix $ sudo cp /usr/share/nginx/conf/nginx.conf.example /etc/nginx/nginx.conf 40 | ``` 41 | 42 | The default `nginx` configuration works just fine as a base nginx 43 | config file. We'll create a specific update server config separately. 44 | To do that, we create a small config file for nginx that will make 45 | it point to our update content directly. 46 | 47 | ``` 48 | ~/mix $ sudoedit /etc/nginx/conf.d/mixer.conf 49 | ``` 50 | 51 | Insert the following content into the `mixer.conf` file: 52 | 53 | ``` 54 | server { 55 | server_name localhost; 56 | location / { 57 | root /home/clear/mix/update/www; 58 | autoindex on; 59 | } 60 | } 61 | ``` 62 | 63 | Make sure to insert the correct username in case your username isn't 64 | `clear`. 65 | 66 | Once this file is in place, we start the http server and you can view 67 | the content by visiting `http://localhost`: 68 | 69 | ``` 70 | ~/mix $ sudo systemctl restart nginx 71 | ``` 72 | 73 | ## Conveying the URL to mixer 74 | 75 | Once this is functional, we tell `mixer` to point to this content by 76 | its URL. To do this, we edit `builder.conf` and change the following 77 | options to properly point the system to the mix we are making. 78 | 79 | Because of an issue with QEMU\* and networking, we point the OS 80 | inside the virtual machine at the external IP address of the host 81 | where `nginx` is running. We can find out the IP address with the 82 | following command: 83 | 84 | ``` 85 | ~/mix $ networkctl status 86 | ``` 87 | 88 | There are likely 2 or 3 entries listed under `Address` in the output 89 | of this command. All entries should be valid, but we recommend you 90 | pick the top one listed for use in the `builder.conf`: 91 | 92 | ``` 93 | CONTENTURL=http://10.276.302.138/ 94 | VERSIONURL=http://10.276.302.138/ 95 | ``` 96 | 97 | Now that that is in place, we can proceed to the next step, which is 98 | to create an image for a VM. 99 | 100 | 101 | ## VM kernel 102 | 103 | We need to create an image for a QEMU virtual machine. We can use the 104 | standard bundle list that mixer uses by default, but this includes 105 | the `native` kernel. This is not very efficient, as the QEMU virtual 106 | machine only includes a very small set of hardware, and we don't need 107 | all the hundreds of drivers that the `native` kernel includes. For 108 | this reason, we want to modify our default bundle set already so we 109 | have a smaller amount of content to process. We'll get a faster boot 110 | time out of it, simply because the kernel image will be smaller and 111 | faster to load. 112 | 113 | To do this, we include the `kernel-kvm` bundle in our mix, and we 114 | use the upstream content for now: 115 | 116 | ``` 117 | ~/mix $ mixer bundle add kernel-kvm 118 | ``` 119 | 120 | We need to generate an update so that the new kernel images are 121 | available in our content stream and the image will be able to 122 | include it: 123 | 124 | ``` 125 | ~/mix $ mixer build all 126 | ``` 127 | 128 | You should verify that the `www` content contains a new version, 129 | and that a `Manifest.kernel-kvm` now exists in the latest version of 130 | the update content. 131 | 132 | 133 | ## Create the image 134 | 135 | We need to generate an `ister` config file that describes what kind 136 | of image we're creating and what needs to go into the image. For the 137 | purpose of this training, we will make a `live` image that we can 138 | boot straight into without the need for an installation step. First, 139 | we create the file `release-image-config.json` with the following 140 | content in the mix folder: 141 | 142 | ``` 143 | { 144 | "DestinationType" : "virtual", 145 | "PartitionLayout" : [ { "disk" : "release.img", "partition" : 1, "size" : "32M", "type" : "EFI" }, 146 | { "disk" : "release.img", "partition" : 2, "size" : "16M", "type" : "swap" }, 147 | { "disk" : "release.img", "partition" : 3, "size" : "10G", "type" : "linux" } ], 148 | "FilesystemTypes" : [ { "disk" : "release.img", "partition" : 1, "type" : "vfat" }, 149 | { "disk" : "release.img", "partition" : 2, "type" : "swap" }, 150 | { "disk" : "release.img", "partition" : 3, "type" : "ext4" } ], 151 | "PartitionMountPoints" : [ { "disk" : "release.img", "partition" : 1, "mount" : "/boot" }, 152 | { "disk" : "release.img", "partition" : 3, "mount" : "/" } ], 153 | "Version": "latest", 154 | "Bundles": ["kernel-kvm", "os-core", "os-core-update"] 155 | } 156 | ``` 157 | 158 | After this file is in place, `mixer` can properly start `ister` 159 | for us with the `build image` subcommand. 160 | 161 | ``` 162 | ~/mix $ sudo mixer build image --native 163 | ``` 164 | 165 | Note: If you have made any changes to `builder.conf` or the content 166 | of your mix, you need to run `sudo mixer build all` before creating 167 | a new image, otherwise your image may be built with old content and 168 | not include the needed URL changes for `swupd` to work properly inside 169 | the image. 170 | 171 | This outputs a file called `release.img` which is directly bootable 172 | in QEMU. We use the standard `start_qemu.sh` script to invoke QEMU 173 | to then invoke the image and redirect the VM output to our local 174 | console. For this we also need the `OVMF.fd` file. These can be found 175 | in the `files` folder inside the training repository. 176 | 177 | ``` 178 | ~/mix $ sudo ./start_qemu.sh release.img 179 | ``` 180 | 181 | You need to log in as root and immediately provide a new root password, 182 | because Clear Linux OS does not ship with a default root password, 183 | as you may already know. 184 | 185 | After logging in as root, `swupd update` will display that there 186 | are no current updates available, and it will list your latest mix 187 | content version as the last available update. 188 | 189 | If we create a new update on the outside of the VM quickly with: 190 | 191 | ``` 192 | ~/mix $ mixer build all 193 | ``` 194 | 195 | Then run `swupd update` inside the VM, we will see the update apply. 196 | 197 | 198 | ## Exercises 199 | 200 | * List your own mix bundles on your target system. 201 | * Verify your install. 202 | * Downgrade your target system with `swupd`, and then update it again. 203 | * Create an installer image using the installer configuration file at 204 | [https://download.clearlinux.org/current/config/image/installer-config.json]. You will need the referenced Python* script. 205 | -------------------------------------------------------------------------------- /02-concepts.md: -------------------------------------------------------------------------------- 1 | 2 | How To Clear - Clear Linux distribution concepts 3 | ================================================ 4 | 5 | ## What this document explains 6 | 7 | In the training, we will use the Clear Linux OS Team vocabulary to 8 | describe processes, concepts, and data. Since this vocabulary has 9 | grown over time in the team, it may not be the most logical naming for 10 | people who start to learn about how Clear Linux OS is created. This 11 | chapter attempts to explain the idea behind the terminology. 12 | 13 | 14 | ## Goal 15 | 16 | The end goal of this training is that the student understands how 17 | to create and deliver a custom version of Clear Linux OS using the 18 | tooling explained in this training. The full picture of how that 19 | looks is graphically summarized in the following graph:. 20 | 21 | ![Clear Mixer Architecture](/files/clear-mixer-architecture.png) 22 | 23 | The graph shows how content from the official Clear Linux OS and any 24 | custom content can be mixed to create both images, and an update 25 | content stream. 26 | 27 | The update content stream will need to be made available as HTTP or 28 | HTTPS data on a network connection. It is the equivalent of a package 29 | repository. 30 | 31 | The initial setup of machines needs to happen with bootable images 32 | that can either live boot, or install the base OS. 33 | 34 | 35 | ## Updates 36 | 37 | The concept of updates is central to the design of the Clear Linux 38 | OS method of delivering and maintaining the OS. In essence, every 39 | modification to the OS is considered an update. This includes 40 | installation, updates themselves, and the addition of optional 41 | components. In principle, even rolling back to an older version is 42 | an update. 43 | 44 | The content originates from an "update server". This is implemented by 45 | a https enabled webserver where the files are served as static content. 46 | The Clear Linux OS periodically queries the data on the update server 47 | and determines whether updates are available, and in case the OS wants 48 | to install optional components, or even when a new installation is 49 | performed. In all those cases, the update content files provides all 50 | the data and metadata to perform all the needed actions. 51 | 52 | More reading: 53 | 54 | [Software Updates in Clear Linux](https://clearlinux.org/documentation/clear-linux/concepts/swupd-about) 55 | 56 | 57 | ## Bundles 58 | 59 | In Clear Linux OS, the choice was made to do away with packages as 60 | the smallest functional component size. One of the main reasons is 61 | that packages in a disproportionate sense are unusable to users by 62 | themselves, and require a large amount of packages to be installed 63 | before the functionality they offer can be used. 64 | 65 | For example, the xorg-server package does not provide a function X 66 | server without the presence of about 30+ additional components. On top 67 | of that, in order to create these 30 or so components, an additional 68 | 200 or so packages are needed for various aspects of the creation of 69 | the X server binary. 70 | 71 | In Clear Linux OS, bundles are the concept of the smallest usable 72 | collection of packages that provide a functional component, and 73 | traditional packages are not visible to the user. In some cases, it 74 | could mean that a bundle effectively contains a single package (e.g. 75 | the "curl" bundle), but in most cases a bundle contains several or 76 | even many packages. 77 | 78 | More Reading: 79 | 80 | [About bundles](https://clearlinux.org/documentation/clear-linux/concepts/bundles-about#bundles-about) 81 | 82 | 83 | ## Manifests 84 | 85 | The Clear Linux OS software update content consists of data and 86 | metadata. The data is the files that end up in the OS. The metadata 87 | contains relevant information to properly provision the data to the OS 88 | file system, as well as update the system and add or remove additional 89 | content to the OS. 90 | 91 | The Manifests are mostly long lists of hashes that describe content. 92 | Each bundle gets its own manifest file. There is a master manifest 93 | file that describes all manifests to tie it all together. 94 | 95 | 96 | ## Fullfiles, packs, and delta packs 97 | 98 | The data that an update provisions to a system can be obtained in 99 | three different ways. There are three different methods, and they 100 | exist to optimize the delivery of content and speed up updates. 101 | 102 | Fullfiles are always generated for every file in every release. This 103 | allows any Clear Linux OS to obtain the exact copy of the content 104 | for each version directly. This would be used if the OS verification 105 | (`swupd verify`) needed to replace a single file, for instance. 106 | 107 | Packs are available for some releases and combine many files to speed 108 | up the creation of installation media and large updates. Delta packs 109 | are an optimized version of packs that only contain updates (binary 110 | diffs) and cannot be used without having the original file content. 111 | 112 | In most `swupd update` scenarios, the delta packs will be used as much 113 | as possible, since they deliver the update content in the smallest 114 | size possible. 115 | 116 | In most `swupd bundle-add` scenarios, the packs will be used as 117 | much as possible, since they deliver the needed content in a single 118 | downloadable unit. 119 | 120 | 121 | ## Upstream 122 | 123 | An Open Source Project is by definition "upstream". The Clear Linux OS 124 | directly consumes project software from upstream as much as possible. 125 | In most cases, this is trivial and the upstream community creates 126 | proper source code releases, and addresses bugs and issues as needed 127 | appropriately. 128 | 129 | In some cases, the Clear Linux OS team maintains some projects as 130 | an upstream project as well. Examples are obviously the mixer and 131 | swupd projects. 132 | 133 | The term upstream describes a relationship where content moves from 134 | upstream to downstream in a fluid manner, and content is generally 135 | aimed to be submitted back to upstream but receive significant review. 136 | This concept applies also when people create a mixed Clear Linux OS 137 | version. In that case, the official Clear Linux OS is the upstream 138 | to the mixed version. 139 | 140 | 141 | ## Packages vs. projects 142 | 143 | In the Clear Linux OS terminology, we separate the source code that 144 | is in upstream repositories, and the distribution adaptation of the 145 | products of that project. 146 | 147 | The definition of a project is the thing that is the upstream of 148 | a package. Projects are maintained by individual maintainers. The 149 | Clear Linux OS team does not modify the project unless they are the 150 | upstream maintainers. In the same way, people who are maintainers 151 | of projects are directly providing content to the Clear Linux OS, 152 | with all the responsibilities that go with it. 153 | 154 | Packages describe how the project content is consumed. Many packages 155 | have optional dependencies and configuration items that may or may not 156 | be desired. On top of that, specifically targeted distributions like 157 | Clear Linux OS may want to further patch the software to configure 158 | the project to better work with the methods and rules that the Clear 159 | Linux OS dictates. 160 | 161 | 162 | ## RPM 163 | 164 | Internally, the Clear Linux OS uses the RPM package format to bridge 165 | the software source code and the binary software update content. The 166 | RPM format is an intermediate way of storing content. 167 | 168 | Note: `swupd` does *not* use or recognize the RPM file format and the 169 | `rpm` program on a Clear Linux OS installation is *not* capable of 170 | using RPM files. 171 | 172 | Within the build mechanisms that Clear Linux OS uses, the format 173 | is used to store the output of the compilation process and provide 174 | dependency relationships to the code that generates bundles. These 175 | bundles rely on the RPMs dependencies, and by extension, the source 176 | code dependency information. Using these dependencies, the mixer can 177 | create meaningful bundles that contain the needed components to make 178 | software functional without having to describe all dependencies in 179 | the bundle definition. 180 | -------------------------------------------------------------------------------- /06-rpms.md: -------------------------------------------------------------------------------- 1 | 2 | How To Clear - Creating and mixing custom RPMs 3 | ============================================== 4 | 5 | ## What you'll learn in this chapter 6 | 7 | * Modifying a Clear Linux OS kernel RPM 8 | * Creating a new RPM from an upstream release 9 | * Adding the custom RPM to the update content 10 | * Deploying the change to the target system 11 | 12 | 13 | ## Concepts 14 | 15 | Creating RPM files is a relatively simple task where source code and 16 | build instructions are used to generate a binary archive with some 17 | metadata. The RPM format has been in use for a very long time and is 18 | an efficient method to convey both content and metadata in a single 19 | file format. 20 | 21 | Clear Linux OS uses the RPM file format for intermediate storage. This 22 | allows developers to solve partial problems and not have to rebuild 23 | an entire OS altogether. This increases speed while still providing 24 | all the tools and data needed to prevent breaking dependencies. 25 | 26 | RPM files are created as the output of `rpmbuild`, which takes a 27 | `spec` file as instructions, together with the source code, patches, 28 | and miscellaneous files that may be needed during the build. For this 29 | purpose, the developer doesn't really work with RPM files, instead 30 | they work with `spec` files and miscellaneous files that are the 31 | input to the `rpmbuild` phase. 32 | 33 | This is what we call a **package** in Clear Linux OS terminology. These 34 | packages all live in their own individual git tree. They are published 35 | for users to see, review, and reuse at the following github URL: 36 | 37 | [https://github.com/clearlinux-pkgs] 38 | 39 | Working with `spec` files is tedious work. A Linux distribution 40 | typically contains thousands of them and they mostly repeat metadata 41 | that can automatically be generated or discovered, so we really don't 42 | want to spend too much time on the metadata aspect. The Clear Linux OS 43 | team also has created tooling to make maintaining and creating these 44 | `spec` files easier and largely automated, so that developers can 45 | focus on the content instead. 46 | 47 | We use the `common` tooling to bypass some of the hurdles that make 48 | packaging more difficult and get to borrow most of the upstream 49 | content for free. This allows us to modify targeted components for 50 | the purpose of our mix quickly. 51 | 52 | The RPM files are built using `mock`. This program creates a buildroot 53 | where the source code that needs to be compiled is in a separated 54 | environment. This prevents the compiler from accidentally pulling 55 | in system libraries that it wasn't meant to pull in. The buildroot 56 | will only contain libraries that are explicitly required through 57 | dependencies or as part of the buildroot toolchain. 58 | 59 | The buildroot dependencies are obtained from both local and upstream 60 | Clear Linux OS `yum` repositories. Dependencies are resolved using 61 | `dnf` or `yum` such that the buildroot is complete, before rpmbuild 62 | commences with the build process. 63 | 64 | 65 | ## `common` 66 | 67 | A repository at github specifically exists to deal with the creation 68 | and maintainance of `spec` files. It needs to be set up once and we 69 | have a handy shell script to do this. You can find it either on the 70 | `https://github.com/clearlinux/common/` repository but it's also in 71 | the `files` folder in the training repository. 72 | 73 | ``` 74 | ~ $ how-to-clear/files/clr/user-setup.sh 75 | ``` 76 | 77 | This command creates a folder called `clearlinux`. Note we are 78 | executing it in the `~` folder, but you can set it up and move it 79 | in its entirety to a new location without issues. Note we assume you 80 | cloned the training git project at this location, too. 81 | 82 | Inside the `clearlinux` workspace you'll find a `Makefile` and the 83 | `projects` and `packages` folders. For now, you won't need to deal with 84 | the `projects` folder other than knowing that this is where `autospec` 85 | is stored and the `common` project lives. These two items are the heart 86 | of the tooling that deals with making packages for Clear Linux OS. 87 | 88 | ``` 89 | ~ $ cd clearlinux 90 | ``` 91 | 92 | ### `packages` 93 | 94 | Under the `packages` folder we will store the folders that contain the 95 | `spec` and other files. From there we will create RPM files as needed 96 | for our training and testing. 97 | 98 | The common tooling allows us to reuse existing Clear Linux OS packages 99 | from upstream, and if we do, these will end up in here. If we end 100 | up making our own custom RPM files, we don't necessarily need to put 101 | them here at all, but we'll set up the workspace so that `mixer` can 102 | use the generated RPM files easily. It is easiest to do this all in 103 | a consistent way. In the end, the `mixer` tooling doesn't care where 104 | the RPM files come from and how they are generated. The RPM files 105 | can even be copied in if you want to bypass building them. 106 | 107 | We can quickly borrow some of the upstream packages if we desire. This 108 | makes for an excellent starting point and shows off the tooling that 109 | we have. 110 | 111 | ``` 112 | ~/clearlinux $ make clone_dmidecode 113 | ~/clearlinux $ cd packages/dmidecode 114 | ``` 115 | 116 | Since `dmidecode` is already present in Clear Linux OS, we can just 117 | start using that package and make modifications to it. This allows us 118 | to bypass most of the hurdles of packaging, and gives us a starting 119 | point that is as close to what Clear Linux OS uses as possible. 120 | 121 | We can make a quick change to this package, and change the revision 122 | to a new number that's higher than the Clear Linux OS version, and 123 | rebuild it. This way we can include it in our mix and know for sure 124 | that it's our version and not the version from the Clear Linux OS 125 | RPM repository instead. 126 | 127 | Add the following lines to the `excludes` file: 128 | 129 | ``` 130 | /usr/bin/biosdecode 131 | /usr/bin/ownership 132 | /usr/bin/vpddecode 133 | /usr/share/man/man8/biosdecode.8 134 | /usr/share/man/man8/ownership.8 135 | /usr/share/man/man8/vpddecode.8 136 | ``` 137 | 138 | These files are not needed by dmidecode. Due to security restrictions, 139 | they are useless on Clear Linux OS because they require `/dev/mem` 140 | to be available, which is not the case on Clear Linux OS. We can 141 | therefore just remove them from the RPM files without penalty. 142 | 143 | ``` 144 | ~/clearlinux/packages/dmidecode $ make autospec 145 | ``` 146 | 147 | We end up with several new RPM files under `results/`. This brings 148 | us to the next phase: Adding `dmidecode` into our mix content and 149 | pushing it to our target device. 150 | 151 | 152 | ## Adding `dmidecode` to our mix 153 | 154 | We need to maintain an RPM repository. An RPM repository is a 155 | combination of a few RPM files and some metadata that allows programs 156 | like `yum` and `dnf` to follow and include dependencies when we're 157 | asking it to use specific RPM files. 158 | 159 | In the mixer folder, we've already created a location for RPM files 160 | when we used the `mixer init --local-rpms` command in an earlier 161 | chapter. We will use this location to put our newly generated RPM files 162 | and thereby convey them to mixer so it can include them as needed. 163 | 164 | The `make autospec` command creates RPM files under each package. We 165 | could copy these files manually over to the `local-rpms` folder in the 166 | `mix` folder structure. In the future, we plan to have tools available 167 | to do this more efficiently. 168 | 169 | ``` 170 | ~/clearlinux/packages/dmidecode $ cp results/*x86_64*rpm ~/mix/local-rpms/ 171 | ``` 172 | 173 | Next, we can include `dmidecode` in several ways to our update 174 | content. We can create a new local bundle, we can modify an existing 175 | upstream bundle, or we can include an upstream bundle that already has 176 | `dmidecode` present. For simplicity, we'll make a new local bundle: 177 | 178 | ``` 179 | ~/mix $ mixer bundle edit dmidecode 180 | ``` 181 | 182 | Edit the file and insert the `dmidecode` package name, without quotes 183 | or leading `#` characters. Save the file, and add it to the bundle 184 | list: 185 | 186 | ``` 187 | ~/mix $ mixer bundle add dmidecode 188 | ``` 189 | 190 | Note that the order here isn't a mistake - `edit` allows you to 191 | register a new bundle, and `add` inserts it to the list of bundles 192 | that will be included in the build. 193 | 194 | Add the `dmidecode` RPM file name to the bundle, and you're ready 195 | to deploy the change: 196 | 197 | ``` 198 | ~/mix $ mixer build all 199 | ``` 200 | 201 | After this, we can go to our target device and install the new bundle 202 | after we update, and use it: 203 | 204 | ``` 205 | ~ # swupd update 206 | ~ # swupd bundle-add dmidecode 207 | ~ # dmidecode 208 | ``` 209 | 210 | We can verify that our `biosdecode` program is missing in the same way, 211 | as expected. 212 | 213 | 214 | ## Linux kernel RPM 215 | 216 | One of the more common use cases that people ask us about is how 217 | they can modify the Linux kernel on the device. This is very similar 218 | to the above demonstration, but we'll go into the kernel specific 219 | aspects in this section. 220 | 221 | First, we will modify the `linux-kvm` kernel that is already used on 222 | the system so that we don't need to switch kernels on the device. For 223 | this reason we want to make sure that we've already started with the 224 | proper kernel package for the target device. In this training, we've 225 | settled on the KVM kernel image, but for your purposes, you may need 226 | to choose a different kernel or perhaps even make a significantly 227 | new kernel. 228 | 229 | ``` 230 | ~/clearlinux $ make clone_linux-kvm 231 | ~/clearlinux $ cd packages/linux-kvm 232 | ``` 233 | 234 | We can do a ton of changes to this kernel. For this example, we'll 235 | disable an option that is easily verifiable later on. On the target, 236 | we can see that the `btrfs` filesystem is supported by the current 237 | kernel by doing: 238 | 239 | ``` 240 | ~ # modprobe btrfs 241 | ~ # grep btrfs /proc/filesystems 242 | ``` 243 | 244 | To disable this, edit the `config` file in the `linux-kvm` package 245 | folder and change the line below from: 246 | 247 | ``` 248 | CONFIG_BTRFS_FS=m 249 | ``` 250 | 251 | to: 252 | 253 | ``` 254 | # CONFIG_BTRFS_FS is not set 255 | ``` 256 | 257 | We need to make sure to increment the package release number to make sure 258 | that our new RPM version is used instead of the version on the remote 259 | RPM repository: 260 | 261 | ``` 262 | ~/clearlinux/packages/linux-kvm $ make bumpnogit 263 | ``` 264 | 265 | Now we can build the RPM files: 266 | 267 | ``` 268 | ~/clearlinux/packages/linux-kvm $ make build 269 | ``` 270 | 271 | This will take a little bit of time, as the kernel isn't small. Once 272 | the process finishes, you should have 2 binary RPM files under the 273 | `results` folder that we can give back to mixer again: 274 | 275 | ``` 276 | ~/clearlinux/packages/linux-kvm $ cp results/*x86_64.rpm ~/mix/local-rpms/ 277 | ``` 278 | 279 | Switch back to the mixer, as we can now mix in our changed kernel. 280 | 281 | ``` 282 | ~/clearlinux/packages/linux-kvm $ cd ~/mix 283 | ``` 284 | 285 | ``` 286 | ~/mix $ mixer add-rpms 287 | ~/mix $ mixer build all 288 | ``` 289 | 290 | Then we can switch to the target device and verify the changes again: 291 | 292 | ``` 293 | ~ # swupd update 294 | ~ # reboot 295 | 296 | ~ # modprobe btrfs 297 | modprobe: FATAL: Module btrfs not found ... 298 | ``` 299 | 300 | Success! 301 | 302 | ## Exercises 303 | 304 | * Use `make shell` in a package folder after building, or when a build 305 | error occurs. 306 | * Use `make autospecnew NAME= URL=` on something that isn't 307 | in Clear Linux OS yet. 308 | * Use `make repoadd` in packages that need new dependencies and build 309 | one custom package against another new custom package. 310 | * Use `make clone` or `make pull` in the top-level folder. 311 | -------------------------------------------------------------------------------- /04-mixing.md: -------------------------------------------------------------------------------- 1 | 2 | How To Clear - Mixing content and creating updates 3 | ================================================== 4 | 5 | ## What you'll learn in this chapter 6 | 7 | * Initializing a new content stream 8 | * Version numbering 9 | * Format version numbering 10 | * Working with the upstream Clear Linux OS 11 | * Creating and modifying bundles 12 | * Creating the update content 13 | * Creating an update 14 | 15 | 16 | ## Reference Documentation: 17 | 18 | The below guide skips over significant details that may be important to 19 | you. In case of doubt, please refer to the official mixer documentation. 20 | 21 | More reading: 22 | [upstream project](https://github.com/clearlinux/mixer-tools) 23 | [Mixer documentation](https://clearlinux.org/documentation/clear-linux/guides/maintenance/mixer) 24 | 25 | 26 | ## The output 27 | 28 | The `swupd` software delivery mechanism treats everything as an update. 29 | It does this by querying metadata from an update server and calculating 30 | what it needs to do and which content to use. The format of this 31 | data and metadata is simple, easy to understand, and reproduce, but 32 | `swupd` needs a lot of it. 33 | 34 | Since the update content describes every file in the OS, it is a 35 | large database that must be kept synchronized on the client system. 36 | Maintaining these metadata lists is expensive, and the system is 37 | designed to do all the hard work on the server side, so that clients 38 | only need to work on small subsets of the data to perform the needed 39 | operations to update or install components. 40 | 41 | `swupd` uses separate files for content and for metadata. Content often 42 | does not change between versions. To save space, the server only stores 43 | one copy of each file in the version that it was last updated. This 44 | content is also compressed on the server, and each piece of content 45 | is identified by a hash value that is unique for both the content of 46 | the data, and the properties of the inode on the file system. In this 47 | way, if a file changes permissions, xattr tags, or ownership, it is 48 | considered a normal update from one content unit to a new content unit. 49 | 50 | The metadata is similarly reused between different versions of the OS. 51 | This means that if a bundle doesn't change between two versions, the 52 | `swupd` client will reuse the older version. If a bundle *does* change, 53 | metadata will be updated according to the precise content changes 54 | introduced in the new version. This allows clients to reconstruct 55 | a full new view of the work that needs to be done while maximizing 56 | reuse of the already known metadata. 57 | 58 | ### Metadata 59 | 60 | The metadata that `swupd` uses is stored in Manifests. These manifest 61 | files come in two different levels. Each bundle is maintained in its 62 | own Manifest file, and there is one Manifest-of-Manifests file that 63 | contains metadata on all the Manifests that exist for each bundle. 64 | 65 | Here is an example of a Manifest file: 66 | 67 | ``` 68 | MANIFEST 25 69 | version: 21810 70 | previous: 21800 71 | filecount: 1975 72 | timestamp: 1523538780 73 | contentsize: 46217487 74 | 75 | D... 6c27df6efcd6fc401ff1bc67c970b83eef115f6473db4fb9d57e5de317eba96e 21530 /boot 76 | D... 6c27df6efcd6fc401ff1bc67c970b83eef115f6473db4fb9d57e5de317eba96e 21530 /dev 77 | D... 6c27df6efcd6fc401ff1bc67c970b83eef115f6473db4fb9d57e5de317eba96e 21530 /etc 78 | L... 3f2e21f5de1fb40955f59667b837b6004254581e64404ce0f0f692bc35a22d76 21530 /usr/bin/b2sum 79 | L... 3f2e21f5de1fb40955f59667b837b6004254581e64404ce0f0f692bc35a22d76 21530 /usr/bin/base32 80 | F... 634ed1be7098435e3a3f28f13740b260e171e9d65891a002998d1e5fc691b471 21530 /usr/bin/chattr 81 | 82 | ``` 83 | 84 | At the top of the manifest is generic metadata that various CLI 85 | commands use to make tasks such as searching produce better output. The 86 | `format version` sits right at the top and describes the epoch, or 87 | generation of the metadata format. While the format itself is largely 88 | stable, it will be changed any time a breaking change is introduced 89 | to the metadata format. Because this change requires a corresponding 90 | client update, a format boundary ensures that clients update to a 91 | certain version to get the new updater. In this way it can be used 92 | as a milepost marker. The `version` and `previous` metadata come from 93 | the actual Clear Linux OS version. 94 | 95 | After the header, there follows a long list of content metadata that 96 | describes the files, directories and links; their content/metadata 97 | hashes; the version last changed; and the various informational flags 98 | for client use. 99 | 100 | From left to right, the columns designate the type of content, the 101 | hash of the content, the content version that introduced the content, 102 | and ends with the name of the content as it appears on disk. 103 | 104 | The hash itself directly points to a content item on the update 105 | server, and can be found on the HTTP service directly. For instance, 106 | from the above list we can find the `chattr` file by the following URL: 107 | 108 | ``` 109 | https://download.clearlinux.org/update/21530/files/634ed1be7098435e3a3f28f13740b260e171e9d65891a002998d1e5fc691b471.tar 110 | ``` 111 | 112 | Note the version number in the URL isn't `21810` but this content 113 | comes from an earlier update, as listed in the Manifest (`21530`). 114 | 115 | 116 | ### Content 117 | 118 | The content that `swupd` delivers is provided to the OS in different 119 | ways to optimize the download and make it as small as possible. For the 120 | most common cases, the server calculates the minimum needed delta 121 | between the content files and creates a binary diff that is very 122 | efficient. If the client needs a clean, and full, copy of the 123 | original file, this is also provided. For the case of someone 124 | installing an entire bundle, the bundle content is provided as a 125 | tarball of the entire content all at once. 126 | 127 | The content comes in compressed tar files: 128 | 129 | ``` 130 | $ curl -s -L -O https://download.clearlinux.org/update/21530/files/634ed1be7098435e3a3f28f13740b260e171e9d65891a002998d1e5fc691b471.tar 131 | $ file 634ed1be7098435e3a3f28f13740b260e171e9d65891a002998d1e5fc691b471.tar 132 | 634ed1be7098435e3a3f28f13740b260e171e9d65891a002998d1e5fc691b471.tar: XZ compressed data 133 | ``` 134 | 135 | The compression is usually XZ, as this is the most efficient 136 | compression method for most of the content. However, in some cases, 137 | the files will be bzip2 or gzip compressed. The server automatically 138 | finds the best compression algorithm for the content. 139 | 140 | The server also creates delta packs and zero packs. These are 141 | optimizations where the server speculatively combines content based on 142 | the assumption that you will need many of them for certain actions. The 143 | client will use them if they are available, but they are entirely 144 | optional. Delta packs contain the binary deltas for changes to files 145 | made from the source update to the target update. For any new files 146 | or files with changes too large for a binary delta, a full file is 147 | included in the delta pack itself. In this way delta packs contain 148 | an update from one arbitrary version to another. 149 | 150 | These delta files are created as much as possible to provide clients 151 | with the smallest download possible between two versions. These are 152 | binary deltas made using the `bsdiff` software. They can provide a 153 | significant reduction to the needed download size, but they only work 154 | if the client has the original file already, so they are not always 155 | used. In order for the client to use the delta files, the original 156 | content must also be verified prior to patching. 157 | 158 | Similarly, zero packs define an update from one version to another, but 159 | the source version is always zero. Since the zero release contained 160 | no files, the zero pack contains all full files present in the 161 | target version. 162 | 163 | 164 | ## Mixing 165 | 166 | The `mixer-tools` suite software generates the update server 167 | content. It does this using the following inputs: 168 | 169 | * The Clear Linux OS official software update content 170 | * Local bundle definitions 171 | * Local RPM files 172 | 173 | This is exactly how the Clear Linux OS team generates the official 174 | software update content, except that the official Clear Linux OS has 175 | no `upstream` that it bases itself on, and always uses its own bundle 176 | definitions and RPM files to create all the content. The tooling is 177 | exactly the same, however. 178 | 179 | We've already seen how the update content looks after it comes out 180 | of the mixer. The goal of this chapter is to get you familiar with 181 | the methods used to create this yourself. 182 | 183 | 184 | ## Initializing the workspace 185 | 186 | The mixer tools use a simple workspace to contain all input and output 187 | in a simple folder hierarchy. We call this the mixer workspace. You 188 | can create it simply by making an empty folder: 189 | 190 | ``` 191 | ~ $ mkdir ~/mix 192 | ~ $ cd ~/mix 193 | ~/mix $ 194 | ``` 195 | 196 | From here on, we will use the `mixer` CLI tool extensively. This is a 197 | good time to review the CLI and read through the options and general 198 | usage before continuing this training. 199 | 200 | ``` 201 | ~/mix $ mixer --help 202 | Mixer is a tool used to compose OS update content and images. 203 | 204 | Usage: 205 | mixer [flags] 206 | mixer [command] 207 | 208 | Available Commands: 209 | add-rpms Add RPMs to local dnf repository 210 | build Build various pieces of OS content 211 | bundle Perform various actions on bundles 212 | config Perform config related actions 213 | help Help about any command 214 | init Initialize the mixer and workspace 215 | versions Manage mix and upstream versions 216 | 217 | Flags: 218 | --check Check all dependencies needed by mixer and quit 219 | -h, --help help for mixer 220 | --offline Skip caching upstream-bundles; work entirely with local-bundles 221 | --version Print version information and quit 222 | 223 | Use "mixer [command] --help" for more information about a command. 224 | ``` 225 | 226 | This is also your last chance to double check that `mixer` is 227 | functionally complete and you don't need to install any more Clear 228 | Linux OS bundles. 229 | 230 | ``` 231 | ~/mix $ mixer --check 232 | ``` 233 | 234 | We start by initializing mixer. Since we will do more detailed 235 | exercises later in subsequent chapters, we will add a few options to 236 | prevent having to reinitialize the workspace later. 237 | 238 | We are also going make updates to the content, both from upstream and 239 | from our own changes. We start the mix with a slightly older version 240 | of Clear Linux OS to demonstrate how this works. 241 | 242 | ``` 243 | ~/mix $ mixer init --local-rpms 244 | ``` 245 | 246 | * `init` tells mixer to create the needed configuration files and 247 | folders in the workspace 248 | * `--local-rpms` tells mixer to create folders where we can later add 249 | our own custom RPM files 250 | 251 | ## builder.conf 252 | 253 | ``` 254 | #VERSION 1.0 255 | 256 | [Builder] 257 | CERT = "/home/clr/mix/Swupd_Root.pem" 258 | SERVER_STATE_DIR = "/home/clr/mix/update" 259 | VERSIONS_PATH = "/home/clr/mix" 260 | YUM_CONF = "/home/clr/mix/.yum-mix.conf" 261 | 262 | [Swupd] 263 | BUNDLE = "os-core-update" 264 | CONTENTURL = "" 265 | VERSIONURL = "" 266 | 267 | [Server] 268 | DEBUG_INFO_BANNED = "true" 269 | DEBUG_INFO_LIB = "/usr/lib/debug" 270 | DEBUG_INFO_SRC = "/usr/src/debug" 271 | 272 | [Mixer] 273 | LOCAL_BUNDLE_DIR = "/home/clr/mix/local-bundles" 274 | LOCAL_REPO_DIR = "" 275 | LOCAL_RPM_DIR = "" 276 | DOCKER_IMAGE_PATH = "clearlinux/mixer" 277 | ``` 278 | 279 | The mixer initialization creates a `builder.conf` file that will store 280 | the basic configuration options for `mixer`. Most of the options are 281 | references to the folder structure in the workspace and some basic 282 | entries that will be needed. 283 | 284 | The items of interest in this file for deployment are the `CONTENTURL` 285 | and `VERSIONURL` entries that are needed by systems that will update 286 | against the mix content that we are generating. At a later stage 287 | we'll fill these in. 288 | 289 | The `CERT` variable sets the path where mixer stores the certificate 290 | file. The certificate file is used to sign the content so it can be 291 | verified. The software update client uses this certificate to verify 292 | the signature. Mixer automatically generates a certificate, if you 293 | do not provide a path to an existing one, and signs the manifest file. 294 | 295 | 296 | ## Bundles 297 | 298 | ``` 299 | ~/mix $ mixer bundle list 300 | bootloader (upstream bundle) 301 | kernel-native (upstream bundle) 302 | os-core (upstream bundle) 303 | os-core-update (upstream bundle) 304 | ``` 305 | 306 | The bundles in the mix are specified in the mix bundle list. Mixer 307 | stores this list as a flat file called mixbundles in the path set 308 | by the `VERSIONS_PATH` variable of the builder.conf file. Mixer 309 | automatically generates the mixbundles list file during initialization. 310 | Mixer reads and writes the bundle list file when you change the 311 | bundles of the mix. 312 | 313 | `mixer bundle list` shows a list of every bundle in the mix. Bundles 314 | can include other bundles. Those nested bundles can themselves 315 | include other bundles. When listing bundles with this command, mixer 316 | automatically recurses through the includes to show every single 317 | bundle in the mix. 318 | 319 | If you see an unexpected bundle in the list, that bundle is probably 320 | included in another bundle. Use `mixer bundle list --tree` to get a 321 | better view of how a bundle ended up in the mix. 322 | 323 | Bundles fall into two categories: upstream and local. Upstream bundles 324 | are those provided by Clear Linux OS. Local bundles are either modified 325 | upstream bundles or new local bundles. 326 | 327 | Mixer automatically downloads and caches upstream bundle definition 328 | files. These definition files are stored in the upstream-bundles 329 | directory in the workspace. Do not modify the files in this directory. 330 | This directory is simply a mirror for mixer to use. 331 | 332 | The mixer tool automatically caches the bundles for the Clear Linux OS 333 | version configured in the upstreamversion file. Mixer also cleans up 334 | old versions once they are no longer needed. You can see the available 335 | upstream bundles with the following command: 336 | 337 | ``` 338 | ~/mix $ mixer bundle list upstream 339 | 340 | ``` 341 | 342 | Local bundles are bundles that you create, or are edited versions of 343 | upstream bundles. 344 | 345 | Local bundle definition files live in the local-bundles directory. The 346 | `LOCAL_BUNDLE_DIR` variable sets the path of this directory in 347 | your builder.conf configuration file. For this example, the path is 348 | `/home/clr/mix/local-bundles`. You can see the available local bundles 349 | with the following command: 350 | 351 | ``` 352 | ~/mix $ mixer bundle list local 353 | bootloader (upstream bundle) 354 | kernel-native (upstream bundle) 355 | os-core (upstream bundle) 356 | os-core-update (upstream bundle) 357 | ``` 358 | 359 | Both the local and upstream bundle list commands accept the --tree 360 | flag to show a visual representation of the inclusion relationships 361 | between the bundles in the mix. 362 | 363 | 364 | ## Create the initial mix content 365 | 366 | ``` 367 | ~/mix $ mixer build all 368 | 369 | 370 | ``` 371 | 372 | This command creates all the needed content for the version we have 373 | selected and produces a functional `version 10` content stream that 374 | can be used to deploy to targets. 375 | 376 | Each time this command is run, the `version` is updated to `+ 10` and 377 | a new update content set is created. If you execute the function a few 378 | times, you'll see the following result in the `www` folder structure: 379 | 380 | ``` 381 | ~/mix $ ls update/www/ 382 | 0 10 20 30 version 383 | ``` 384 | 385 | 386 | ## Updating 387 | 388 | If you desire to update the upstream version of your mix and pull in 389 | upstream changes, you can do this selectively or automatically. 390 | 391 | ``` 392 | ~/mix $ mixer versions update --upstream-version 22180 393 | ``` 394 | 395 | Or: 396 | 397 | ``` 398 | ~/mix $ mixer versions update --upstream-version latest 399 | ``` 400 | 401 | An important note here is that you can go **back** upstream versions as 402 | long as you're not crossing a `format version` change. If you want to 403 | roll back an upstream change or skip a version, this is all supported. 404 | After doing the `versions update` change, you can simply rebuild the 405 | content again and you are finished. 406 | 407 | 408 | ``` 409 | ~/mix $ mixer build all 410 | ``` 411 | 412 | ## What else to try 413 | 414 | * Edit an upstream bundle and change the package list to include or 415 | exclude packages. 416 | * Validate a modified bundle. 417 | * Downgrade your mix to an older upstream version. 418 | --------------------------------------------------------------------------------