├── .gitattributes ├── 0-init.md ├── 1-user.md ├── 2-dev.md ├── 3-adv.md ├── 4-compute-storage.md ├── 5-network.md ├── README.md ├── hack ├── api.md ├── automation.md ├── com.md ├── db.md ├── framework.md ├── packaging.md ├── service.md ├── spec.md ├── testing.md ├── ui.md └── usage.md └── primer ├── capc.md ├── capi-capc.pdf ├── dp.md ├── dsys.md ├── efficiency.md ├── git.md ├── index.md └── networking-refresh.pdf /.gitattributes: -------------------------------------------------------------------------------- 1 | *.mp4 filter=lfs diff=lfs merge=lfs -text 2 | *.ogv filter=lfs diff=lfs merge=lfs -text 3 | -------------------------------------------------------------------------------- /0-init.md: -------------------------------------------------------------------------------- 1 | # Getting Started 2 | 3 | Video: https://s3-eu-west-1.amazonaws.com/shapeblue-engineering-videos/hackerbook/0-init/0-init.mp4 4 | 5 | * [About](#about) 6 | * [Prerequisites](#prerequisites) 7 | * [What is CloudStack?](#what-is-cloudstack) 8 | * [Know the Project](#know-the-project) 9 | * [Core Concepts](#core-concepts) 10 | * [Workstation Setup](#workstation-setup) 11 | * [References](#references-additional-books-and-readings--list) 12 | 13 | ## About 14 | 15 | This fast learning course is designed to onboard new CloudStack developers 16 | and get them to speed in few weeks by means of 17 | [hacking](http://www.catb.org/esr/faqs/hacker-howto.html). Following guildelines 18 | are recommended: 19 | 20 | - Learn by programming, debugging and experimentation 21 | - Avoid being stuck with only theory 22 | - Do not attempt to understand everything the first time 23 | - Do not read everything end to end 24 | - Use Stackoverflow and Google for terms, concepts, programming questions 25 | - Learn to ask the smart way: http://www.catb.org/esr/faqs/smart-questions.html 26 | 27 | ## Prerequisites 28 | 29 | We assume you know: 30 | 31 | - Basic Linux adminstration and terminal usage 32 | - Programming in Java, Python, shell scripting 33 | - Development: git, maven3, IntelliJ IDEA, MySQL 34 | - Tools: bash/zsh, ssh, scp, vi/emacs, wget/curl, tmux 35 | - Basic virtualization (try KVM, VirtualBox, VMware fusion/workstation etc) 36 | 37 | ## What is CloudStack? 38 | 39 | - Official answer: https://cloudstack.apache.org/about.html 40 | - 2 minute video: https://www.youtube.com/watch?v=oJ4b8HFmFTc 41 | - CloudStack 101: https://www.youtube.com/watch?v=pASzZR57V_8 42 | 43 | Hour long talks: (optional) 44 | - Building IaaS with Apache CloudStack: https://www.youtube.com/watch?v=bxEL06BPGNw 45 | - Building Clouds with Apache CloudStack: https://www.youtube.com/watch?v=4qFFwyK9hos 46 | 47 | ## Know the Project 48 | 49 | - [Project website](https://cloudstack.apache.org) 50 | - [Join dev, users and other MLs](https://cloudstack.apache.org/mailing-lists.html) 51 | - IRC channels on freenode: \#cloudstack, \#cloudstack-dev 52 | 53 | Useful links: 54 | - Docs: http://docs.cloudstack.apache.org/ 55 | - API docs: https://cloudstack.apache.org/api.html 56 | - Git repo: https://github.com/apache/cloudstack 57 | - Pull requests: https://github.com/apache/cloudstack/pulls 58 | - Bug tracking: https://github.com/apache/cloudstack/issues 59 | - Old bug tracking: https://issues.apache.org/jira/browse/CLOUDSTACK 60 | 61 | Contribution guideline: 62 | https://github.com/apache/cloudstack/blob/master/CONTRIBUTING.md 63 | 64 | ## Core Concepts 65 | 66 | An IaaS (infrastructure as a service) platform provide means to partition and 67 | primarily consume compute, storage and network resources usually by means of 68 | virtualization. 69 | 70 | Apache CloudStack is an IaaS platform with support for several hypervisors 71 | such as KVM, VMware and XenServer. The usual setup consists of a management 72 | server and the resoures it should manage such as the hypervisor and storage 73 | hosts and networking configuration such as IP address ranges, VLANs etc. 74 | 75 | The management server is a monolith that provides orchestration and control 76 | plane accessible via query based http APIs through its UI and CLI. Various 77 | authentication mechanisms such as the default authentication (pbkdf2), ldap, 78 | saml2, api/secret key based etc are supported. Authorization is supported 79 | through `Roles` which have access to a set/subset of APIs. 80 | 81 | For user management CloudStack has `Domains` that have `Accounts` that have 82 | `Users`, and all `Accounts` have some `Role`. It has `Projects` that allows 83 | users across domains to participate as a team. 84 | 85 | For cloud infrastructure management, CloudStack has concepts of organization 86 | units much like filesystems have files and directories. Resources are in a 87 | `Region` that represents a management server environment that can have `Zones` 88 | that may be a datacenter. `Zone` can have `Pods` (like racks) that can have 89 | `Clusters` that can have `Hosts` which runs your workload i.e. VMs. In addition, 90 | there are `Primary Storage` (zone or cluster wide) that have the disks of an 91 | instance and `Secondary Storage` (zone wide) that have disk templates, ISO 92 | images and snapshots. 93 | 94 | CloudStack supports many networking models and topologies: 95 | 96 | - AWS-styled shared/flat network with L3 isolation (security groups) 97 | - NAT-ed network with single (isolated network) and multiple tiers (VPC) with L2 98 | isolation such as VLANs with L3 services (nat, routing, firewall, 99 | port-forwarding, dhcp, dns etc) provided by a virtual router 100 | - Pure L2 network with VLAN isolation 101 | 102 | What is L2/L3, VLAN etc? We'll cover that in later chapters. 103 | 104 | Lastly, CloudStack has several features including events and customizations via 105 | compute, network, storage/disk, system offerings, and limits/thresholds/settings 106 | for various resources. 107 | 108 | CloudStack architecture overview: https://www.youtube.com/watch?v=FLRtAzp_YuM 109 | 110 | **Recommended reading**: 111 | http://docs.cloudstack.apache.org/en/latest/conceptsandterminology/index.html 112 | 113 | ## Workstation Setup 114 | 115 | The workstation setup varies depending on personal preference. One can buy a performant and expensive workstation laptop or a more optimal setup of a cost-effective thin/light laptop (such as MacBooks) and a powerful and cost-effective mini PC. For example, getting a decent x86 (i7/i9) laptop with 32-64GB RAM, or a MacBook Air M3/M4 24-32GB RAM (usually in USD 1200-1800 range) with a x86 mini-PC (i7/i9, 64-96GB RAM, 1TB NVMe, usually in USD 800-1200 range) which can used with [wireguard and mbx](https://github.com/shapeblue/mbx/tree/main#mbx-) for development purposes. 116 | 117 | Suggested laptop spec: 118 | - Intel x64 i7/i9 with VTx/VTd enabled or equivalent AMD, with 16+ CPU cores, 32-64GB RAM & Ubuntu 24.04 (LTS) 119 | - MacBook Air M3/M4 24-32GB RAM with a 32-64GB RAM, with x86 Mini PC with 12-24 CPU cores and 64-96GB RAM 120 | 121 | Reference laptop models/series: Dell XPS, HP ZBook/Elite/Spectre, Think P/X Extreme 122 | 123 | Other Laptop spec/build purchase questions to review: 124 | - Does laptop have good input devices, ports, extensionabilty? Could you use a type-C hub for ports? 125 | - Is the laptop durable, and have sturdy hinges, flex-resistant screen and keyboard? 126 | - Is the laptop model/brand serviceable in your area? 127 | - What is the post-sales support and warranty? 128 | 129 | ### Software Setup 130 | 131 | Check if hardware virutalization is enabled on the workstation 132 | 133 | apt install cpu-checker 134 | kvm-ok 135 | 136 | Setup your workstation with Ubuntu 24.04 and install following: 137 | 138 | sudo apt-get update 139 | sudo apt-get dist-upgrade 140 | 141 | # general packages 142 | sudo apt-get install vim git subversion mercurial patch rsync curl wget sed openssh-client gpg gnupg2 build-essential gzip bzip2 zip unzip p7zip-full p7zip-rar 143 | # cloudstack related development 144 | sudo apt-get install openjdk-11-jdk maven mysql-client mysql-server nfs-kernel-server quota genisoimage qemu-kvm qemu-utils libvirt-daemon virt-manager ipmitool jq uuid uuid-runtime python3 python3-dev python3-setuptools python3-openssl python3-pip libffi-dev build-essential libssl-dev dpkg-dev libffi-dev rpm rpm2cpio bridge-utils iproute2 iptables ebtables ethtool vlan ipset tcpdump telnet fakeroot ca-certificates 145 | # (optional) misc and development related 146 | sudo apt-get install zsh guake kazam ipython3 pv sshpass htop tmux tig vlc mutt bc cmake cowsay gcc g++ net-tools wireshark openvpn network-manager-openvpn clisp 147 | 148 | In order to launch a VM from the `virt-manager` (GUI), Perform the following steps 149 | 150 | sudo systemctl enable --now libvirtd 151 | sudo systemctl start libvirtd 152 | sudo systemctl status libvirtd 153 | sudo usermod -aG kvm $USER 154 | sudo usermod -aG libvirt $USER 155 | 156 | Logout and Logback so that the changes are applied 157 | 158 | sudo virt-manager 159 | 160 | (Preferred) Install software using `snap`: 161 | 162 | sudo snap install slack --classic 163 | sudo snap install intellij-idea-community --classic 164 | sudo snap install code --classic 165 | 166 | Note: Intellij IDEA is the preferred and recommended IDE for developing CloudStack 167 | 168 | (Optional) Gnome extensions: 169 | 170 | https://extensions.gnome.org/extension/1060/timezone/ (useful extension to see team around the world) 171 | https://extensions.gnome.org/extension/120/system-monitor/ 172 | 173 | Productivity recommendations: 174 | - Develop muscle memory for git, maven, vi/vim and IntelliJ IDEA. 175 | - Create and maintain a `dotfiles` repo for your env configuration. For example, 176 | see https://github.com/rohityadavcloud/dotfiles. 177 | - Learn to touch type faster. 178 | 179 | ## References: Additional Books and Readings List 180 | 181 | CloudStack: 182 | - [The Apache Way](http://theapacheway.com/) 183 | - [Apache CloudStack bylaws](http://cloudstack.apache.org/bylaws.html) 184 | - [Apache CloudStack docs](http://docs.cloudstack.apache.org) 185 | 186 | Java: 187 | - Effective Java 188 | - Java Concurrency in Practice 189 | - Java Performance 190 | - Clean Code 191 | 192 | Design and software engineering: 193 | - The Pragmatic Programmer 194 | - Release It!: Design and Deploy Production-Ready Software (Pragmatic Programmers) 195 | - [SOLID papers](http://butunclebob.com/ArticleS.UncleBob.PrinciplesOfOod) 196 | 197 | Distributed systems: 198 | - [Distributed Systems for fun and profit](http://book.mixu.net/distsys/) 199 | - [Perspective on the CAP theorem](https://groups.csail.mit.edu/tds/papers/Gilbert/Brewer2.pdf) (paper) 200 | - [Time, clocks and the ordering of events in a distributed system](https://amturing.acm.org/p558-lamport.pdf) (paper) 201 | - [A comprehensive study of C/CRDTs](https://hal.inria.fr/file/index/docid/555588/filename/techreport.pdf) (paper) 202 | -------------------------------------------------------------------------------- /1-user.md: -------------------------------------------------------------------------------- 1 | # Test Drive CloudStack 2 | 3 | The best way to learn about CloudStack is to start as a user, learn how to setup 4 | and install it and test drive its features. 5 | 6 | * [Installing CloudStack](#installing-cloudstack) 7 | * [Validate your VM](#validate-your-vm) 8 | * [Configure Networking](#configure-networking) 9 | * [Install Packages](#install-packages) 10 | * [MySQL Server and CloudStack DB](#mysql-server-and-cloudstack-db) 11 | * [NFS Server](#nfs-server) 12 | * [SystemVM Template](#systemvm-template) 13 | * [KVM Setup](#kvm-setup) 14 | * [Security Configuration](#security-configuration) 15 | * [Setup Management Server](#setup-management-server) 16 | * [Deploying Advanced Zone](#deploying-advanced-zone) 17 | * [Setup Zone](#setup-zone) 18 | * [Setup Network](#setup-network) 19 | * [Add Resources](#add-resources) 20 | * [Finishing Deployment](#finishing-deployment) 21 | * [Using CloudStack](#using-cloudstack) 22 | * [Access](#access) 23 | * [CloudStack Feature Set](#cloudstack-feature-set) 24 | * [CloudStack Ops](#cloudstack-ops) 25 | 26 | ## Installing CloudStack 27 | 28 | Video: https://s3-eu-west-1.amazonaws.com/shapeblue-engineering-videos/hackerbook/1-user/1-user-task1.mp4 29 | 30 | **Recommended Reading**: 31 | http://docs.cloudstack.apache.org/en/latest/installguide/ 32 | 33 | On your workstation, with KVM enabled and installed, using the 34 | [virt-manager](https://virt-manager.org/) create a new VM 35 | using [Ubuntu](https://www.ubuntu.com/download/server) 22.04 ISO with at least 36 | 30GB disk, 4 CPUs and 8GB RAM. Before starting the VM, go to the VM's 37 | setting and tick `Copy host CPU configuration`. Start the VM and complete the 38 | installation. 39 | 40 | NOTE: DO NOT install or experiment anything from this chapter on your laptop, 41 | but do them in a VM. 42 | 43 | Next, find out the IP of the VM, ensure that you're able to SSH into the host. 44 | Ensure that you can SSH as `root` user with a known password. Also ensure 45 | that the `universe` *apt* repository is enabled. 46 | 47 | Next, we'll be building a local all-in-a-box cloud using CloudStack. You may 48 | optionally refer to the CentOS based quick installation guide here: 49 | http://docs.cloudstack.apache.org/en/latest/quickinstallationguide/qig.html 50 | 51 | You'll use the `Ubuntu 22.04` host you've created to install CloudStack. This 52 | guide assumes that the VM is on a 192.168.122.0/24 network, can run KVM on it. 53 | 54 | Reference: 55 | https://rohityadav.cloud/blog/cloudstack-kvm/ 56 | 57 | ### Validate your VM 58 | 59 | SSH into your VM and make sure that hardware acceleration is available: 60 | 61 | cat /proc/cpuinfo| grep vmx | wc -l 62 | 63 | All in a box setup: 64 | - CloudStack Management server and Usage server 65 | - MySQL server 66 | - NFS server 67 | - KVM hypervisor and CloudStack agent 68 | 69 | Install basic packages: 70 | 71 | apt-get install openntpd vim htop bridge-utils 72 | 73 | ### Configure Networking 74 | 75 | Create a file `/etc/netplan/01-netcfg.yaml` with following contents but remove 76 | any yaml file at /etc/netplan: (change interface names, ip ranges accordingly) 77 | 78 | network: 79 | version: 2 80 | renderer: networkd 81 | ethernets: 82 | ens3: 83 | dhcp4: false 84 | dhcp6: false 85 | optional: true 86 | bridges: 87 | cloudbr0: 88 | addresses: [192.168.122.10/24] 89 | routes: 90 | - to: default 91 | via: 192.168.122.1 92 | nameservers: 93 | addresses: [1.1.1.1,8.8.8.8] 94 | interfaces: [ens3] 95 | dhcp4: true 96 | dhcp6: false 97 | parameters: 98 | stp: false 99 | forward-delay: 0 100 | 101 | Save the file and apply network config, finally reboot: 102 | 103 | netplan generate 104 | netplan apply 105 | 106 | Your connection will break since the VM's IP has changed, SSH again: 107 | 108 | ssh root@192.168.122.10 109 | 110 | ### Install Packages 111 | 112 | Note that now your VM should be accessible on the address 192.168.122.10 113 | (or IP of your VM). SSH into it and install CloudStack management server 114 | and all other packages: 115 | 116 | mkdir -p /etc/apt/keyrings 117 | wget -O- http://packages.shapeblue.com/release.asc | gpg --dearmor | sudo tee /etc/apt/keyrings/cloudstack.gpg > /dev/null 118 | echo deb [signed-by=/etc/apt/keyrings/cloudstack.gpg] http://packages.shapeblue.com/cloudstack/upstream/debian/4.20 / > /etc/apt/sources.list.d/cloudstack.list 119 | apt-get update -y 120 | apt-get install cloudstack-management cloudstack-usage cloudstack-agent mysql-server nfs-kernel-server quota qemu-kvm 121 | 122 | ### MySQL Server and CloudStack DB 123 | 124 | Make a note of the MySQL server's root user password. Configure InnoDB settings 125 | in mysql server's `/etc/mysql/mysql.conf.d/mysqld.cnf` as follows: 126 | 127 | [mysqld] 128 | 129 | server_id = 1 130 | sql-mode="STRICT_TRANS_TABLES,NO_ENGINE_SUBSTITUTION,ERROR_FOR_DIVISION_BY_ZERO,NO_ZERO_DATE,NO_ZERO_IN_DATE,NO_ENGINE_SUBSTITUTION" 131 | innodb_rollback_on_timeout=1 132 | innodb_lock_wait_timeout=600 133 | max_connections=1000 134 | log-bin=mysql-bin 135 | binlog-format = 'ROW' 136 | 137 | Restart MySQL server and setup CloudStack database: 138 | 139 | systemctl restart mysql 140 | cloudstack-setup-databases cloud:cloud@localhost --deploy-as=root: -i 192.168.122.10 141 | 142 | ### NFS Server 143 | 144 | Create NFS exports: 145 | 146 | mkdir -p /export/primary /export/secondary 147 | echo "/export *(rw,async,no_root_squash,no_subtree_check)" > /etc/exports 148 | exportfs -a 149 | 150 | Configure and restart NFS server: 151 | 152 | sed -i -e 's/^RPCMOUNTDOPTS="--manage-gids"$/RPCMOUNTDOPTS="-p 892 --manage-gids"/g' /etc/default/nfs-kernel-server 153 | sed -i -e 's/^STATDOPTS=$/STATDOPTS="--port 662 --outgoing-port 2020"/g' /etc/default/nfs-common 154 | echo "NEED_STATD=yes" >> /etc/default/nfs-common 155 | sed -i -e 's/^RPCRQUOTADOPTS=$/RPCRQUOTADOPTS="-p 875"/g' /etc/default/quota 156 | service nfs-kernel-server restart 157 | 158 | ### SystemVM Template 159 | 160 | CloudStack systemvm template is a special purpose guest template based on Debian 161 | that provides a building block for creating service VMs in CloudStack such as 162 | the SSVM, CPVM, virtual routers etc. 163 | 164 | Note: From ACS 4.16 onwards, this is automatically seeded and you don't need to do anything. 165 | This section has been added just for reference. 166 | 167 | For 4.15 and older versions, something like the following are needed. 168 | 169 | wget http://packages.shapeblue.com/systemvmtemplate/4.15/systemvmtemplate-4.15.0-kvm.qcow2.bz2 170 | /usr/share/cloudstack-common/scripts/storage/secondary/cloud-install-sys-tmplt \ 171 | -m /export/secondary -f systemvmtemplate-4.15.0-kvm.qcow2.bz2 -h kvm \ 172 | -o localhost -r cloud -d cloud 173 | 174 | ### KVM Setup 175 | 176 | Enable VNC for console proxy: 177 | 178 | sed -i -e 's/\#vnc_listen.*$/vnc_listen = "0.0.0.0"/g' /etc/libvirt/qemu.conf 179 | 180 | Enable libvirtd in listen mode and configure non-TLS setup: 181 | 182 | sed -i -e 's/.*libvirtd_opts.*/libvirtd_opts="-l"/' /etc/default/libvirtd # For Ubuntu 18.04/20.04 183 | sed -i -e 's/^LIBVIRTD_ARGS=""/LIBVIRTD_ARGS="--listen"/' /etc/default/libvirtd # For Ubuntu 22.04 184 | echo 'listen_tls=0' >> /etc/libvirt/libvirtd.conf 185 | echo 'listen_tcp=1' >> /etc/libvirt/libvirtd.conf 186 | echo 'tcp_port = "16509"' >> /etc/libvirt/libvirtd.conf 187 | echo 'mdns_adv = 0' >> /etc/libvirt/libvirtd.conf 188 | echo 'auth_tcp = "none"' >> /etc/libvirt/libvirtd.conf 189 | systemctl mask libvirtd.socket libvirtd-ro.socket libvirtd-admin.socket libvirtd-tls.socket libvirtd-tcp.socket # For Ubuntu 20.04/22.04 190 | systemctl restart libvirtd 191 | 192 | Note: while adding KVM host (default, via ssh) it may fail on newer distros which has OpenSSH version 7+ which has deprecated some legacy algorithms. This is only necessary for older ACS versions and you may not need to do this. To fix that the `sshd_config` on the KVM host may temporarily be changed to following before adding the KVM host in CloudStack: 193 | 194 | PubkeyAcceptedKeyTypes=+ssh-dss 195 | HostKeyAlgorithms=+ssh-dss 196 | KexAlgorithms=+diffie-hellman-group1-sha1 197 | 198 | ### Security Configuration 199 | 200 | Configure firewall to allow port accessibility: 201 | 202 | # configure iptables 203 | NETWORK=192.168.122.0/24 204 | iptables -A INPUT -s $NETWORK -m state --state NEW -p udp --dport 111 -j ACCEPT 205 | iptables -A INPUT -s $NETWORK -m state --state NEW -p tcp --dport 111 -j ACCEPT 206 | iptables -A INPUT -s $NETWORK -m state --state NEW -p tcp --dport 2049 -j ACCEPT 207 | iptables -A INPUT -s $NETWORK -m state --state NEW -p tcp --dport 32803 -j ACCEPT 208 | iptables -A INPUT -s $NETWORK -m state --state NEW -p udp --dport 32769 -j ACCEPT 209 | iptables -A INPUT -s $NETWORK -m state --state NEW -p tcp --dport 892 -j ACCEPT 210 | iptables -A INPUT -s $NETWORK -m state --state NEW -p tcp --dport 875 -j ACCEPT 211 | iptables -A INPUT -s $NETWORK -m state --state NEW -p tcp --dport 662 -j ACCEPT 212 | 213 | apt-get install iptables-persistent 214 | 215 | # Disable apparmour on libvirtd 216 | ln -s /etc/apparmor.d/usr.sbin.libvirtd /etc/apparmor.d/disable/ 217 | ln -s /etc/apparmor.d/usr.lib.libvirt.virt-aa-helper /etc/apparmor.d/disable/ 218 | apparmor_parser -R /etc/apparmor.d/usr.sbin.libvirtd 219 | apparmor_parser -R /etc/apparmor.d/usr.lib.libvirt.virt-aa-helper 220 | 221 | ### Setup Management Server 222 | 223 | Your installation and configuration is complete! You can now start the 224 | management server as follows: 225 | 226 | cloudstack-setup-management 227 | 228 | Open up http://192.168.122.10:8080/client in Chrome (recommended) or any 229 | modern browser and log into the CloudStack UI using the username `admin` and 230 | password `password. 231 | 232 | To troubleshoot, check the management server logs for errors: 233 | 234 | tail -f /var/log/cloudstack/management/management-server.log 235 | 236 | ## Deploying Advanced Zone 237 | 238 | Video: https://s3-eu-west-1.amazonaws.com/shapeblue-engineering-videos/hackerbook/1-user/1-user-task2.mp4 239 | 240 | In this section, you'll create an advanced KVM-based zone in your CloudStack 241 | environment. Skip the basic zone guided tour installation wizard, and proceed 242 | to the `Infrastructure` Tab > Zone in the UI. 243 | 244 | ### Setup Zone 245 | 246 | Click on add zone, select advanced zone and provide following configuration: 247 | 248 | Name - any name 249 | Public DNS 1 - 8.8.8.8 250 | Internal DNS1 - 192.168.122.1 251 | Hypervisor - KVM 252 | 253 | ### Setup Network 254 | 255 | Use the default, which is `VLAN` isolation method on a single physical nic (on 256 | the host) that will carry all traffic types (management, public, guest etc). 257 | 258 | Public traffic configuration: 259 | 260 | Gateway - 192.168.122.1 261 | Netmask - 255.255.255.0 262 | VLAN/VNI - (leave blank) 263 | Start IP - 192.168.122.11 264 | End IP - 192.168.122.30 265 | 266 | Pod Configuration: 267 | 268 | Name - any name 269 | Gateway - 192.168.122.1 270 | Start/end reserved system IPs - 192.168.122.31 - 192.168.122.50 271 | 272 | Guest traffic: 273 | 274 | VLAN range: 100-200 275 | 276 | ### Add Resources 277 | 278 | Create a cluster with following: 279 | 280 | Name - any name 281 | Hypervisor - Choose KVM 282 | 283 | Add your default/first host: 284 | 285 | Hostname - 192.168.122.10 286 | Username - root 287 | Password - 288 | 289 | Note: please check/enable ssh for `root` user in `sshd_config` 290 | 291 | Add primary storage: 292 | 293 | Name - any name 294 | Scope - zone-wide 295 | Protocol - NFS 296 | Server - 192.168.122.10 297 | Path - /export/primary 298 | 299 | Add secondary storage: 300 | 301 | Provider - NFS 302 | Name - any name 303 | Server - 192.168.122.10 304 | Path - /export/secondary 305 | 306 | Next, click `Launch Zone` which will perform following actions: 307 | 308 | Create Zone 309 | Create Physical networks: 310 | - Add various traffic types to the physical network 311 | - Update and enable the physical network 312 | - Configure, enable and update various network provider and elements such as the virtual network element 313 | Create Pod 314 | Configure public traffic 315 | Configure guest traffic (vlan range for physical network) 316 | Create Cluster 317 | Add host 318 | Create primary storage (also mounts it on the KVM host) 319 | Create secondary storage 320 | Complete zone creation 321 | 322 | Finally, confirm and enable the zone! 323 | 324 | ### Finishing Deployment 325 | 326 | Wait for the system VMs to start before you can use your newly deployed zone. 327 | You may troubleshoot using the management server logs for any errors. 328 | 329 | **Recommended Exercise**: 330 | - Deploy a Basic KVM based zone using the guided tour wizard 331 | 332 | ## Using CloudStack 333 | 334 | Congratulations, you've successfully installed and deployed a zone in your test 335 | VM. In this section, you'll learn CloudStack administration and usage. 336 | 337 | Once your zone is enabled, before you proceed wait for two the system VMs to 338 | come online - the Console Proxy VM (CPVM) and the Secondary Storage VM (SSVM). 339 | You can track them in the UI at Infrastructure > System VMs. After they are up 340 | you can use your CloudStack setup for testing various features and resource 341 | lifecycles. 342 | 343 | **Recommended Reading**: 344 | http://docs.cloudstack.apache.org/en/latest/adminguide/ 345 | 346 | ### Access 347 | 348 | CloudStack has a query based HTTP API endpoint that can be used to access and 349 | use the management server. There are two common modes of access: 350 | 351 | - UI: The CloudStack UI is a VueJS and Ant Design. The UI is by default 352 | accessible at `http://:8080/client`. 353 | - CLI: The CloudStack cloudmonkey (cmk) is the official CLI, it can be installed 354 | on Linux, Mac and Windows to access the API endpoint by default at 355 | `http://:8080/client/api`. 356 | 357 | ### CloudStack Feature Set 358 | 359 | CloudStack feature sets can be broadly divided into three types: 360 | - Business features 361 | - Cloud (user) features 362 | - Infrastructure (admin/management) features 363 | 364 | #### Business 365 | 366 | - Roles: Set of allow/reject APIs. The `Admin` and `User` are two widely used 367 | default roles. 368 | - Accounts: Record of an individual user/customer or a team, all the resources 369 | in CloudStack are owned by an account. The `system` and `admin` are default 370 | CloudStack accounts. All accounts have a `role`. 371 | - Users: A member/user of an account. Users in an account can be treated as 372 | aliases to the account. All accounts have one or more users. 373 | - Domains: Accounts in a group. All account in a domain, by default in `/` the 374 | root domain. 375 | - Authentication: Means of checking if you are who you say you are. 376 | - Authorization: Means of checking if you can access resource `X` or have 377 | suitable privilege for other actions. 378 | - LDAP and SAML: Two widely used authentication plugins in CloudStack. 379 | - Projects: For organizing users and resources. 380 | - Regions: Represents a CloudStack installation. Largely, an incomplete feature. 381 | - Events: Used for auditing, monitoring, usage records and billing, and 382 | event-driven integration with other external systems. 383 | 384 | #### Cloud 385 | 386 | - Instance: Virtual machines. 387 | - Volume: Disks of virtual machines. 388 | - Volume snapshot: Checkpoint or snapshot of disk of a VM. 389 | - VM snapshot: Checkpoint or snapshot of disk and memory of a VM. 390 | - Template: Virtual machines disk that has a guest OS installed and can 391 | directly be used for creation of a VM. 392 | - ISO: CDrom files for installation of a guest OS in a VM. 393 | - Network: 394 | - Shared Network: a flat L3 network where VMs directly receive a public IP, a 395 | virtual router is usually deployed that provides DHCP and DNS services. 396 | - L2 Network: a flat L2 network with no services or virtual routers. 397 | - Isolated Network: a NAT-ed network that is provided by a virtual router that 398 | itself takes public IPs and provides a RFC1918 L3 private network with 399 | services to guest VMs like DHCP, DNS, NAT/SNAT, port-forwarding, firewall, 400 | vpn, load balancing etc. 401 | - VPC: similar to isolated network but provides multiple guest network tiers 402 | and ACLs for those network tiers. 403 | 404 | #### Infrastructure 405 | 406 | CloudStack has several organization units such as zones, pods, clusters, hosts 407 | etc. 408 | 409 | - Zone: Represents a datacenter or an availability zone 410 | - Basic zone: massively scalable AWS styled flat-network cloud usually with 411 | isolation and multitenancy implemented at host/hypervisor level by L2 412 | firewall (ebtables) rules by a feature called security groups. 413 | - Advanced zone: in additional to shared networks, provides enterprise 414 | network models with isolated and VPC networks with isolation provided by 415 | VLAN, VXLAN etc. 416 | - Pod: Represents a rack. 417 | - Cluster: Represents a group of hosts. 418 | - Host: Hypervisor host that runs workloads/VMs. For example, KVM, VMware and 419 | XenServer. 420 | - Storage: 421 | - Primary storage: Storage pool for virtual machine's disks. 422 | - Local storage: When disks are on same machine as the VM. 423 | - Shared storage: When disks are on a different machine, for example on a 424 | NFS server/host, Ceph, etc. 425 | - Secondary Storage: Image storage pool for templates, ISOs and snapshots. 426 | - Physical network: Allows configuration of physical network, traffic labels, 427 | VLAN and IP address ranges for guest, public and private networks. 428 | - System VMs: Special service VMs created and managed by CloudStack management 429 | server that implements an infrastructural service. They are based on a Debian 430 | based guest template called the `systemvmtemplates`. All system VMs run `sshd` 431 | on port `3922` and can be accessed as follows based on the hypervisor: 432 | - KVM and XenServer: ssh to the link-local IP of the systemvm on port `3922` 433 | using local `~/.ssh` key file. 434 | - VMware: ssh to the private IP of the systemvm from a management server host 435 | using the ssh key file at `/var/cloudstack/management/.ssh` location. 436 | 437 | Notable systemvm types: 438 | - SSVM: Secondary storage virtual machine provides means to manage the 439 | secondary storage, register/copy templates/ISOs, host snapshots and copy 440 | templates/ISOs to primary storage for consumption. 441 | - CPVM: Console proxy virtual machine provides means to access console of a 442 | VM. They act as a VNC/RDP proxy between the end user (browser) and the 443 | hypervisor where VMs run. 444 | - Virtual router: They provide router functionalities for various network 445 | models. 446 | - Service offerings: Provides admins a way to create a catalogue of resource 447 | offerings to end users such as: 448 | - Compute 449 | - Disk 450 | - Network 451 | - System 452 | - Limits and thresholds: Provides means for admins to define usage limits and 453 | threshold for various resource. Popular example is to set limit on 454 | accounts/domains and cpu/memory thresholds on clusters etc. 455 | 456 | ### CloudStack Ops 457 | 458 | CloudStack management/usage/agent config directories and logs location: 459 | 460 | | Service | Config | Logs | 461 | | ------- | ------ | ---- | 462 | | cloudstack-management | `dir:/etc/cloudstack/management/`, `file:/etc/default/cloudstack-management` | `dir:/var/log/cloudstack/management/` | 463 | | cloudstack-usage | `dir:/etc/cloudstack/usage/`, `file:/etc/default/cloudstack-usage` | `dir:/var/log/cloudstack/usage/` | 464 | | cloudstack-agent | `dir:/etc/cloudstack/agent/`, `file:/etc/default/cloudstack-agent` | `dir:/var/log/cloudstack/agent/` | 465 | 466 | Go through each of the directories and files, notable files: 467 | - `server.properties`: management server config file 468 | - `db.properties`: database config file 469 | - `agent.properties`: agent config file 470 | - `log4j-cloud.xml`: log config file 471 | - `key`: encryption password file 472 | 473 | Troubleshooting references: 474 | - http://docs.cloudstack.apache.org/en/latest/adminguide/troubleshooting.html 475 | - https://www.slideshare.net/ShapeBlue/cloudstack-top-5-technical-issues-and-troubleshooting 476 | - https://cwiki.apache.org/confluence/display/CLOUDSTACK/SSVM%2C+templates%2C+Secondary+storage+troubleshooting 477 | 478 | **Recommended Exercises**: 479 | - Repeat fresh installation and deployment with an EL (AlmaLinux, Rocky Linux or RHEL) environment https://www.youtube.com/watch?v=9gXEmWbgX2o 480 | - Learn to read the CloudStack management server logs, `tail -f` the logs and 481 | deploy a fresh virtual machine with a new network, read and try to understand 482 | all the steps that happen during VM deployment. 483 | - Attempt CloudStack [Automation](hack/automation.md) challenge using 484 | CloudMonkey and Ansible. (16 hours) 485 | -------------------------------------------------------------------------------- /2-dev.md: -------------------------------------------------------------------------------- 1 | # Basic CloudStack Development 2 | 3 | * [CloudStack Development 101](#cloudstack-development-101) 4 | * [Setting up Development Environment](#setting-up-development-environment) 5 | * [Getting the source code](#getting-the-source-code) 6 | * [Setup IDE](#setup-ide) 7 | * [Setup MySQL Server](#setup-mysql-server) 8 | * [Setup NFS storage](#setup-nfs-storage) 9 | * [Configure Environment](#configure-environment) 10 | * [Building CloudStack](#building-cloudstack) 11 | * [Testing CloudStack](#testing-cloudstack) 12 | * [Unit Testing](#unit-testing) 13 | * [Functional Testing](#functional-testing) 14 | * [Simulator Based Development](#simulator-based-development) 15 | * [MonkeyBox Based Development](#monkeybox-based-development) 16 | * [Debugging CloudStack](#debugging-cloudstack) 17 | * [Using Logs](#using-logs) 18 | * [Using IDE](#using-ide) 19 | * [Instrumentation](#instrumentation) 20 | * [CloudStack Packaging](#cloudstack-packaging) 21 | * [Contributing to CloudStack](#contributing-to-cloudstack) 22 | * [Basic Development Topics](#basic-development-topics) 23 | 24 | ## CloudStack Development 101 25 | 26 | Overview video: https://s3-eu-west-1.amazonaws.com/shapeblue-engineering-videos/hackerbook/2-dev/2-dev-overview.mp4 27 | 28 | Guided walkthrough video: https://s3-eu-west-1.amazonaws.com/shapeblue-engineering-videos/hackerbook/2-dev/2-dev-guided-walkthrough.mp4 29 | 30 | From an application setup and functioning perspective, a typical CloudStack 31 | deployment consists of the following: 32 | 33 | - CloudStack Management server(s) 34 | - CloudStack Agent (on KVM) 35 | - CloudStack Usage server(s) 36 | 37 | The CloudStack management server is a monolith Java application that embeds 38 | the control plane, orchestrator and overall cloud controller. 39 | 40 | The management server (sometimes written as mgmt server) has three types of 41 | APIs: 42 | 43 | - Platform API (REST-like or query-based API for end users and admins) 44 | - Agent API (ServerResource based json/API that uses command-answer pattern 45 | handled by an implementation that can talk to a hardware resource) 46 | - Plugin API (Set of Java interfaces and APIs to allow extension and 47 | modification of CloudStack) 48 | 49 | The management server monolith has various layers: 50 | 51 | - Presentation or API: the layer that implements and handles REST-like or query 52 | based APIs. 53 | - Service/Business/Orchestration layer: the layer that usually handles API 54 | requests, manages business entities and participates in resource control. 55 | - DB/Data access: the data access layer implements set of building blocks to talk 56 | to the database (MySQL). 57 | - Kernel: Provides building blocks, polling, IPC, message bus, implements 58 | orchestration and controller for compute, network, storage, security etc. 59 | - Agent/Cluster management: the layer that handles distributed system of mgmt 60 | server nodes and agents, and handles RPC mechanisms. 61 | 62 | ![](https://cwiki.apache.org/confluence/download/attachments/29687953/image2012-4-27%2012-43-50.png) 63 | 64 | **Recommended Reading**: 65 | - https://cwiki.apache.org/confluence/display/CLOUDSTACK/Development+101 66 | - https://cwiki.apache.org/confluence/display/CLOUDSTACK/How+to+build+CloudStack 67 | - https://cwiki.apache.org/confluence/display/CLOUDSTACK/CloudStack+packages+and+dependencies 68 | - https://cwiki.apache.org/confluence/display/CLOUDSTACK/Putting+CloudStack+together 69 | - https://cwiki.apache.org/confluence/display/CLOUDSTACK/Data+Access+Layer 70 | - https://cwiki.apache.org/confluence/display/CLOUDSTACK/Exceptions+and+logging 71 | - https://cwiki.apache.org/confluence/display/CLOUDSTACK/Coding+conventions 72 | - http://docs.cloudstack.apache.org/en/latest/developersguide 73 | 74 | Old videos: 75 | - https://www.youtube.com/watch?v=D1K_8rvDhic 76 | - https://www.youtube.com/watch?v=uSwisRfJVdM 77 | 78 | ## Setting up Development Environment 79 | 80 | Video: https://s3-eu-west-1.amazonaws.com/shapeblue-engineering-videos/hackerbook/2-dev/2-dev-chapter.mp4 81 | 82 | The recommended development environment is Linux based, in this course Ubuntu 83 | Linux 24.04+ is preferred. Run the following to install packages required for 84 | CloudStack development on Ubuntu: (see first chapter on software installation) 85 | 86 | $ sudo apt-get install openjdk-11-jdk maven mysql-server mysql-client bzip2 nfs-common uuid-runtime python3-setuptools ipmitool genisoimage nfs-kernel-server quota 87 | 88 | Older CloudStack versions may require older jdk/jre version, therefore setup, 89 | install and learn to use `jenv`: http://www.jenv.be and also run `jenv 90 | enable-plugin maven`. 91 | 92 | The recommended setup is to run MySQL and NFS servers locally on your laptop 93 | and the hypervisor in a VM or an external host. The CloudStack management and 94 | usage server can be run using maven or via an IDE. 95 | 96 | Tip: get the latest maven from https://maven.apache.org/download.cgi 97 | 98 | ### Getting the source code 99 | 100 | Apache CloudStack source code can be cloned from the following remotes: 101 | 102 | - https://github.com/apache/cloudstack.git (Github, preferred) 103 | - https://gitbox.apache.org/repos/asf/cloudstack.git (Gitbox, ASF hosted) 104 | 105 | Create a Gitbub account in case you do not have one already: https://github.com 106 | 107 | You may generate a public SSH key (if do not already have one at: `~/.ssh/id_rsa.pub`): 108 | ```` 109 | ssh-keygen -t rsa -b 4096 -C "user_email@gmail.com" 110 | eval "$(ssh-agent -s)" 111 | ssh-add ~/.ssh/id_rsa 112 | ```` 113 | 114 | Add your SSH key to you Github account: 115 | - Go to your `Github` account 116 | - Click on Settings and select SSH and GPG keys 117 | - Click on `New SSH` key 118 | - Add a title and paste the content of your key 119 | - Click `Add SSH` key 120 | 121 | You may create a personal workspace and clone the repository, for example: 122 | 123 | mkdir -p ~/lab/ 124 | cd ~/lab/ 125 | git clone https://github.com/apache/cloudstack.git 126 | 127 | The recommended directory structure may look something like: 128 | 129 | ```bash 130 | ~/lab/ 131 |    ├── cloudstack 132 |    └── cloudmonkey 133 |    └── shapeblue 134 |     └── ... private repositories ... 135 |    └── ... other projects ... 136 | ``` 137 | 138 | Reference and reading resources: 139 | - ProGit: https://git-scm.com/book/ 140 | - Try Git: https://try.github.io/ 141 | - Learn Git: https://www.codecademy.com/learn/learn-git 142 | 143 | Development tools to learn: 144 | - git, tig 145 | - maven 146 | - IntelliJ IDEA IDE, vim 147 | - java, javac, jps, jstack, jmap 148 | - IntelliJ remote debugger, VisualVM, MAT 149 | 150 | ### Setup IDE 151 | 152 | Setup IntelliJ IDEA (recommended) or any IDE of your choice. Get IntelliJ IDEA 153 | community edition from: 154 | 155 | https://www.jetbrains.com/idea/download/#section=linux 156 | 157 | Or you can install then using snap: (preferred) 158 | 159 | sudo snap install intellij-idea-community --classic 160 | 161 | Start the IDE, configure as needed and import CloudStack as: 162 | - Click on import project 163 | - Select the cloned `cloudstack` directory 164 | - Select `Maven` as build system 165 | - Select options suitably and import! 166 | 167 | You may configure IDEA per your preference or use this [settings 168 | jar](https://github.com/rohityadavcloud/dotfiles/tree/main/intellij) which you can import 169 | in IDEA as `File > Import Settings > select jar file`. 170 | 171 | ### Setup MySQL Server 172 | 173 | After installing MySQL server, configure the following settings in its config 174 | file such as at `/etc/mysql/mysql.conf.d/mysqld.cnf` and restart mysql-server: 175 | 176 | [mysqld] 177 | 178 | sql-mode="STRICT_TRANS_TABLES,NO_ENGINE_SUBSTITUTION,ERROR_FOR_DIVISION_BY_ZERO,NO_ZERO_DATE,NO_ZERO_IN_DATE,NO_ENGINE_SUBSTITUTION" 179 | server_id = 1 180 | innodb_rollback_on_timeout=1 181 | innodb_lock_wait_timeout=600 182 | max_connections=1000 183 | log-bin=mysql-bin 184 | binlog-format = 'ROW' 185 | # add the following for MySQL 8.x 186 | default-authentication-plugin=mysql_native_password 187 | 188 | Tip: ensure that your mysql server only listens on `127.0.0.1` and reset the 189 | mysql root password to blank to get CloudStack db-deployment using `mvn` work 190 | out of the box, run `sudo mysql -u root -e "ALTER USER 'root'@'localhost' 191 | IDENTIFIED WITH mysql_native_password BY ''"` (tested with mysql 5.7+). 192 | 193 | ### Setup NFS storage 194 | 195 | After installing nfs server, configure the exports: 196 | 197 | echo "/export *(rw,async,no_root_squash,no_subtree_check)" > /etc/exports 198 | mkdir -p /export/testing/primary /export/testing/secondary 199 | 200 | Tip: You may want to have separate secondary storage for each version of 201 | CloudStack. Rename and add more directories to the `/export/testing` path as and 202 | when required. 203 | 204 | The following is one way to seed a systemvmtemplate, for example for the 4.15: 205 | 206 | wget http://packages.shapeblue.com.s3-eu-west-1.amazonaws.com/systemvmtemplate/4.20/systemvmtemplate-4.20.0-x86_64-kvm.qcow2.bz2 207 | ./scripts/storage/secondary/cloud-install-sys-tmplt \ 208 | -m /export/testing/secondary -f systemvmtemplate-4.20.0-x86_64-kvm.qcow2.bz2 \ 209 | -h kvm -o localhost -r cloud -d cloud 210 | 211 | Notes: 212 | - Please check the branch you're building/working against, and use the suitable 213 | systemvmtemplate version for your branch. For `master`, use the latest systemvmtemplate 214 | or build your own using packer, in the source see `tools/appliance/README.md` for details. 215 | - Deploy the systemvm template after deploying the CloudStack database, refer to the 216 | `Building CloudStack` section below. 217 | 218 | ## Configure Environment 219 | 220 | Put the following in your `sudoers` file using visudo etc. to allow processes 221 | owned by your user (such as the CloudStack management server) to execute some 222 | privileged commands: 223 | 224 | Cmnd_Alias CLOUDSTACK = /bin/mkdir, /bin/mount, /bin/umount, /bin/cp, /bin/chmod, /usr/bin/keytool, /bin/keytool 225 | 226 | Defaults:username !requiretty 227 | 228 | username ALL=(ALL) NOPASSWD:CLOUDSTACK 229 | 230 | Tip: replace `username` with your Linux or Mac OS username. 231 | 232 | ## Building CloudStack 233 | 234 | Noredist CloudStack builds requires additional jars that may be installed from: 235 | 236 | https://github.com/shapeblue/cloudstack-nonoss 237 | 238 | Clone the above repository and execute the install script to install the noredist jar dependencies: 239 | 240 | $ cd /path/to/cloudstack-nonoss/repo 241 | $ bash -x install-non-oss.sh 242 | 243 | To build CloudStack with `noredist` (this include vmware plugins etc): 244 | ``` 245 | $ cd /path/to/cloudstack 246 | $ mvn clean install -Dnoredist -P developer,systemvm 247 | ``` 248 | Deploy CloudStack database using: 249 | 250 | $ mvn -Pdeveloper -pl developer -Ddeploydb 251 | 252 | Run management server using: 253 | 254 | $ mvn -pl :cloud-client-ui jetty:run -Dnoredist -Djava.net.preferIPv4Stack=true 255 | 256 | 257 | To deploy an environment, you can use the deploy datacenter script as: 258 | 259 | $ python3 tools/marvin/marvin/deployDataCenter.py -i /path/to/config.cfg 260 | 261 | Note : The sample cfg files are present in setup/dev 262 | 263 | Note: Use pip3 to install/upgrade any dependencies for Marvin 264 | 265 | $ pip3 install --upgrade tools/marvin/dist/Marvin-*.tar.gz 266 | 267 | Example of how to run a marvin based integration test: (change parameters suitably) 268 | 269 | $ nosetests --with-xunit --xunit-file=results.xml --with-marvin --marvin-config=/path/to/config.cfg -s -a tags=advanced --hypervisor=KVM test/integration/smoke/test_vm_life_cycle.py 270 | 271 | Note: Python version 3.10 may require you to install pynose 272 | 273 | $ pip install pynose 274 | 275 | When needed, the usage server can be started using: 276 | 277 | $ mvn -P usage -Drun -Dpid=$$ -pl usage 278 | 279 | Note: due to bug in `surefire` plugin you may need to use 280 | `-Djdk.net.URLClassPath.disableClassPathURLCheck=true`. 281 | 282 | Build tips: 283 | 284 | - In case the build fails on Ubuntu 22.04, please make sure there is correct alias for python 285 | 286 | ``` 287 | $ sudo apt install python-is-python3 288 | ``` 289 | or create an alias. Place this into ~/.bashrc or ~/.zshrc file: 290 | 291 | ``` 292 | $ alias python=python3 293 | ``` 294 | - For an iterative styled development and code building, you may use the mvn 295 | `-pl` or `--projects` flag to which you can pass comma separate list of maven 296 | projects or paths you've changed and the `client` (which builds a fatjar based 297 | on all other projects), for example if you only changed `api` and `vmware`: 298 | 299 | $ mvn clean install -Dnoredist -P developer,systemvm -pl api,plugins/hypervisor/vmware,client 300 | 301 | - You may skip unit tests and even build with parallel threads: 302 | 303 | $ mvn clean install -Dnoredist -P developer,systemvm -DskipTests -T8 304 | 305 | - You will need to build and install/update marvin library every time you change 306 | a major CloudStack branch and/or if you add/remove/modify CloudStack APIs. 307 | 308 | - To bring up the Cloudstack Ui 309 | ``` 310 | $ cd /path/to/cloudstack/ui 311 | $ sudo apt install nodejs npm 312 | $ npm install 313 | $ npm run serve 314 | ``` 315 | Make sure to set CS_URL=http://localhost:8080/client on .env.local file on ui. 316 | 317 | You should be able to run the management server on http://localhost:5050 318 | 319 | 320 | ## Testing CloudStack 321 | 322 | ### Unit Testing 323 | 324 | Unit tests in CloudStack are generally written with JUnit4 that also use 325 | mockito, powermock and sometimes wiremock. You may learn more about JUnit4 and 326 | usage of other libraries using existing unit tests (Ctrl+Shift+t in IntelliJ to 327 | see an existing class's unit test) or by using following references: 328 | 329 | - https://junit.org/junit4/faq.html 330 | - http://www.vogella.com/tutorials/JUnit/article.html 331 | - https://javacodehouse.com/blog/junit-tutorial 332 | 333 | ### Functional Testing 334 | 335 | Functional or integration tests for CloudStack are written in Python 2.7 using 336 | the Marvin test library. At the core of it, these are basically Python 337 | `unittest` tests where the following test-probes are available: 338 | 339 | - API client: run and get result of CloudStack APIs 340 | - DB client: query MySQL database 341 | - SSH client: allow remote access into a host endpoint/port 342 | 343 | Typical integration tests are run using nose (a test runner), and an integration 344 | test basically is a Python class that extends `cloudstackTestCase` which are 345 | based on `unittest.case.TestCase`. The nature and mechanism of how these tests 346 | run, make them integration or functional tests. 347 | 348 | You may use following references to know more about using Python `unittest` 349 | framework: 350 | 351 | - https://docs.python.org/2/library/unittest.html 352 | - https://pythontest.com/framework/unittest/unittest-introduction 353 | - https://www.geeksforgeeks.org/unit-testing-python-unittest 354 | 355 | The [functional testing](hack/testing.md) exercise will cover in much detail 356 | how to write marvin based integration tests and use the utilities, probes and 357 | other building blocks of the Marvin library. As an example, you may look at the 358 | [test/integration/smoke/test_dynamicroles.py](https://github.com/apache/cloudstack/blob/master/test/integration/smoke/test_dynamicroles.py) marvin test. 359 | 360 | For developing and debuging Marvin tests you may want to use a python IDE, for example PyCharm (https://www.jetbrains.com/pycharm/download/) and its community edition. 361 | 362 | Once installed PyCharm, you'll need to configure the project interpreter into its preferences. When the 363 | interpreter is set, Marvin lib shouold appear in the available packages 364 | 365 | In order to execute a marvin test you'll need to setup a Run Configuration 366 | - In PyCharm, click Run -> Edit Configurations -> add new python test configuration (nosetest) 367 | - Then add the following line into Additional Configurations `--with-marvin --marvin-config=[path-to-config-file] -s -a tags=advanced --hypervisor=[xenserver|kvm|vmware|simulator] --zone=[zone-id]` 368 | - Target -> Script Path and select the particular test file you want to execute for example: `test/integration/smoke/test_dynamicroles.py` 369 | Now you should be able to run/debug any Marvin test. 370 | 371 | ## Simulator Based Development 372 | 373 | CloudStack has a mocked hypervisor called `simulator` that may be used for 374 | development of presentation, service, db and orchestration layer features. 375 | 376 | You can use the `simulator` flag to build the simulator hypervisor plugin as: 377 | 378 | $ mvn clean install -Dsimulator -P developer,systemvm 379 | 380 | Note in addition to deploying database the following must be run: 381 | 382 | $ mvn -Pdeveloper -pl developer -Ddeploydb 383 | $ mvn -Pdeveloper -pl developer -Ddeploydb-simulator 384 | 385 | Run the management server using the `simulator` flag as well: 386 | 387 | $ mvn -pl :cloud-client-ui jetty:run -Djava.net.preferIPv4Stack=true -Dsimulator 388 | 389 | Simulator based environment can be deployed using: 390 | 391 | $ pip3 install --upgrade tools/marvin/dist/Marvin-*.tar.gz 392 | $ python3 tools/marvin/marvin/deployDataCenter.py -i setup/dev/advanced.cfg 393 | 394 | ## MBX Based Development 395 | 396 | `mbx` (MonkeyBox) is a VM appliance that runs a hypervisor such as KVM/VMware/XenServer 397 | in a Intel-VTx/AMD-v enabled VM on KVM (your laptop). Follow the MonkeyBox 398 | project for details: https://github.com/shapeblue/mbx 399 | 400 | Video - https://www.youtube.com/watch?v=a8xuWHrzw88 401 | 402 | Old Video for older monkeybox tool: (please refer to the above repo for latest `mbx` usage) 403 | https://s3-eu-west-1.amazonaws.com/shapeblue-engineering-videos/hackerbook/2-dev/2-dev-monkeybox.mp4 404 | 405 | ## Debugging CloudStack 406 | 407 | To debug any java process started by maven, you can export the following in 408 | your shell (or include this by default in zshrc or bashrc): 409 | 410 | export MAVEN_OPTS="$MAVEN_OPTS -Xdebug -Xrunjdwp:transport=dt_socket,address=8787,server=y,suspend=n" 411 | 412 | For installed CloudStack management server, you can change its `JAVA_DEBUG` in 413 | the `/etc/default/cloudstack-management` ([see PR](https://github.com/apache/cloudstack/pull/2535/files)). 414 | 415 | To remote-debug the KVM agent, put the following in 416 | `/etc/default/cloudstack-agent` in your monkeybox and restart cloudstack-agent: 417 | 418 | JAVA=/usr/bin/java -Xdebug -Xrunjdwp:transport=dt_socket,address=8787,server=y,suspend=n 419 | 420 | This will then allow you to attach a remote debugger on port `8787` (or any 421 | other port you may have configured). 422 | 423 | For reference, please keep in mind the various ports used by CloudStack: 424 | - 8080: Authenticated API service 425 | - 8096: Unauthenticated API service 426 | - 9090: Cloudstack Management server cluster 427 | - 45219: JMX console 428 | - 8250: Management server port for agents 429 | - 3922: SSH port for systemvms 430 | - 3306: MySQL server 431 | - 22: KVM, XenServer/XAPI 432 | - 443: XenServer, vCenter 433 | - 53: DNS 434 | - 111/2049: NFS service/communication port, secondary storage VM 435 | - 860/3260: iSCSI communication port for iSCSI software connector 436 | - 7080: AWS API server (deprecated in recent versions) 437 | 438 | ### Using Logs 439 | 440 | For a typical CloudStack installation, logs are found per service as follows: 441 | 442 | - cloudstack-management: `dir:/var/log/cloudstack/management/` 443 | - cloudstack-usage: `dir:/var/log/cloudstack/usage/` 444 | - cloudstack-agent: `dir:/var/log/cloudstack/agent/` 445 | 446 | However, when management server is launched using `maven` the logs will be in 447 | the root directory of your source directory: 448 | - vmops.log: the management server log 449 | - api.log: the API log 450 | - usage.log: the usage server log 451 | 452 | Start the management server using `mvn` and try to read, follow and understand 453 | what may be happening when the management server starts. For example, the first 454 | thing you'll notice is that it: 455 | - Loads module context (`spring-bootstrap-context.xml`) and creates a hierarchy 456 | of modules to load 457 | - It starts loading modules in a certain order and starts integrity checks 458 | - It configures various CloudStack components, instantiates registeries etc. 459 | - Discovers API, starts various CloudStack components 460 | - Finally the management server API service is available at port 8080 (default) 461 | and log statement like the following are seen: 462 | ```bash 463 | 2018-10-27 00:53:06,417 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-4108c094) (logid:505420fb) We are good, no orphan management server msid in host table is found 464 | 2018-10-27 00:53:06,420 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-4108c094) (logid:505420fb) No inactive management server node found 465 | 2018-10-27 00:53:06,437 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-4108c094) (logid:505420fb) Detected management node joined, id:1, nodeIP:127.0.0.1 466 | ``` 467 | 468 | Tip: Several CloudStack operations are scheduled and executed by its job 469 | framework which gives such operations a unique job ID such as `job-123` and this 470 | makes it easier to grep and investigate logs of a passed/failed job using the 471 | management server logs of a multi-tenant (huge) CloudStack deployment. 472 | 473 | **Challenge**: Deploy a VM (either using Simulator or KVM/monkeybox) and tail 474 | through the logs and try to read and make sense of various steps that were 475 | performed between the API request to deploy a VM was initiated and when the VM 476 | came online. 477 | 478 | ### Using IDE 479 | 480 | With the Java process (management server or agent) launched with above mentioned 481 | flags, in IntelliJ IDEA go to `Run > Debug... > Edit configuration > Add > 482 | Remote` and configure it suitably. Put breakpoint in code, start the debugger, 483 | and wait for code execution such as an API request to reach the breakpoint to 484 | debug. 485 | 486 | **Challenge**: Add a breakpoint at the `execute()` of the `DeployVMCmdByAdmin` 487 | class and step through the code to explore what gets executed when CloudStack 488 | deploys a VM. Reference you findings against what you learnt by going through 489 | the logs. 490 | 491 | ### Instrumentation 492 | 493 | Several instrumentation and debugging tools exists that may be used to debug a 494 | general Java/JVM application. 495 | 496 | To list Java processes on a host: 497 | 498 | jps -l # or ps aux | grep java 499 | 500 | To get the thread dump of a process: 501 | 502 | jstack -l 503 | 504 | To get the heap dump of a process: 505 | 506 | jmap -dump:format=b,file=heap.bin 507 | 508 | Few popular instrumentation tools: 509 | 510 | 1. Visual VM 511 | 512 | - Good tool for monitoring Java process CPU performance and memory, visualize 513 | threads, profile performance and memory usage, take and explore thread dumps, 514 | take and browse heap dumps and analyze core dumps. 515 | - Download: https://visualvm.github.io/ 516 | - https://visualvm.github.io/documentation.html 517 | - https://docs.oracle.com/javase/8/docs/technotes/guides/visualvm/ 518 | 519 | Debugging remotely: 520 | 521 | - Setup socks proxy to the remote server: ssh -v -D 9696 root@ 522 | - Configure VisualVM to use this as SOCKS proxy in Tools->Options->Network 523 | - Run CloudStack management server with option: -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=1099 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false 524 | - Add remote host IP and add JMX connection on it on port 1099 525 | 526 | **Case study**: https://github.com/apache/cloudstack/pull/2314 527 | 528 | 2. Eclipse MAT 529 | 530 | - Memory Analyzer Tool (MAT) is great heap analyzer and useful for finding 531 | memory leaks. 532 | - Download: https://www.eclipse.org/mat 533 | 534 | **Case study**: https://github.com/apache/cloudstack/pull/1729 535 | 536 | Other notable mentions: 537 | - Java Mission Control (jmc): https://www.oracle.com/technetwork/java/javaseproducts/mission-control/index.html 538 | - JProfiler: https://www.ej-technologies.com/products/jprofiler/overview.html 539 | 540 | ## CloudStack Packaging 541 | 542 | CloudStack is typically packages and shipped as a deb or rpm repository. The 543 | docker container images from https://hub.docker.com/u/shapeblue can be used to 544 | build CloudStack for CentOS (rpm) and Ubuntu (rpm). 545 | 546 | The pre-requisite is that all the build and runtime dependencies are installed 547 | on a build system (CentOS or Debian based) along with any nonoss dependencies. 548 | 549 | Building on Debian/Ubuntu: 550 | 551 | # Option 1: Building using commands 552 | cd /root/of/the/cloudstack/repo 553 | dpkg-buildpackage -uc -us -b 554 | 555 | # Option 2: Building using script 556 | cd /root/of/the/cloudstack/repo 557 | bash -x packaging/build-deb.sh # run with -h for help 558 | 559 | Building on CentOS: 560 | 561 | # For el6/centos6 562 | cd /root/of/the/cloudstack/repo 563 | bash -x packaging/package.sh -p noredist -d centos63 # run with -h for help 564 | 565 | # For el7/centos7 566 | cd /root/of/the/cloudstack/repo 567 | bash -x packaging/package.sh -p noredist -o rhel7 -d centos7 # run with -h for help 568 | 569 | ## Contributing to CloudStack 570 | 571 | For bug reporting create an issue: https://github.com/apache/cloudstack/issues 572 | 573 | For any bugfix or improvement change(s) send a pull request: https://github.com/apache/cloudstack/pulls 574 | 575 | For feature submission the typical process is as follows: 576 | 577 | - Write a high level functional specification (FS), this can be either shared on cwiki 578 | or the content be shared via a Github issue or PR. 579 | - Start a discussion on dev@ mailing list with reference to the FS and/or any 580 | issues. You can use `[DISCUSS]` in the email subject. Continue the discussion 581 | and engage with the community if anyone posts any feedback or has a question. 582 | - Complete the feature, send a pull request (PR). 583 | - Participate in code review, iterate implementation, request committers for 584 | review and merging. Typically every PR will be reviewed and (regression) 585 | tested. It is expected feature/bugfix PRs to have unit and integration tests. 586 | - Send documentation PR. 587 | 588 | **Case Study**: Dynamic Roles feature 589 | - Functional specification: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Dynamic+Role+Based+API+Access+Checker+for+CloudStack 590 | - Mailing list: https://lists.apache.org/thread/610q6b96vkvqddltxco1dl6b0kxszm8s 591 | - Jira/bug ticket: https://issues.apache.org/jira/browse/CLOUDSTACK-8562 592 | - Pull request and reviews: https://github.com/apache/cloudstack/pull/1489 593 | - Documentation PR: https://github.com/apache/cloudstack-docs-admin/pull/37 594 | 595 | ## Basic Development Topics 596 | 597 | | # | Topic | Est | 598 | | - | ----- | --- | 599 | | #1 | [Functional Spec](hack/spec.md) | 8 hours | 600 | | #2 | [API Development](hack/api.md) | 16 hours | 601 | | #3 | [Service Development](hack/service.md) | 16 hours | 602 | | #4 | [DB Development](hack/db.md) | 16 hours | 603 | | #5 | [Pluggable Framework and Plugin development](hack/framework.md) | 8 hours | 604 | | #6 | [IPC/RPC](hack/com.md) | 8 hours | 605 | | #7 | [Usage Development](hack/usage.md) | 8 hours | 606 | | #8 | [UI Development](hack/ui.md) | 8 hours | 607 | | #9 | [Functional Testing](hack/testing.md) | 8 hours | 608 | | #10 | [Packaging](hack/packaging.md) | 4 hours | 609 | 610 | Tip: Do api, service and db exercises together. 611 | -------------------------------------------------------------------------------- /3-adv.md: -------------------------------------------------------------------------------- 1 | # Advanced CloudStack Development 2 | 3 | * [CloudStack Bootstrapping](#cloudstack-bootstrapping) 4 | * [Jobs Framework](#jobs-framework) 5 | * [RPC Framework](#rpc-framework) 6 | * [Agent Framework](#agent-framework) 7 | * [SystemVMs](#systemvms) 8 | * [Self Study](#self-study) 9 | 10 | ## CloudStack Bootstrapping 11 | 12 | Building blocks: spring, managed-context, api, component-api 13 | 14 | Manager, ManagerBase 15 | Adapter, AdapterBase 16 | ComponentLifecycle, ComponentLifecycleBase, 17 | ComponentContext 18 | 19 | 20 | Deployment descriptor: WEB-INF/web.xml 21 | 22 | The `web.xml` has: 23 | - Servlets (CloudStartupServlet, ApiServlet, ConsoleProxyServlet) 24 | - Configuration (classpath:log4j-cloud.xml) 25 | - Listeners (CloudStackContextLoaderListener) 26 | - Mappings or filters (what url for what servlet) 27 | - Error pages 28 | - Security constraints 29 | 30 | The CloudStartupServlet's init() runs 31 | `ComponentContext.initComponentsLifeCycle()` that kickstart the management 32 | server module/component loading, registration, configuration and start. 33 | 34 | (Related: see how usage server starts in `UsageServer.java`) 35 | 36 | The `framework/spring/module` implements a spring based module framework 37 | that discovers and constructs the hierarchy of CloudStack module based on the 38 | module.properties file (defined in most CloudStack maven projects). It is 39 | initialized via the `CloudStackContextLoaderListener`. 40 | 41 | https://cwiki.apache.org/confluence/display/CLOUDSTACK/Putting+CloudStack+together 42 | 43 | ## Jobs Framework 44 | 45 | AsyncJobManager 46 | 47 | ## RPC Framework 48 | 49 | Cluster and Agent framework 50 | CloudStack clustering, agent LB etc. Agent/Cluster manager 51 | 52 | ## SystemVMs 53 | 54 | ### SystemVM Template and systemvm.iso 55 | 56 | Packer, build etc. (note: systemvm.iso has been deprecated with 4.17 onwards) 57 | https://cwiki.apache.org/confluence/display/CLOUDSTACK/SystemVm.iso 58 | 59 | ### CPVM and SSVM 60 | 61 | ### Virtual Routers 62 | 63 | How it builds, programs etc. 64 | VR python codebase 65 | 66 | ## Self Study 67 | 68 | - API framework: 69 | - API handling and execution: ApiServlet, ApiServer/ApiServerService, DispatchWorker, DispatchTask, ApiResponseSerializer 70 | - API jobs execution: ApiDispatcher, ApiAsyncJobDispatcher 71 | - API security: APIAuthenticationManager, APIAuthenticator, UserAuthenticator, APIChecker 72 | - Misc: [CallContext](https://cwiki.apache.org/confluence/display/CLOUDSTACK/Using+CallContext), LogContext, ManagedContext, ManagedThreadLocal 73 | - DB framework: GenericDao, GenericDaoBase, TransactionCallback, TransactionLegacy, StateMachine, GenericQueryBuilder, GenericSearchBuilder, Filter, EntityManagerImpl 74 | - CA framework: CAProvider, RootCAProvider, Link, [TLS v1.2](https://tls.ulfheim.net), [TLS v1.3](https://tls13.ulfheim.net/) 75 | - Orchestration engine: 76 | - Agent related: AgentAttache, ConnectedAgentAttache, DirectAgentAttache, ClusteredAgentAttache, ClusteredDirectAgentAttache, AgentManagerImpl, ClusteredAgentManagerImpl 77 | - Cluster related: ClusterBasedAgentLoadBalancerPlanner, ClusteredAgentRebalanceService 78 | - VM related: ClusteredVirtualMachineManagerImpl, VirtualMachineManagerImpl, VirtualMachinePowerStateSyncImpl, VmWork (cloud-engine-components-api) 79 | - Misc Plugin Development: 80 | - Host and Storage pool allocator plugin: http://docs.cloudstack.apache.org/en/latest/developersguide/alloc.html 81 | - Storage plugin: http://docs.cloudstack.apache.org/en/latest/developersguide/plugins.html#storage-plugins 82 | - Misc videos: 83 | - State of Cloud https://www.youtube.com/watch?v=RJMSUDTl6Ds 84 | - Scalability in CloudStack https://www.youtube.com/watch?v=JqktvtVKnX8 (first half) 85 | - Misc reading: 86 | - https://cloudierthanthou.wordpress.com/2017/02/22/design-patterns-in-orchestrators-part-1/ 87 | - https://cloudierthanthou.wordpress.com/2017/02/23/design-patterns-in-orchestrators-part-2/ 88 | - https://cloudierthanthou.wordpress.com/2017/06/22/design-patterns-in-orchestrators-transfer-of-desired-state-part-3n/ 89 | 90 | Misc links: 91 | - https://cwiki.apache.org/confluence/display/CLOUDSTACK/201+-+Orchestration+and+Plugins 92 | - https://cwiki.apache.org/confluence/display/CLOUDSTACK/Alex%27s+Architecture+Notes 93 | - https://cwiki.apache.org/confluence/display/CLOUDSTACK/VM+Deployment+Planning+and+Resource+Allocation 94 | - https://cwiki.apache.org/confluence/display/CLOUDSTACK/High+Availability+Developer%27s+Guide 95 | -------------------------------------------------------------------------------- /4-compute-storage.md: -------------------------------------------------------------------------------- 1 | Note: The follow sections requires going through code and study the mentioned classes. 2 | 3 | # Hypervisor 4 | 5 | ServerResource 6 | ServerResourceBase 7 | VirtualRouterDeployer 8 | CommandWrapper 9 | 10 | ## KVM 11 | 12 | LibvirtComputingResource, KvmServerDiscoverer/LibvirtServerDiscoverer 13 | 14 | ## XenServer 15 | 16 | CitrixResourceBase, XenServerResourceNewBase, XcpServerDiscoverer 17 | 18 | ## VMWare 19 | 20 | VmwareResource 21 | VmwareServerDiscoverer 22 | VmwareHostService 23 | 24 | How cloudstack orchestrates, programs hypervisor cmd-answers 25 | 26 | TOs (transfer objects), service layer -> hypervisor (serverresource) 27 | 28 | # Storage 29 | 30 | https://cwiki.apache.org/confluence/display/CLOUDSTACK/Storage+subsystem+2.0 31 | 32 | StorageProcessor 33 | StoragePoolResource 34 | StorageSubsystemCommandHandler 35 | 36 | ## NFS and Local Storage 37 | 38 | SecondaryStorageResource 39 | LocalSecondaryStorageResource 40 | NfsSecondaryStorageResource 41 | PremiumSecondaryStorageResource 42 | SecondaryStorageDiscoverer 43 | 44 | Focus on NFS, how SSVM+NFS works, mount/unmount/copy/delete etc, use of apache2 45 | 46 | # Case Study: Simulator 47 | 48 | AgentRoutingResource 49 | SimulatorDiscoverer 50 | SimulatorSecondaryDiscoverer 51 | -------------------------------------------------------------------------------- /5-network.md: -------------------------------------------------------------------------------- 1 | # Hacking Networking 2 | 3 | Outline: (self-study, refer docs http://docs.cloudstack.apache.org/) 4 | 5 | - Overview 6 | - Basic Vs Advanced Zone 7 | - Network Models: L2, Shared, Isolated, VPC 8 | - Isolation: SG, VLAN, VXLAN etc. 9 | - Virtual Router 10 | - Network tools, usage and debugging 11 | 12 | 101: 13 | - Networking 101: https://iximiuz.com/en/posts/computer-networking-101/ 14 | - Networking Fundamentals 101 video modules: https://www.youtube.com/watch?v=bj-Yfakjllc&list=PLIFyRwBY_4bRLmKfP1KnZA6rZbRHtxmXi&index=2 15 | - Networking in CloudStack https://www.youtube.com/watch?v=FBG7ko8z3o8 16 | 17 | 201: 18 | - Professional Networking: https://www.youtube.com/watch?v=MancoYQcxh8&list=PLHh55M_Kq4OCZOAxs2KZyCawhX38YR154 19 | 20 | Reading list: 21 | - Switching and Routing: https://www.practicalnetworking.net/series/packet-traveling/packet-traveling/ 22 | - ARP: https://www.practicalnetworking.net/series/arp/address-resolution-protocol/ 23 | - NAT, DNAT, SNAT: https://www.practicalnetworking.net/series/nat/nat/ 24 | - VLAN: https://www.practicalnetworking.net/stand-alone/vlans/ 25 | - Linux Virtual Networking 101: https://developers.redhat.com/blog/2018/10/22/introduction-to-linux-interfaces-for-virtual-networking/ 26 | - https://www.shapeblue.com/a-beginners-guide-to-cloudstack-networking/ 27 | 28 | Recommended learning: 29 | - CloudStack Networking series by Chiradeep: https://www.youtube.com/playlist?list=PLDF5F6BBB1C3A7CDD 30 | - Inside CloudStack VR by Rohit: https://docs.google.com/presentation/d/1fTfOaur4BNTStd_NCuwNEGGJ3E4xbf-aTFIP0nqGCiY/edit#slide=id.g38be540065_6_22 31 | - [Networking Refresh by Dag](primer/networking-refresh.pdf) 32 | 33 | Video Series: 34 | - CloudStack Networking Models - A Step-by-step Guide - Part 1 https://www.youtube.com/watch?v=XQwqqcPWDbE 35 | - CloudStack Networking Models - A Step-by-step Guide - Part 2 https://www.youtube.com/watch?v=hSfl2QfSJMg 36 | 37 | ### Sessions 38 | 39 | The following session structure (self-learning or with a colleague/mentor) can be used: 40 | 41 | Session 1: 42 | - Basic terms, terminologies 43 | - OSI layer, L1-3, L4-7 layers 44 | - Basic devices (switch, router, bridge, tap, tun, etc.) 45 | - Basic protocols and addressing (arp, dhcp, dns, tcp, udp, icmp, igmp, ipv4, ipv6, optional: ospf, bgp) 46 | - Isolation: vlan, vxlan, sg 47 | - Bridge networking 48 | - CloudStack network models basics 49 | - Practical demo using monkeybox 50 | 51 | Session 2: 52 | - Revisit layers 53 | - CloudStack network models adv. with diagrams for each type 54 | - Linux nf framework (netfilters) 55 | - Network tools (iproute2, iptables/ebtables/nftables, tcpdump, ping/arping, netstat, nslookup, arp, traceroute etc.) 56 | - Debugging network stack across machines 57 | 58 | Session 3: 59 | - CloudStack SystemVM building, patching, init 60 | - CloudStack agent framework, CPVM/SSVM use-cases 61 | 62 | Session 4: 63 | - Practical network 64 | - CloudStack VR programming part 65 | - Implement X: Wrap up, end to end demo and walkthrough 66 | - Debug, find/extend feature etc. 67 | 68 | Misc: 69 | - Extending CloudStack networking: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Extending+CloudStack+Networking 70 | - Understanding CloudStack network implementation: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=31823180 71 | - Adv network tutorial: https://cwiki.apache.org/confluence/display/CLOUDSTACK/CloudStack+Advanced+Network+Tutorial+-+Step+by+Step 72 | 73 | Implementation specific: 74 | https://cwiki.apache.org/confluence/display/CLOUDSTACK/Network+Manager+refactoring 75 | 76 | https://cwiki.apache.org/confluence/display/CLOUDSTACK/Refactoring+Redundant+Virtual+Router+Implementation 77 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # CloudStack HackerBook 2 | 3 | Hackerbook is a rapid learning framework for onboarding and training new 4 | CloudStack developers. This learn by doing yourself course is aimed at anybody 5 | who wants to learn how to develop a feature for [Apache 6 | CloudStack](http://cloudstack.apache.org/). The basic course can be completed in 7 | 4-5 weeks and overall it can be completed in about 6-8 weeks. 8 | 9 | In this course, each chapter has some short videos and suggested exercises which 10 | the new developer can work on to learn by doing them, hence the name 11 | `hackerbook`. The course starts with chapter 1 on general guideline on getting 12 | started, and then encourages the developer to learn CloudStack as a user 13 | in chapter 2 where they are asked to install, use and work with CloudStack 14 | using the API, UI and have `cmk` (CLI) and ansible-based automation exercises. 15 | Next, in chapter 3 the developer is asked to work on a fictious feature which 16 | helps them learn about various aspect of building parts of a feature such as 17 | creating an API, handling API via a service layer manager, DB handling, UI etc. 18 | Rest of the remaining chapters encourage self learning and exploration with 19 | recommended reading and references around advanced CloudStack topics. 20 | 21 | ShapeBlue started `hackerbook` course material internally in late 2018 to onboard 22 | and train new engineers to work on Apache CloudStack. After successfully 23 | onboarding and training a bunch of new engineers and improving hackerbook, 24 | ShapeBlue opensourced `hackerbook` for the Apache CloudStack community in 2021. 25 | 26 | ## Contents 27 | 28 | | Chapter | Topic | Est. Effort | 29 | | ------- | ----- | ----------- | 30 | | #1 | [Getting Started](0-init.md) | 10 hours | 31 | | #2 | [Test Drive CloudStack](1-user.md) | 40 hours | 32 | | #3 | [Basic CloudStack Development](2-dev.md) | 150 hours | 33 | | #4 | [Advanced CloudStack Development](3-adv.md) | 40 hours | 34 | | #5 | [Hypervisor and Storage](4-compute-storage.md) | 40 hours | 35 | | #6 | [Networking](5-network.md) | 40 hours | 36 | | | [Appendix: Primers](primer/index.md) | | 37 | | | | **320 hours** (6-8 weeks) | 38 | 39 | ## Bookmarks 40 | 41 | - [CloudStack Awesome List](https://github.com/resmo/awesome-cloudstack) 42 | - [CloudStack Docs](http://docs.cloudstack.apache.org/en/latest/) 43 | 44 | ## Contribution and Getting Help 45 | 46 | Raise a pull request to contribute changes to the course documentation. We may not 47 | be able to work on any reported issue and offer individual help to the reader. 48 | We encourage readers to join and ask questions on the Apache CloudStack dev/user 49 | mailing lists: http://cloudstack.apache.org/mailing-lists.html 50 | 51 | ## Changelog 52 | 53 | - 18 Feb 2025 - updated for Ubuntu 24.04 & MacOS as dev platforms 54 | - 1 Jan 2023 - updated for Ubuntu 22.04 as dev platform 55 | - 26 Feb 2021 - hackerbook opensourced 56 | - 22 Feb 2021 - repository updated against Ubuntu 20.04 as dev platform 57 | - 15 Oct 2018 - hackerbook started by Rohit Yadav to train new engineers at ShapeBlue 58 | 59 | ## License 60 | 61 | Creative Commons License
This work 64 | is licensed under a Creative Commons 66 | Attribution-ShareAlike 4.0 International License. 67 | -------------------------------------------------------------------------------- /hack/api.md: -------------------------------------------------------------------------------- 1 | # CloudStack API Development 2 | 3 | ## Project Skeleton 4 | 5 | For the purpose of the exercise, you may implement the feature as a separate 6 | maven project under `plugins`: 7 | 8 | mkdir -p plugins/hackerbook/feature 9 | 10 | To the `feature` directory, add a maven project `pom.xml` file: 11 | 12 | ```java 13 | 16 | 4.0.0 17 | cloud-plugin-hackerbook-feature 18 | Apache CloudStack Plugin - HackerBook Coffee Feature 19 | 20 | org.apache.cloudstack 21 | cloudstack-plugins 22 | 4.15.1.0-SNAPSHOT 23 | ../../pom.xml 24 | 25 | 26 | 27 | org.apache.cloudstack 28 | cloud-api 29 | ${project.version} 30 | 31 | 32 | org.apache.cloudstack 33 | cloud-utils 34 | ${project.version} 35 | 36 | 37 | 38 | ``` 39 | 40 | Note: please change the version suitably as per the base-branch you're using. 41 | 42 | The pom.xml has `cloud-api` and `cloud-utils` artifacts as dependencies to 43 | use/extend/implementing API interfaces and use any CloudStack utility classes. 44 | 45 | Note: CloudStack master branch's `version` may change, please fix accordingly. 46 | 47 | Add the project to CloudStack's plugin pom.xml to get your newly added maven 48 | project (feature) built along with other CloudStack artifacts: 49 | 50 | ```java 51 | --- a/plugins/pom.xml 52 | +++ b/plugins/pom.xml 53 | @@ -73,6 +73,8 @@ 54 | event-bus/rabbitmq 55 | 56 | + hackerbook/feature 57 | 58 | ``` 59 | 60 | Also, add the `feature` maven project to CloudStack's `client/pom.xml` which 61 | builds and bundles various CloudStack artifacts (jars) into a single uberjar: 62 | 63 | ```java 64 | --- a/client/pom.xml 65 | +++ b/client/pom.xml 66 | @@ -218,6 +218,11 @@ 67 | cloud-plugin-network-vsp 68 | ${project.version} 69 | 70 | + 71 | + org.apache.cloudstack 72 | + cloud-plugin-hackerbook-feature 73 | + ${project.version} 74 | + 75 | ``` 76 | 77 | Now, when you build and run CloudStack your `feature` will be part of the 78 | management server. 79 | 80 | Next, per the standard maven convention you'll need to add directories/code per 81 | the following hierarchy: 82 | 83 | ```bash 84 | feature 85 |    ├── pom.xml # maven project config 86 |    ├── target # build directory 87 |    └── src 88 |    └── main 89 |    └── java 90 |    └── org.apache.cloudstack 91 |    └── api 92 |     └── command # for API command classes 93 |     └── response # for API response classes 94 |    └── feature # for feature classes 95 |    └── dao # for feature DAO classes 96 |    └── resources 97 |    └── resources/META-INF/cloudstack/feature 98 |    └── module.properties 99 |    └── spring-feature-context.xml 100 |    └── test 101 |    └── java 102 |    └── org.apache.cloudstack.feature # for feature unit tests 103 | ``` 104 | 105 | The spring module skeleton and setup will be discussed in the next exercise. 106 | 107 | `License notice`: all files contributed to Apache CloudStack should have the 108 | Apache License 2.0 header. See existing files for reference and example. 109 | 110 | ## Introduction 111 | 112 | CloudStack has two API service ports on the default API path 113 | `host:/client/api`: 114 | - 8080 (default): the authenticated API service port. 115 | - 8096: the unauthenticated API service port as defined by the 116 | `integration.api.port` global setting. It is disabled by default on production 117 | installations. 118 | 119 | CloudStack has two types of (REST-like, query-based end-user/admin) APIs: 120 | 121 | - Synchronous: 122 | - Extends `BaseCmd` class or a child class. 123 | - Blocking execution of API until an API response is returned. 124 | - Asynchronous: 125 | - Extends `BaseAsyncCmd` or `BaseAsyncCreateCmd` class or a child class. 126 | - Creates an async job and returns a job ID. The API response can be checked 127 | by pollable the `queryAsyncJobResult` API providing it the `jobid`. 128 | 129 | A CloudStack API is a class that encapsulates an API request parameters, a 130 | common pattern in CloudStack is to pass an API object to business/service layer. 131 | 132 | **Suggested API examples**: 133 | https://github.com/apache/cloudstack/tree/master/api/src/main/java/org/apache/cloudstack/api/command/admin/acl 134 | 135 | References: 136 | - http://cloudstack.apache.org/api.html 137 | - https://cwiki.apache.org/confluence/display/CLOUDSTACK/CloudStack+API+Development 138 | - https://cwiki.apache.org/confluence/display/CLOUDSTACK/Annotations+use+in+the+API 139 | - https://cwiki.apache.org/confluence/display/CLOUDSTACK/CloudStack+API+Coding+Guidelines 140 | - https://cwiki.apache.org/confluence/display/CLOUDSTACK/How+To+Generate+CloudStack+API+Documentation 141 | - https://cwiki.apache.org/confluence/display/CLOUDSTACK/List*+API+commands+rules 142 | - https://cwiki.apache.org/confluence/display/CLOUDSTACK/CloudStack+IAM+guidelines+for+API+and+Service+Layer 143 | - https://cwiki.apache.org/confluence/display/CLOUDSTACK/Coding+conventions 144 | 145 | ## API implementation 146 | 147 | Depending on the type of API you want to implement, create an API class 148 | with its name same or similar to the API name. For example, for the API 149 | `createCoffee` you may create a CreateCoffeeCmd.java, and for `listCoffees` 150 | ListCoffeesCmd.java etc. 151 | 152 | Every API is based on two classes (sometimes reusable): a request class and a 153 | response class. 154 | 155 | Each API class needs to have an `APICommand` annotation on the class that is 156 | used to export metadata about the API such as the `name`, `description` etc. 157 | Each API class needs to also declare an API response class which is a class that 158 | captures a response of the API. The `since` can have information about 159 | CloudStack version in which the API was introduced. You may also declare API 160 | security doc parameters `requestHasSensitiveInfo` and 161 | `responseHasSensitiveInfo`. Finally, the `authorized` parameter controls which 162 | type of user account may be allowed to execute the API. 163 | 164 | Example API code: 165 | 166 | ```java 167 | @APICommand(name = "myAPI", 168 | description = "My API short description here", 169 | responseObject = MyAPIResponse.class, 170 | since = "4.xx.yy", 171 | requestHasSensitiveInfo = false, responseHasSensitiveInfo = false, 172 | authorized = {RoleType.Admin, RoleType.ResourceAdmin, RoleType.DomainAdmin, RoleType.User}) 173 | public class MyAPICmd extends BaseCmd { 174 | public static final Logger LOG = Logger.getLogger(MyAPICmd.class); 175 | 176 | ``` 177 | 178 | Next, API can have parameters that can be defined using the `Parameter` 179 | annotation that can defined several attributes of an API parameter such as the 180 | parameter `name`, `description`, `required` etc. Please explore the `Parameter` 181 | interface for full list of attributes. 182 | 183 | Example parameter code: 184 | 185 | ```java 186 | ///////////////////////////////////////////////////// 187 | //////////////// API parameters ///////////////////// 188 | ///////////////////////////////////////////////////// 189 | 190 | @Parameter(name = ApiConstants.ID, 191 | type = CommandType.UUID, 192 | required = false, 193 | entityType = MyAPIResponse.class, 194 | description = "ID of my resource") 195 | private Long id; 196 | ``` 197 | 198 | `CommandType` defines the type of the API parameter. The API layer uses this 199 | annotation and declared metadata to validate an API request, for example when 200 | the `type` is BOOLEAN it may try to convert the argument input to a `boolean` 201 | value etc. The following API command types are supported per the 202 | `BaseCmd::CommandType` enum: 203 | 204 | ```java 205 | CommandType { 206 | BOOLEAN, DATE, FLOAT, DOUBLE, INTEGER, SHORT, LIST, LONG, OBJECT, MAP, STRING, TZDATE, UUID 207 | } 208 | ``` 209 | 210 | Just like an API, an API `Parameter` may also declare its own `authorized` 211 | field. The API parameters can also define `validations` to use one of the 212 | commonly used API validators, for example: 213 | 214 | ```java 215 | validations = {ApiArgValidator.NotNullOrEmpty} 216 | validations = {ApiArgValidator.PositiveNumber} 217 | ``` 218 | 219 | Next, the API can define accessors (getters usually) for the parameters. For 220 | example: 221 | 222 | ```java 223 | ///////////////////////////////////////////////////// 224 | /////////////////// Accessors /////////////////////// 225 | ///////////////////////////////////////////////////// 226 | 227 | public Long getId() { 228 | return id; 229 | } 230 | ``` 231 | 232 | The API implementation is a group of methods that exports 233 | the account ID of the resource owner on which the API is 234 | acted up (`getEntityOwnerId`) and the `execute()` method that handles the API 235 | request. The `getEntityOwnerId()` method can also make use of `CallContext` 236 | utility to get information about the current thread/execution context. For 237 | example, get the account ID that made the API request by using 238 | `CallContext.current().getCallingAccountId()`. 239 | Method to export the API name (`getCommandName`) is not needed now in 4.18 branch or later, after changes introduced in https://github.com/apache/cloudstack/pull/7022. 240 | 241 | Reference reading: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Using+CallContext 242 | 243 | Example API implementation: 244 | 245 | ```java 246 | ///////////////////////////////////////////////////// 247 | /////////////// API Implementation/////////////////// 248 | ///////////////////////////////////////////////////// 249 | 250 | @Override 251 | public long getEntityOwnerId() { 252 | return Account.ACCOUNT_ID_SYSTEM; 253 | } 254 | 255 | @Override 256 | public void execute() { 257 | // logic to handle API request 258 | 259 | final MyAPIResponse response = new MyAPIResponse(); 260 | // logic to setup the API response object 261 | response.setResponseName(getCommandName()); 262 | response.setObjectName("object-name"); 263 | setResponseObject(response); 264 | } 265 | ``` 266 | 267 | ### Asynchronous APIs 268 | 269 | Asynchronous APIs in CloudStack also need to export the event type (also see 270 | `EventTypes` class) and description information. Such APIs generally have three 271 | phases: API request is received, API execution is asynchronously started by the 272 | job framework, and the API execution finishes and the response is saved. They 273 | typically need to implement these methods: 274 | 275 | ```java 276 | 277 | @Override 278 | public String getEventType() { 279 | return EventTypes.EVENT_XYZ; 280 | } 281 | 282 | @Override 283 | public String getEventDescription() { 284 | return "string description usually with action details and entity ids"; 285 | } 286 | ``` 287 | 288 | An asynchronous API extending the `BaseAsyncCreateCmd` or child class usually 289 | also implements a `create()` method which is run before the `execute()` and 290 | allows creation of an resource entity (usually in the database) before the API 291 | executes. Have a look at the `BaseAsyncCmd` and `BaseAsyncCreateCmd` classes 292 | for overridable methods. 293 | 294 | ```java 295 | @Override 296 | public void create() { 297 | Resources res = someService.createResource(this); 298 | if (res != null) { 299 | this.setEntityId(res.getId()); 300 | this.setEntityUuid(res.getUuid()); 301 | } 302 | ... 303 | ``` 304 | 305 | CloudStack async APIs can also export events that usually describe the state 306 | of execution (Created, Scheduled, Started, Completed). For this, the async API 307 | implementation's `getEventType` and `getEventDescription` are used by the event 308 | framework to publish these events on the event bus. The handler method defined 309 | in the service/manager class can also define an annotation `@ActionEvent` to 310 | export event type and description metadata that gets captured by the 311 | `ActionEventInterceptor` class by means of `spring-aop`. 312 | 313 | ```java 314 | @Override 315 | @ActionEvent(eventType = EventTypes.EVENT_COFFEE_CREATE, eventDescription = "creating coffee", async = true) 316 | public Coffee createCoffee(CreateCoffeeCmd createCoffeeCmd) { 317 | // Logic to create coffee 318 | ``` 319 | 320 | Reference reading: 321 | - http://docs.cloudstack.apache.org/en/4.11.2.0/adminguide/events.html 322 | - https://docs.spring.io/spring/docs/current/spring-framework-reference/core.html#aop 323 | 324 | ## API response implementation 325 | 326 | An API response class typically extends `BaseResponse` and contains response 327 | attributes with `@Param` and `@SerializedName` annotations that define the 328 | serialized attribute/key name and their description. This metadata is used 329 | by CloudStack build system to generate `apidocs` (see in tools/apidocs). The 330 | class may sometimes have a `@EntityReference` annotation to mark the type of 331 | resource the response class represents. Use this annotation when you've a VO 332 | class in future exercise, that implements the resource interface. 333 | 334 | An API response class may typically be reused by a resource's list/create/update 335 | APIs and generally contains setters (and sometimes getters). For example: 336 | 337 | ```java 338 | @EntityReference(value = MyResource.class) 339 | public class MyAPIResponse extends BaseResponse { 340 | @SerializedName(ApiConstants.ID) 341 | @Param(description = "the ID of my resource") 342 | private String id; 343 | 344 | public void setId(String id) { 345 | this.id = id; 346 | } 347 | ``` 348 | 349 | ### API UUID Translation 350 | 351 | Most CloudStack resources/objects have a unique uuid (string), and in 352 | the database they also have a `bigint` ID. The UUID command type allows APIs 353 | with both integer and uuid (string) IDs to be translated and validated to a 354 | CloudStack resource and set that resource's ID to the `Long` field. This 355 | translation is done with help of the `@Parameter` `entityType` field 356 | which generally is a `Response` class having an `@EntityReference` annotation 357 | that declares an interface class which typically implements a `VO` (view object) 358 | class declaring a `@Table`. This way, for each parameter the API layer can 359 | try to find the resource from a database table by the passed `uuid` value and 360 | perform the translation and validation. We'll revisit how API layer work in 361 | detail in future chapters. 362 | 363 | ### API Docs 364 | 365 | When adding a new set of APIs around a resource, the build around `apidocs` 366 | may fail. This is because API docs may not know how to categorize those new 367 | APIs. For this, add a the resource name (for example `Coffee`): 368 | 369 | ```python 370 | --- a/tools/apidoc/gen_toc.py 371 | +++ b/tools/apidoc/gen_toc.py 372 | @@ -190,7 +190,8 @@ known_categories = { 373 | 'Sioc' : 'Sioc', 374 | - 'Diagnostics': 'Diagnostics' 375 | + 'Diagnostics': 'Diagnostics', 376 | + 'Coffee': 'Coffee' 377 | } 378 | ``` 379 | 380 | When you build CloudStack, API docs are generated at 381 | `tools/apidoc/target/xmldoc/html`. 382 | 383 | ## Exercises 384 | 385 | 1. Implement the following APIs based on the spec, under 386 | `org.apache.cloudstack.api.command` package: 387 | - createCoffee (extend BaseAsyncCreateCmd) 388 | - listCoffee (extend BaseListCmd) 389 | - updateCoffee (extend BaseAsyncCmd) 390 | - removeCoffee (extend BaseAsyncCmd, use SuccessResponse as response class) 391 | 392 | 2. Implement API response class `CoffeeResponse` that represents a Coffee 393 | resource under `org.apache.cloudstack.api.response` package. 394 | 395 | 3. Write basic unit tests for the classes. (IntelliJ: Ctrl+Shift+t to 396 | create/browse unit test of a java class) 397 | 398 | Challenge: Attempt and fix any upstream CloudStack API related issue(s): 399 | https://github.com/apache/cloudstack/labels/component%3Aapi 400 | -------------------------------------------------------------------------------- /hack/automation.md: -------------------------------------------------------------------------------- 1 | # CloudStack Automation 2 | 3 | CloudStack installation, setup, deployment and management can be primarily be 4 | performed via CloudMonkey, the official CLI and Ansible. 5 | 6 | ## CloudMonkey 7 | 8 | CloudMonkey is the official CLI for Apache CloudStack. 9 | 10 | Install the old Python based CloudMonkey, follow this guide: 11 | https://cwiki.apache.org/confluence/display/CLOUDSTACK/CloudStack+cloudmonkey+CLI 12 | 13 | Also, install and test the new Golang based CloudMonkey. Get the binary from: 14 | https://github.com/apache/cloudstack-cloudmonkey/releases 15 | 16 | Documentation for the modern CloudMonkey: 17 | https://github.com/apache/cloudstack-cloudmonkey/wiki 18 | 19 | API documentation: 20 | https://cloudstack.apache.org/api.html 21 | 22 | Things to hack and learn: 23 | - Understand how to configure CloudMonkey 24 | - How to create and use CloudMonkey server profiles 25 | - How to make API requests in interpreter and command line modes 26 | - How to get API response outputs in various display format 27 | - How to use sed/awk and/or jq to process API response 28 | 29 | Tools to use: 30 | - cloudmonkey (legacy) 31 | - cmk (modern) 32 | - bash and jq 33 | 34 | **Challenges**: 35 | - Easy/Medium: Write a bash script to deploy few VMs using CloudMonkey and using 36 | jq capture and print list of running VMs. The bash script should take this 37 | number as a command line argument, for example to deploy 5 VMs: 38 | 39 | bash deploy-script.sh 5 40 | 41 | - Hard: Write a bash script that can deploy an advanced CloudStack zone using 42 | CloudMonkey (hint: deploy a zone using the UI, before hitting the `Launch 43 | zone` button start tracing the API requests using Chrome/devtools) 44 | 45 | ## Ansible and CloudStack 46 | 47 | Ansible is a popular Python based configuration management tool. You can learn 48 | more about it here: https://www.ansible.com/resources/get-started 49 | 50 | Recommended reading: 51 | http://docs.cloudstack.apache.org/projects/archived-cloudstack-getting-started/en/latest/ansible.html 52 | https://docs.ansible.com/ansible/2.9/scenario_guides/guide_cloudstack.html 53 | 54 | Example projects using Ansible for CloudStack automation: 55 | - Trillian, ACS environment automation: https://github.com/shapeblue/trillian 56 | - All in a box setup example: https://github.com/shapeblue/cloudstack-ansible 57 | - KVM automation example: https://github.com/rhtyd/peppercorn 58 | 59 | Things to hack and learn: 60 | - How to use Ansible for any kind of automation 61 | - How to configure ansible and use the `cloudstack` ansible module 62 | 63 | Tools to use: 64 | - ansible 65 | - bash 66 | 67 | **Challenges**: 68 | - Deploy a VM in an isolated network or a VPC with two tiers 69 | 70 | # Terraform 71 | 72 | https://github.com/apache/cloudstack-terraform-provider 73 | 74 | # CAPC 75 | 76 | https://github.com/kubernetes-sigs/cluster-api-provider-cloudstack 77 | 78 | https://cluster-api-cloudstack.sigs.k8s.io/ 79 | 80 | # mbx 81 | 82 | https://github.com/shapeblue/mbx 83 | -------------------------------------------------------------------------------- /hack/com.md: -------------------------------------------------------------------------------- 1 | # Message Passing and Communication 2 | 3 | CloudStack IPC (inter-process communication) is implemented by the 4 | `cloud-framework-ipc` project. However, not all the building blocks and utilities 5 | in `cloud-framework-ipc` are complete or used in the codebase. 6 | 7 | Related reference: https://cwiki.apache.org/confluence/display/CLOUDSTACK/FS+-+VMSync+improvement 8 | 9 | ## Async Callbacks 10 | 11 | The `cloud-framework-rpc` provides building blocks to write (nested) 12 | asynchronous code by using callbacks and futures: 13 | 14 | ``` 15 | AsyncCallbackDispatcher 16 | AsyncCallFuture 17 | AsyncCompletionCallback 18 | AsyncRpcContext 19 | ``` 20 | 21 | These building blocks are useful for writing asynchronous code that need callbacks 22 | to execute a method (logic) from a different layer. Its usage is mostly seen in 23 | storage (volume, snapshot, template) related code in CloudStack. 24 | 25 | Here is an example on how these building blocks can be used to create an 26 | asynchronous coffee brewing mechanism that may be nested (passed around layers) 27 | and still work with help from callbacks and context. 28 | 29 | First define the context class that extends the `AsyncRpcContext` and used to 30 | hold objects: 31 | 32 | ```java 33 | private class CreateCoffeeContext extends AsyncRpcContext { 34 | private final CoffeeMachine coffeeMachine; 35 | 36 | public CreateCoffeeContext(AsyncCompletionCallback callback, CoffeeMachine coffeeMachine) { 37 | super(callback); 38 | this.coffeeMachine = coffeeMachine; 39 | } 40 | 41 | public CoffeeMachine getCoffeeMachine() { 42 | return this.coffeeMachine; 43 | } 44 | } 45 | ``` 46 | 47 | Next, define define an asynchronous callback method that must be called when 48 | the dispatcher (caller) finishes its job and define the async method where the 49 | dispatcher can be called to complete when the task is complete. As an example, 50 | we can the async-callback method to brew coffee using the configured 51 | `CoffeeMachine` plugin after the coffee state transitions to brewing: 52 | 53 | ```java 54 | // Note: callback method should be `protected Void` for `Enhancer` to work 55 | protected Void brewCoffeeAsyncCallback(AsyncCallbackDispatcher callback, CreateCoffeeContext context) { 56 | Coffee coffee = callback.getResult(); 57 | context.getCoffeeMachine().brew(coffee); 58 | stateTransitTo(coffee, Coffee.Event.OrderReady); 59 | return null; 60 | } 61 | 62 | public Coffee brewCoffeeAsync(Coffee coffee, AsyncCompletionCallback callback) { 63 | stateTransitTo(coffee, Coffee.Event.OrderReceived); 64 | callback.complete(coffee); 65 | return coffee; 66 | } 67 | 68 | ``` 69 | 70 | The `createCoffee` can next be made to create the async dispatcher (caller) 71 | instance and configure the async callback method to it, before calling the 72 | async-create method: 73 | 74 | ```java 75 | @Override 76 | @ActionEvent(eventType = EventTypes.EVENT_COFFEE_CREATE, eventDescription = "creating coffee", async = true) 77 | public Coffee createCoffee(CreateCoffeeCmd cmd) { 78 | // Validations, checks, example code to save Coffee in DB: 79 | final CoffeeVO coffee = coffeeDao.persist(new CoffeeVO(cmd.getName(), CallContext.current().getCallingAccountId())); 80 | // Create coffee context object and save any objects that may be useful 81 | // for the (nested) layers 82 | CreateCoffeeContext context = new CreateCoffeeContext<>(null, getCoffeeMachine()); 83 | // Create an async call dispatcher that can call a callback/handler once an async job completes 84 | AsyncCallbackDispatcher caller = AsyncCallbackDispatcher.create(this); 85 | // The getTarget() enhances the class instance (this) 86 | // The callback handler method when evoked is intercepted and saved 87 | caller.setCallback(caller.getTarget().BrewCoffeeAsyncCallback(null, null)); 88 | caller.setContext(context); 89 | // Call the async method 90 | return brewCoffeeAsync(coffee, caller); 91 | } 92 | ``` 93 | 94 | Tip: use these building blocks when you need to have actions performed by an 95 | upper layer where the caller may be far away from the callee. 96 | 97 | ## Message Bus 98 | 99 | CloudStack messagebus can be used for message/event driven feature 100 | implementation where a subscriber can react to published events/topics. 101 | 102 | To use it you can inject `MessageBus` in your class: 103 | 104 | ```java 105 | @Inject 106 | private MessageBus messageBus; 107 | ``` 108 | 109 | You can publish, subscribe, unsubscribe on the message bus for a topic (usually 110 | a constant string): 111 | 112 | ``` 113 | // Publish example 114 | messageBus.publish(sender, MESSAGE_RESOURCE_CRUD_EVENT, PublishScope.LOCAL, resouceVO); 115 | 116 | // Subscribe example 117 | messageBus.subscribe(SomeResourceManager.MESSAGE_RESOURCE_CRUD_EVENT, new MessageSubscriber() { 118 | @Override 119 | public void onPublishMessage(String senderAddress, String subject, Object args) { 120 | try { 121 | // Handle message 122 | } catch (final Exception e) { 123 | LOG.error("Caught exception while handling xyz: ", e); 124 | } 125 | } 126 | }); 127 | ``` 128 | 129 | Note: `PublishScope.GLOBAL` is not currently implemented to publish between 130 | multiple-management servers. 131 | 132 | ## Events and Alerts 133 | 134 | CloudStack event framework is implemented by the `cloud-framework-events` 135 | project which allows exporting of events to external queues such as RabbitMQ and 136 | Kafka. 137 | 138 | Reference: http://docs.cloudstack.apache.org/en/4.11.2.0/adminguide/events.html 139 | 140 | Events are generated by async API as well as using `ActionEventUtils`, for 141 | example: 142 | 143 | ```java 144 | ActionEventUtils.onActionEvent(userId, accountId, domainId, EventTypes.EVENT_SOME_ACTION, description); 145 | ``` 146 | 147 | The `AlertManager` can be used to send alerts which will email the admin, as well 148 | as create events. For example: 149 | 150 | ```java 151 | @Inject 152 | private AlertManager alertManager; 153 | 154 | // Example usage: 155 | alertManager.sendAlert(AlertManager.AlertType.ALERT_TYPE_XYZ, zoneId, podId, subject, message); 156 | ``` 157 | 158 | ## Agent Framework based RPC 159 | 160 | CloudStack uses the `command` design pattern to send commands to a 161 | `ServerResource` resource (directly via shared memory, or indirectly via 162 | network) and these commands are handled by `executeRequest` and an answer is 163 | returned back. 164 | 165 | Tip: CPVM/SSVM/KVM agents work as indirect or connected agents. 166 | 167 | The `cloud-engine-orchestration` implements `AgentManagerImpl` that manages 168 | agents by means of `AgentAttache`. 169 | 170 | CloudStack management server supports two kinds of agents: 171 | - Direct agent: Uses `DirectAgentAttache`, commands are handled by the same JVM 172 | which runs the management server. 173 | - Indirect/Connect agent: Uses `ConnectedAgentAttache`, agents connect to the 174 | management server on its service port `8250` and commands are sent to remote 175 | agent via a custom RPC and custom serialization/deserialization mechanism. For 176 | connection and communication it uses `NioServer`, `NioClient`, 177 | `NioConnection`, `Link` as building blocks secured by the `cloud-ca-framework` 178 | and sends commands wrapped in `Request` by serializing commands to json, 179 | gzipping it and for answers the process is reversed. The serializing and 180 | deserializing logic is implemented in `Request` class. 181 | 182 | The CloudStack `cloud-agent` implements `Agent` and `AgentShell` classes that 183 | implement a shell layer between a `ServerResource` and the managment server. The 184 | `AgentShell` handles the agent/shell process and connection, while the `Agent` 185 | class facilitates RPC and passing of commands/answers to/from the 186 | `ServerResource`. 187 | 188 | Some popular `ServerResource` support resources are: KVM 189 | (LibvirtComputingResource), VMWare (VmwareResource), XenServer 190 | (CitrixResourceBase), SSVM (NfsSecondaryStorageResource), and Simulator 191 | (AgentResourceBase). 192 | 193 | To use agent based RPC, first define your `Command` class. For example: 194 | 195 | ```java 196 | public class CoffeeBrewCommand extends Command { 197 | private Coffee coffee; 198 | 199 | public CoffeeBrewCommand(final Coffee coffee) { 200 | this.coffee = coffee; 201 | } 202 | 203 | @Override 204 | public boolean executeInSequence() { 205 | return false; 206 | } 207 | } 208 | ``` 209 | 210 | Next, you can send a command instance using the `AgentManager`. 211 | 212 | ```java 213 | @Inject 214 | private AgentManager agentManager; 215 | 216 | // Example code 217 | CoffeeBrewCommand command = new CoffeeBrewCommand(coffee); 218 | Answer answer = null; 219 | try { 220 | answer = agentManager.send(hostId, command); 221 | } catch (AgentUnavailableException e) { 222 | } catch (OperationTimedoutException e) { 223 | } 224 | // process answer 225 | ``` 226 | 227 | The command can be handled by writing a handler method or wrapper class that 228 | handles the command for a `ServerResource`. For example, in case of KVM: 229 | 230 | ```java 231 | package com.cloud.hypervisor.kvm.resource.wrapper; 232 | 233 | @ResourceWrapper(handles = CoffeeBrewCommand.class) 234 | public final class CoffeeCommandWrapper extends CommandWrapper { 235 | @Override 236 | public Answer execute(final CoffeeBrewCommand command, final LibvirtComputingResource libvirtComputingResource) { 237 | // handle brew-ops 238 | return new Answer(command); 239 | } 240 | } 241 | ``` 242 | 243 | Note: Scp/copy the agent/kvm jars to the KVM host(s). 244 | 245 | ## Exercises 246 | 247 | 1. Implement the create coffee method with `async` callbacks. 248 | 249 | 2. Write messagebus handler to create/remove coffee when account is 250 | create/removed. Send alerts to the admin when a coffee is discarded by the GC 251 | background task. 252 | 253 | 3. Refactor one of the `CoffeeMachine` plugins to brew coffee remotely on a 254 | `ServerResource` using command-answer pattern. You can use KVM or simulator. 255 | -------------------------------------------------------------------------------- /hack/db.md: -------------------------------------------------------------------------------- 1 | # CloudStack DB 2 | 3 | ## Upgrade Paths 4 | 5 | CloudStack has its own database framework that is based on 6 | [DAO](https://www.oracle.com/technetwork/java/dataaccessobject-138824.html). 7 | 8 | CloudStack uses MySQL as the database, any schema for any feature maybe define 9 | in SQL. Before you start understanding and defining the DAO framework and its 10 | usage in CloudStack, understand how CloudStack schema is defined and upgraded. 11 | Use of terminal based `mysql-client` is recommended, you may also use [MySQL 12 | workbench](https://dev.mysql.com/downloads/workbench/). 13 | 14 | References: 15 | - https://www.ibm.com/developerworks/java/library/j-genericdao/index.html 16 | - https://www.w3schools.com/sql/ 17 | - https://www.tutorialspoint.com/sql/ 18 | - http://www.mysqltutorial.org/ 19 | - https://cwiki.apache.org/confluence/display/CLOUDSTACK/DB+Upgrade+in+CloudStack 20 | 21 | CloudStack has following databases: 22 | - `cloud`: the main database where most of CloudStack tables are 23 | - `cloud_usage`: the database used by the usage server for usage record generation 24 | - `simulator`: the database used by simulator plugin (only for developers, not 25 | for production usage) 26 | 27 | The `DatabaseUpgradeChecker` class is responsible for CloudStack schema upgrade. 28 | When the management server starts, an instance of this class kicks in to 29 | check the version of the schema based on the `cloud.version` table against the 30 | code version (from the jar). This class defines a map/sequence/hierarchy of 31 | upgrade paths from a starting version number. The upgrade path is a class 32 | that implements the `DbUpgrade` interface. For example, if the code version 33 | (as defined in the root pom.xml file) is `4.12.0.0-SNAPSHOT` look for an upgrade 34 | path class that may be named as `Upgrade4XXXXto41200`. This class would define 35 | two sql scripts, one that upgrades CloudStack's schema and other that runs 36 | to perform any schema cleanup. This class also defines metadata about the 37 | upgrade path, the from/to version ranges etc. You may use any of the existing 38 | upgrade paths to learn how to write one as well. 39 | 40 | ## Defining Schema 41 | 42 | Pick the upgrade path for which you intend your feature for (i.e. the target 43 | CloudStack version), and add relevant schema to that sql upgrade path file 44 | (for example, `META-INF/db/schema-41120to41200.sql`). For example: 45 | 46 | ```sql 47 | CREATE TABLE IF NOT EXISTS `cloud`.`coffee` ( 48 | `id` bigint(20) unsigned NOT NULL AUTO_INCREMENT, 49 | `uuid` varchar(40) UNIQUE, 50 | `name` varchar(255) NOT NULL, 51 | `state` varchar(40) NOT NULL, 52 | `account_id` bigint unsigned NOT NULL, 53 | `created` datetime NOT NULL COMMENT 'date of creation', 54 | `removed` datetime COMMENT 'date of removal', 55 | PRIMARY KEY (`id`), 56 | KEY (`uuid`), 57 | KEY `i_coffee` (`name`, `account_id`, `created`) 58 | ) ENGINE=InnoDB DEFAULT CHARSET=utf8; 59 | ``` 60 | 61 | Tips: 62 | - Avoid using plural form for the table name (for example, `cloud.user` not 63 | `cloud.users`) 64 | - Don't create schema keys that you won't need 65 | - For performance gains use `views` instead of simply the `tables`, the general 66 | use case could be to speed up list API execution 67 | 68 | ## Resource DAO 69 | 70 | **Recommended reading**: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Data+Access+Layer 71 | 72 | CloudStack data access layer is implemented by `GenericDaoBase` abstract class 73 | that implements the DAO. For feature/resource specific tables, you would usually 74 | implement a `Dao` interface that defines the contract for the `DaoImpl`, the 75 | implementation class would extend `GenericDaoBase`. 76 | 77 | The VO (view object) captures the schema/table and a VO instance typically 78 | represents a row in the table. The VO typically implements the resource 79 | interface (contract), however for purpose of any subsystem consuming a resource 80 | object, the resource interface should be used/passed around than a VO instance. 81 | 82 | The VO class exports and uses several annotations for its table/db fields and 83 | `@Table` to define the name of the table that the VO represents. Most CloudStack 84 | tables have an internal (bigint) ID or database `id`, but the resources are 85 | queried by users based on an external string based `uuid`. 86 | 87 | Define the `VO` and make it implement the feature/resource interface. For 88 | example: 89 | 90 | ```java 91 | @Entity 92 | @Table(name = "coffee") 93 | public class CoffeeVO implements Coffee { 94 | @Id 95 | @GeneratedValue(strategy = GenerationType.IDENTITY) 96 | @Column(name = "id") 97 | private long id; 98 | 99 | @Column(name = "uuid") 100 | private String uuid; 101 | 102 | @Column(name = "name") 103 | private String name; 104 | 105 | @Column(name = "state", nullable = false) 106 | @Enumerated(value = EnumType.STRING) 107 | private CoffeeState state = CoffeeState.Created; 108 | 109 | @Column(name = "account_id") 110 | private long accountId; 111 | 112 | @Column(name = GenericDao.CREATED_COLUMN) 113 | protected Date created; 114 | 115 | @Column(name = GenericDao.REMOVED_COLUMN) 116 | protected Date removed; 117 | 118 | // This empty constructor is needed for reflection to work 119 | public CoffeeVO() { 120 | uuid = UUID.randomUUID().toString(); 121 | } 122 | 123 | // Custom constructor example 124 | public CoffeeVO(String name, long accountId) { 125 | this.uuid = UUID.randomUUID().toString(); 126 | this.name = name; 127 | this.accountId = accountId; 128 | } 129 | 130 | // more custom constructors, getters, setters etc. 131 | ``` 132 | 133 | Define the `Dao` interface, for example: 134 | 135 | ```java 136 | package org.apache.cloudstack.feature.dao; 137 | 138 | public interface CoffeeDao extends GenericDao { 139 | // method definitions here 140 | } 141 | ``` 142 | 143 | Define the `DaoImpl`, for example: 144 | 145 | ```java 146 | package org.apache.cloudstack.feature.dao; 147 | 148 | public class CoffeeDaoImpl extends GenericDaoBase implements CoffeeDao { 149 | // method implementations here 150 | } 151 | ``` 152 | 153 | Declare the `DaoImpl` in the spring context xml file so that a bean gets created 154 | and can be injected in the service layer class. For example: 155 | 156 | ```xml 157 | --- a/plugins/hackerbook/feature/src/main/resources/META-INF/cloudstack/feature/spring-feature-context.xml 158 | +++ b/plugins/hackerbook/feature/src/main/resources/META-INF/cloudstack/feature/spring-feature-context.xml 159 | @@ -2,6 +2,9 @@ 160 | xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 161 | xsi:schemaLocation="http://www.springframework.org/schema/beans 162 | http://www.springframework.org/schema/beans/spring-beans.xsd"> 163 | 164 | + 165 | ``` 166 | 167 | Now, the `DaoImpl` class can be injected and used by the service/manager class. 168 | For example: 169 | 170 | ```java 171 | public class CoffeeManagerImpl extends ManagerBase implements CoffeeManager, Configurable, PluggableService { 172 | // .. code redacted .. 173 | @Inject 174 | private CoffeeDao coffeeDao; 175 | // .. code redacted .. 176 | @Override 177 | public List listCoffees(ListCoffeesCmd cmd) { 178 | // Perform validations checks etc. following is just an example 179 | return new ArrayList<>(coffeeDao.listAll()); 180 | } 181 | // .. code redacted .. 182 | @Override 183 | @ActionEvent(eventType = EventTypes.EVENT_COFFEE_CREATE, eventDescription = "creating coffee", async = true) 184 | public Coffee createCoffee(CreateCoffeeCmd cmd) { 185 | // Perform validations checks etc. following is just an example 186 | return coffeeDao.persist(new CoffeeVO(cmd.getName(), CallContext.current().getCallingAccountId())); 187 | } 188 | ``` 189 | 190 | By default, the Dao will have several ready to use methods such as `listAll`, 191 | `findById`, `update`, `persist`, `remove` etc. When persisting a new VO, your 192 | code does not need to create/set the `id`, `created` which are handled by the 193 | DAO framework (GenericDaoBase). 194 | 195 | Read the DAO wiki article to know various building blocks and utilities you can 196 | use to create custom methods for searching and querying (for example, the 197 | `SearchBuilder`). 198 | 199 | ### NOTE : 200 | > The ORM uses interceptors to track which fields of a VO have changed to construct the SQL query. 201 | It determines the field / column to be searched or updated from the getter or setter used. 202 | Therefore it is important to name the getters and setters properly. 203 | eg: A variable "abc" should have the appropriate getter "getAbc()" and setter "setAbc()". 204 | It is safer to allow the IDE to create the getters / setters. Be especially careful with boolean variables, 205 | which need to follow the same get / set prefix for thier getters and setters rather than is / enable / disable. 206 | 207 | ## DB Transactions 208 | 209 | **Recommended reading**: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Database+Transactions 210 | 211 | You can wrap a set of complex DB operations (for example, deleting of details 212 | when coffee is deleted/removed) in a DB transaction using the `Transaction` 213 | utility. For example: 214 | 215 | ```java 216 | return Transaction.execute(new TransactionCallback() { 217 | @Override 218 | public CoffeeVO doInTransaction(TransactionStatus status) { 219 | return coffeeDao.persist(new CoffeeVO(name, accountId)); 220 | } 221 | }); 222 | ``` 223 | 224 | ## Resource Details Dao 225 | 226 | For any resource, a details table can be designed as well. The VOs and Daos 227 | can be implemented in the same way. Typically a `_details` table will have 228 | following columns: 229 | - `id`: the auto increment internal DB ID (bigint) 230 | - `resource_id`: the (bigint) ID of the resource 231 | - `name`: the name/key of the detail (string) 232 | - `value`: the value of the detail (string) 233 | - `display`: boolean (int:1) to show/hide that detail 234 | 235 | The `VO` class for the resource detail table can implement `ResourceDetail` 236 | interface. The `Dao` interface can extend the `ResourceDetailsDao` interface 237 | and the `DaoImpl` class can extend the `ResourceDetailsDaoBase` abstract 238 | class. Using this pattern, your resource's `DetailsDao` can be used to 239 | list/add/remove/save details. 240 | 241 | ## FSM 242 | 243 | A Finite State Machine or 244 | [FSM](https://en.wikipedia.org/wiki/Finite-state_machine) defines a transition 245 | table that takes in a state and an event to transition to a new state. In 246 | CloudStack, FSM is used to implement state machine and restrict how state of a 247 | resource such as a VM, volume, network etc. transition given an event occurs. 248 | The resource state and events are both defined as `enum`, usually in the 249 | resource interface. The resource interface need to extend the `StateObject` 250 | interface. For example: 251 | 252 | ```java 253 | public interface Coffee extends InternalIdentity, Identity, StateObject { 254 | // .. code redacted .. 255 | enum Event { 256 | OrderReceived, 257 | OrderReady, 258 | OrderDiscarded 259 | } 260 | 261 | enum CoffeeState { 262 | Created, Brewing, Brewed, Discarded; 263 | 264 | private final static StateMachine2 FSM = new StateMachine2<>(); 265 | static { 266 | FSM.addTransition(Created, Event.OrderReceived, Brewing); 267 | FSM.addTransition(Brewing, Event.OrderReady, Brewed); 268 | FSM.addTransitionFromStates(Event.OrderDiscarded, Discarded, Created, Brewing, Brewed); 269 | } 270 | 271 | public static StateMachine2 getStateMachine() { 272 | return FSM; 273 | } 274 | } 275 | 276 | // .. code redacted .. 277 | CoffeeState getState(); 278 | ``` 279 | 280 | The resource specific `Dao` class needs to handle state transitions, Daos of 281 | resources that are `StateObject` can extend/implement `StateDao`. 282 | 283 | ```java 284 | public interface CoffeeDao extends GenericDao, StateDao { 285 | // .. code redacted .. 286 | ``` 287 | 288 | The `DaoImpl` can implement `updateState`. For example: 289 | 290 | ```java 291 | public class CoffeeDaoImpl extends GenericDaoBase implements CoffeeDao { 292 | 293 | @Override 294 | public boolean updateState(Coffee.CoffeeState currentState, Coffee.Event event, Coffee.CoffeeState nextState, Coffee vo, Object data) { 295 | // Update logic/check here 296 | // May use an update_counter from the vo for lock-free update 297 | vo.setState(nextState); 298 | return update(vo.getId(), (CoffeeVO) vo); 299 | } 300 | ``` 301 | 302 | Tip: due to several threads of execution and multiple management server, it may 303 | be possible that for the same resource object (VO instance) its `state` may get 304 | updated. For this purpose, in several CloudStack VOs an `update_counter` may be 305 | defined that ties up with the updates of its `State`. This provides a lock-free 306 | mechanism of updating a resource state, where the logic is simply enforced using 307 | a `SearchBuilder` that updates a `VO` object based on its previous update 308 | counter value and its previous state. For example, see `VolumeVO` and 309 | `VolumeDaoImpl`. 310 | 311 | With the FSM implemented and DAO made state-aware, the FSM can be then used to 312 | transit to a state based on an event which can automatically handle any DB 313 | updates. This can be done by having a utility method that uses FSM's `transitTo` 314 | method. For example, the following method in the service/manager impl class: 315 | 316 | ```java 317 | private boolean stateTransitTo(Coffee coffee, Coffee.Event event) throws NoTransitionException { 318 | return Coffee.CoffeeState.getStateMachine().transitTo(coffee, event, null, coffeeDao); 319 | } 320 | ``` 321 | 322 | You can also add a listener of state changes to a CloudStack FSM using 323 | `registerListener()`, the listener could be any class that implements the 324 | `StateListener` interface. 325 | 326 | ## Exercises 327 | 328 | 1. Implement the VO, Dao and DaoImpl classes for your feature, for each of the 329 | schema/tables `coffee` and `coffee_details`. 330 | 331 | 2. Integrate DB with the service layer, ensure all the APIs actually perform CRUD 332 | against the DB. 333 | 334 | 3. Write/update the unit tests (tip: use `mockito` to mock db interactions for 335 | the manager impl unit test, you can run a unit test using 336 | `@RunWith(MockitoJUnitRunner.class)`, and use `@Spy` and `@InjectMocks` on 337 | your service/manager impl class). 338 | 339 | 4. Implement FSM, use events for state transition and integrate it with a custom 340 | brewing simulation algorithm that introduces random sleep intervals. Finish 341 | the feature based on the spec. 342 | 343 | Challenge: Attempt and fix any upstream CloudStack DB related issue(s): 344 | https://github.com/apache/cloudstack/issues 345 | -------------------------------------------------------------------------------- /hack/framework.md: -------------------------------------------------------------------------------- 1 | # CloudStack Framework and Plugin Development 2 | 3 | A typical CloudStack framework would define a policy and provide building 4 | blocks, utilities and perhaps export global settings etc and define a plugin 5 | interface that would have methods to extend and implement a mechanism. 6 | 7 | ## Plugin Development 8 | 9 | Your service/manager implementation class can be used for any framework (policy) 10 | implementation. In addition, you need to define a plugin interface that 11 | can implement some mechanism (usually a bunch of methods). The interface needs 12 | to extend the `Adapter` interface which is used throughout CloudStack for plugin 13 | implementation. 14 | 15 | For example: 16 | 17 | ```java 18 | package org.apache.cloudstack.api; 19 | 20 | public interface CoffeeMachine extends Adapter { 21 | Coffee brew(Coffee coffee); 22 | String getName(); 23 | } 24 | ``` 25 | 26 | Next, you can implement the plugin(s) in the same feature maven project or as a 27 | separate maven project. The plugin implementation class need to implement the 28 | defined plugin interface, and may extend the `AdapterBase` abstract class. 29 | 30 | For example: 31 | 32 | ```java 33 | public class DummyCoffeeMachine extends AdapterBase implements CoffeeMachine { 34 | @Override 35 | public Coffee brew(Coffee coffee) { 36 | try { 37 | Thread.sleep(5000L); 38 | } catch (InterruptedException ignored) { 39 | } 40 | return coffee; 41 | } 42 | 43 | @Override 44 | public String getName() { 45 | return "dummy"; 46 | } 47 | } 48 | ``` 49 | 50 | Whether the plugin class is implemented in your feature maven project or as a 51 | separate maven project, define a bean so it may be discovered and consumed by 52 | the framework. For example: 53 | 54 | ```xml 55 | 56 | ``` 57 | 58 | Notes: 59 | - In CloudStack, the plugin class is typically named as a `Provider`. 60 | - The plugin may also export its own global settings by implementing 61 | `Configurable`. It may export its own APIs etc as well. 62 | 63 | ## Plugin Discovery and Usage 64 | 65 | Once you've your plugins implemented, you need to define a registry of plugins 66 | based on the plugin interface, then such a registry can be used to inject/set 67 | a list of discovered plugin (beans implementing a certain plugin interface) 68 | to your framework service/manager impl class. 69 | 70 | Spring can be used to create a registry, usually of type `ExtensionRegistry`, 71 | and use `RegistryLifecycle` to discover and add the plugin (bean) to the 72 | registry based on the type of class (the plugin interface). 73 | 74 | For example: 75 | 76 | ```xml 77 | 78 | 79 | 80 | 81 | 82 | 83 | 84 | 85 | ``` 86 | 87 | Finally, the registry can be used to inject/set the list of discovered plugin 88 | beans to the framework service/manager impl class. For example: 89 | 90 | ```xml 91 | 92 | 93 | 94 | ``` 95 | 96 | ```java 97 | public class CoffeeManagerImpl extends ManagerBase implements CoffeeManager, Configurable, PluggableService { 98 | // .. code redacted .. 99 | List coffeeMachines = new ArrayList<>(); 100 | // the following setter will be used by Spring to set the discovered plugins 101 | public void setCoffeeMachines(final List coffeeMachines) { 102 | this.coffeeMachines = coffeeMachines; 103 | } 104 | ``` 105 | 106 | Note: in case of implementing an external plugin, the 107 | `org.apache.cloudstack.spring.lifecycle.registry.RegistryLifecycle` bean may be 108 | re-defined on the plugin's context xml for the bean to get discovered and 109 | injected in the manager class. Alternatively, you can define a hierarical plugin 110 | dependency mechanism that will be a cleaner approach, see the 111 | `OutOfBandManagementDriver` and the plugins as an example. 112 | 113 | The framework could read a `ConfigKey` that defines the plugin selected for a 114 | scope (global, zone, cluster etc) and use the configured plugin to carry out 115 | an action (mechanism), while it can implement the policy. For finding the plugin 116 | by name, the plugin typically should export its name or some identifier and the 117 | framework can create an internal map of string->plugin to find and use the 118 | configured plugin. 119 | 120 | For example: 121 | ```java 122 | // Define map of string -> plugin 123 | Map coffeeMachineMap = new HashMap<>(); 124 | // .. code redacted .. 125 | // Example code of how a map can be built at the time your framework starts 126 | public boolean start() { 127 | coffeeMachineMap.clear(); 128 | for (final CoffeeMachine machine : coffeeMachines) { 129 | coffeeMachineMap.put(machine.getName(), machine); 130 | } 131 | return true; 132 | } 133 | // Example code of writing a plugin getter based on `ConfigKey` 134 | private CoffeeMachine getCoffeeMachine() { 135 | final String coffeeMachinePlugin = CoffeeMachinePlugin.value(); 136 | if (coffeeMachineMap.containsKey(coffeeMachinePlugin)) { 137 | return coffeeMachineMap.get(coffeeMachinePlugin); 138 | } 139 | throw new CloudRuntimeException("Invalid coffee machine configured!"); 140 | } 141 | // Example code of how configured plugin can be used to carry out task 142 | public Coffee createCoffee(CreateCoffeeCmd cmd) { 143 | // Validations here 144 | final CoffeeVO coffee = coffeeDao.persist(new CoffeeVO(cmd.getName(), CallContext.current().getCallingAccountId())); 145 | // Policy: Change state to reflect ongoing operation 146 | coffee.setState(Coffee.CoffeeState.Brewing); 147 | coffeeDao.update(coffee.getId(), coffee); 148 | // Mechanism: Brew coffee using configured Coffee Machine plugin 149 | getCoffeeMachine().brew(coffee); 150 | // Policy: Update state to brewed 151 | coffee.setState(Coffee.CoffeeState.Brewed); 152 | coffeeDao.update(coffee.getId(), coffee); 153 | return coffee; 154 | } 155 | ``` 156 | 157 | Reference examples of features that implement framework and plugins: 158 | - https://cwiki.apache.org/confluence/display/CLOUDSTACK/Secure+Agent+Communications 159 | - https://cwiki.apache.org/confluence/display/CLOUDSTACK/Out-of-band+Management+for+CloudStack 160 | - https://cwiki.apache.org/confluence/display/CLOUDSTACK/Host+HA 161 | 162 | ## Exercises 163 | 164 | 1. Implement at least two `CoffeeMachine` plugins, with at least one of them 165 | implemented as a separate maven project in whose module.properties your 166 | feature should be the parent. Implement the `ConfigKey` for the 167 | coffee machine plugin name and use it to drive brewing operations. 168 | 169 | 2. Write unit tests for the classes. 170 | -------------------------------------------------------------------------------- /hack/packaging.md: -------------------------------------------------------------------------------- 1 | # Packaging 2 | 3 | RPM packaging references: 4 | - https://docs.fedoraproject.org/en-US/package-maintainers/Packaging_Tutorial_2_GNU_Hello/ 5 | - https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html-single/rpm_packaging_guide/index 6 | - https://rpm-guide.readthedocs.io/en/latest/rpm-guide.html 7 | 8 | DEB packaging references: 9 | - https://www.debian.org/doc/devel-manuals#packaging-tutorial 10 | - https://www.debian.org/doc/manuals/maint-guide/index.en.html 11 | - https://wiki.debian.org/Packaging/Intro 12 | - https://wiki.debian.org/BuildingTutorial 13 | - http://packaging.ubuntu.com/html/packaging-new-software.html 14 | 15 | Reference implementation of a CloudStack feature/plugin packages outside of 16 | the CloudStack tree: https://github.com/shapeblue/ccs/tree/master/packaging 17 | 18 | ## RPM Package Development 19 | 20 | The rpm `spec` files for `centos{6,7}` are at `packaging` directory where you 21 | can define the package. 22 | 23 | As an guided example, here's how you can define a new `rpm` package 24 | `cloudstack-hackerbook-coffee` in `packaging/centos7/cloud.spec`: 25 | 26 | Define the package name and metadata such as requirements, descriptions etc: 27 | 28 | ``` 29 | %package hackerbook-coffee 30 | Summary: Apache CloudStack Hackerbook Coffee feature 31 | Requires: %{name}-management = %{_ver} 32 | Group: System Environmnet/Libraries 33 | %description hackerbook-coffee 34 | Apache CloudStack Hackerbook Coffee feature 35 | ``` 36 | 37 | In the spec's `%install` section define how you want the feature artifacts to be 38 | installed: 39 | 40 | ``` 41 | # hackerbook-coffee feature 42 | mkdir -p ${RPM_BUILD_ROOT}%{_datadir}/%{name}-hackerbook-coffee/lib 43 | cp -r plugins/hackerbook/feature/target/cloud-plugin-hackerbook-feature-%{_maventag}.jar ${RPM_BUILD_ROOT}%{_datadir}/%{name}-management/lib 44 | ``` 45 | 46 | You may add additional `%preun`, `%pre`, `%post` sections for the package to 47 | define how it gets installed/upgraded/uninstalled etc. 48 | 49 | Finally define the `%files` section: 50 | 51 | ``` 52 | %files hackerbook-feature 53 | %defattr(0644,cloud,cloud,0755) 54 | %attr(0644,root,root) %{_datadir}/%{name}-management/lib/*hackerbook-feature*jar 55 | ``` 56 | 57 | ## DEB Package Development 58 | 59 | Debian packaging rules and configuration files are in the `debian` directory. 60 | 61 | As an guided example, here's how you can define a new deb package 62 | `cloudstack-hackerbook-coffee` in `debian`: 63 | 64 | Define your new package in `debian/control`: 65 | 66 | ``` 67 | Package: cloudstack-hackerbook-coffee 68 | Architecture: all 69 | Depends: ${misc:Depends}, cloudstack-management (= ${source:Version}) 70 | Description: The CloudStack Hackerbook feature 71 | ``` 72 | 73 | In the `override_dh_auto_install` section of `debian/rules` define how your 74 | feature's artifacts are copied, installed: 75 | 76 | ``` 77 | # cloudstack-hackerbook-coffee 78 | mkdir -p $(DESTDIR)/usr/share/$(PACKAGE)-management/lib 79 | cp -r plugins/hackerbook/feature/target/cloud-plugin-hackerbook-feature-*jar $(DESTDIR)/usr/share/$(PACKAGE)-management/lib/ 80 | ``` 81 | 82 | Define what artifacts your package will install, for example create a file 83 | `debian/cloudstack-hackerbook-coffee.install` that lists files that will be 84 | installed by your package: 85 | 86 | ``` 87 | /usr/share/cloudstack-management/lib/*hackerbook-coffee* 88 | ``` 89 | 90 | In addition, you can define custom script to execute for before/after package 91 | install/uninstall/upgrade etc by defining the 92 | `debian/cloudstack-hackerbook-coffee.{preinst,postinst}` files as needed. 93 | 94 | ## Exercises 95 | 96 | Package the feature and any of its plugins (jars and UI plugin) as a separately 97 | installed rpm/deb package. 98 | -------------------------------------------------------------------------------- /hack/service.md: -------------------------------------------------------------------------------- 1 | # CloudStack Service Layer Development 2 | 3 | ## Declaring Loadable Module 4 | 5 | Most CloudStack APIs are handled by a service layer class that usually called 6 | a `XXManagerImpl` or `YYServiceImpl`. For a new feature developed as a plugin, 7 | the plugin needs to export how it should be loaded when management server 8 | starts for which CloudStack has its own Spring based module loading system. 9 | 10 | The following is a typical CloudStack module hierarchy where various CloudStack 11 | features implemented as separate maven projects or plugins and usually built as 12 | separate jars (such as hypervisors, networks, storages) are loaded: 13 | 14 | ```bash 15 | bootstrap 16 | └── system 17 |    └── core 18 |    └── allocator 19 |    └── api 20 |    └── backend 21 |    └── ca 22 |    └── compute 23 |    └── kvm 24 |    └── vmware 25 |    └── xenserver 26 |    └── ... 27 |    └── network 28 |    └── storage 29 |    └── discoverer 30 | ``` 31 | 32 | Following this, you need to create a module definition file that declares 33 | how your `feature` should be loaded. For example, at 34 | `plugins/hackerbook/feature/src/main/resources/META-INF/cloudstack/feature`: 35 | 36 | ```bash 37 | > cat module.properties 38 | name=feature 39 | parent=backend 40 | ``` 41 | 42 | Next, in the same folder you need to declare a context `xml` file where you 43 | declare a java class that can be instantiated and loaded by Spring as a bean. 44 | The name of the context file is usually named as 45 | `spring-nameofyourmodule-context.xml`. Use the module name same as declared 46 | in the `module.properties` file. 47 | 48 | ```xml 49 | > cat spring-feature-context.xml 50 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | ``` 61 | 62 | The `Spring` framework is widely used in CloudStack primarily for dependency 63 | injection, context, AOP and web (servlet). 64 | 65 | Reference readings: 66 | - https://cwiki.apache.org/confluence/display/CLOUDSTACK/Using+Spring+in+CloudStack 67 | - https://docs.spring.io/spring/docs/5.1.3.RELEASE/spring-framework-reference/ 68 | - https://docs.spring.io/spring/docs/current/spring-framework-reference/core.html#aop 69 | - https://dzone.com/articles/how-dependency-injection-di-works-in-spring-java-a 70 | - https://www.journaldev.com/2410/spring-dependency-injection 71 | - https://www.baeldung.com/inversion-control-and-dependency-injection-in-spring 72 | - https://www.tutorialspoint.com/spring 73 | - https://www.javatpoint.com/spring-tutorial 74 | 75 | ## Manager Implementation 76 | 77 | In CloudStack each implementation has a separate API contract class which is 78 | typically a Java interface (and sometimes an abstract class) and separates 79 | how an entity/object/resource is declared and defined. 80 | 81 | For the manager, implement a `CoffeeManager` interface that defines a contract 82 | of methods and fields that the implementation class should implement. For 83 | example: 84 | 85 | ```java 86 | package org.apache.cloudstack.api; 87 | 88 | public interface CoffeeManager { 89 | // define methods here as you progress your implementation 90 | } 91 | ``` 92 | 93 | The manager implementation class would implement the `CoffeeManager` interface 94 | and few other interfaces. The `ManagerBase` defines a typical manager component 95 | that internally is based on `ComponentLifecycleBase`. The `Configurable` 96 | interface is used if the manager needs to export some global settings. The 97 | `PluggableService` is used if the manager needs to export some API commands. 98 | 99 | A manager lifecycle has basically three methods: 100 | - `configure`: this is called before a manager class is started and useful to 101 | configure and instantiate any internal datastructures and configurations. 102 | - `start`: this is called before a manager class is started. Only after the 103 | manager starts it can then handle any API requests etc. 104 | - `stop`: this is called before a manager is stopped, for example due to 105 | management server shutdown, and provides means to perform cleanups. 106 | 107 | The following manager boilerplate may be used: 108 | 109 | ```java 110 | package org.apache.cloudstack.feature; 111 | 112 | // ...declare imports... 113 | 114 | public class CoffeeManagerImpl extends ManagerBase implements CoffeeManager, Configurable, PluggableService { 115 | 116 | private static Logger LOGGER = Logger.getLogger(CoffeeManagerImpl.class); 117 | 118 | @Override 119 | public boolean configure(String name, Map params) throws ConfigurationException { 120 | super.configure(name, params); 121 | // Add code on how to handle when this is configured 122 | return true; 123 | } 124 | 125 | @Override 126 | public boolean start() { 127 | // Add code on how to handle when this is started 128 | return true; 129 | } 130 | 131 | @Override 132 | public boolean stop() { 133 | // Add code on how to handle when this is stopped 134 | return true; 135 | } 136 | 137 | @Override 138 | public List> getCommands() { 139 | final List> cmdList = new ArrayList<>(); 140 | // add API cmd classes here 141 | return cmdList; 142 | } 143 | 144 | @Override 145 | public String getConfigComponentName() { 146 | return CoffeeManager.class.getSimpleName(); 147 | } 148 | 149 | @Override 150 | public ConfigKey[] getConfigKeys() { 151 | return new ConfigKey[]{ 152 | // Add ConfigKeys here 153 | }; 154 | } 155 | } 156 | ``` 157 | 158 | Naming convention: depending on the use-case, whether the service/business class 159 | is managing an object or providing a service its name may either have `Manager` 160 | or `Service`. For example, `CoffeeService` and `CoffeeManager`, and their 161 | implementing class may be named `CoffeeServiceImpl` or `CoffeeManagerImpl`. For 162 | the exercise, since the service layer class will be managing the resource 163 | `Coffee` the advice is to use the names `CoffeeManager/CoffeeManagerImpl`. 164 | 165 | ## Defining Resource Interface 166 | 167 | The main resource/object for the feature is `Coffee`, define an interface 168 | that can be used to export contract (methods, enums, constants etc) for that 169 | resource. For example: 170 | 171 | ```java 172 | 173 | package org.apache.cloudstack.api; 174 | public interface Coffee extends InternalIdentity, Identity { 175 | 176 | enum Size { 177 | SMALL, MEDIUM, LARGE 178 | } 179 | 180 | enum Offering { 181 | Espresso, Cappuccino, Mocha, Latte 182 | } 183 | 184 | enum CoffeeState { 185 | Created, Brewing, Brewed, Discarded; 186 | } 187 | 188 | // Define other enum, constants, events, fsm etc. 189 | 190 | String getName(); 191 | String getDescription(); 192 | // Define other methods 193 | } 194 | ``` 195 | 196 | Ensure that you've also used the resource interface to annotate the response 197 | class such as: 198 | 199 | ```java 200 | @EntityReference(value = Coffee.class) 201 | public class CoffeeResponse extends BaseResponse { 202 | ``` 203 | 204 | We do this to make it easier for the API layer for the id->uuid translation 205 | that we'll discuss in the DB exercise. 206 | 207 | ## API Request Handling 208 | 209 | The typical handler implementation can be to inject the manager in the API 210 | command and in the API's execute() call that dependency (manager/service) 211 | to handle the request by passing arguments or the `cmd` object. For example: 212 | 213 | Define the handler in the manager/service interface: 214 | 215 | ```java 216 | public interface CoffeeManager { 217 | List listCoffees(ListCoffeesCmd cmd); 218 | // Other definitions 219 | } 220 | ``` 221 | 222 | The implementation can implement it, such as: 223 | 224 | ```java 225 | public class CoffeeManagerImpl extends ManagerBase implements CoffeeManager, Configurable, PluggableService { 226 | // ... code redacted ... 227 | @Override 228 | public List listCoffees(ListCoffeesCmd cmd) { 229 | // Gather inputs from cmd and validate them 230 | // Search DB based on input params 231 | // Returned paginated list 232 | return Collections.emptyList(); 233 | } 234 | ``` 235 | 236 | In the API command, inject the manager/service and use its methods: 237 | ```java 238 | public class ListCoffeesCmd extends BaseListCmd { 239 | // ... redacted code ... 240 | @Inject 241 | private CoffeeManager coffeeManager; 242 | 243 | // ... redacted code ... 244 | @Override 245 | public void execute() { 246 | final List coffees = coffeeManager.listCoffees(this); 247 | 248 | // Validate coffee list here 249 | 250 | // Create response, for example in-line or in separate method 251 | final List responseList = new ArrayList<>(); 252 | for (final Coffee coffee : coffees) { 253 | CoffeeResponse response = new CoffeeResponse(); 254 | response.setId(coffee.getUuid()); 255 | response.setName(coffee.getName()); 256 | // other setters etc. 257 | } 258 | final ListResponse coffeeResponses = new ListResponse<>(); 259 | coffeeResponses.setObjectName("coffee"); 260 | coffeeResponses.setResponses(responseList); 261 | coffeeResponses.setResponseName(getCommandName()); 262 | setResponseObject(coffeeResponses); 263 | } 264 | ``` 265 | 266 | Ensure that the API is exported by the service layer class: 267 | 268 | ```java 269 | public List> getCommands() { 270 | final List> cmdList = new ArrayList<>(); 271 | cmdList.add(ListCoffeesCmd.class); 272 | ... 273 | ``` 274 | 275 | ## Global Settings 276 | 277 | The `cloud-framework-config` provides means of configuration for various 278 | parts of CloudStack that an admin/user can tune. Depending on the use-case, 279 | a `ConfigKey` can be defined in the service layer implementation class (to keep 280 | access restricted) or the service/manager interface (to allow external access). 281 | A `ConfigKey` defines a setting that has a category, type (string, long, int 282 | etc), name, default value, description, if it's dynamic. In addition, it can 283 | have a scope and a multipler. 284 | 285 | **Recommended reading**: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Configuration 286 | 287 | The following is one example to add a zone-level (global) setting: 288 | 289 | ```java 290 | public class CoffeeManagerImpl extends ManagerBase implements CoffeeManager, Configurable, PluggableService { 291 | // ... code redacted ... 292 | private static final ConfigKey CoffeeTTLInterval = new ConfigKey("Advanced", Long.class, 293 | "coffee.ttl.interval", "600", 294 | "The max time in seconds after which coffee becomes stale.", 295 | true, ConfigKey.Scope.Zone); 296 | 297 | // ... code redacted ... 298 | public ConfigKey[] getConfigKeys() { 299 | return new ConfigKey[]{ 300 | CoffeeTTLInterval, 301 | }; 302 | } 303 | ``` 304 | 305 | To access the setting you can use `CoffeeTTLInterval.value()` or 306 | `CoffeeTTLInterval.valueIn(zoneId)`. 307 | 308 | ## Background Tasks 309 | 310 | You can create a class that extends `ManagedContextRunnable` and implements 311 | `BackgroundPollTask` to create a background task. Such a background task 312 | is scheduled to run continuously using the `scheduleWithFixedDelay` method 313 | that the background polling task manager implementation uses on an executor 314 | service. 315 | 316 | Reference reading: 317 | - https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/ExecutorService.html 318 | - http://tutorials.jenkov.com/java-util-concurrent/executorservice.html 319 | - https://www.baeldung.com/java-executor-service-tutorial 320 | 321 | Here is an example of how you may declare a background task: 322 | 323 | ```java 324 | private static final class CoffeeGCTask extends ManagedContextRunnable implements BackgroundPollTask { 325 | private CoffeeManager coffeeManager; 326 | 327 | private CoffeeGCTask(final CoffeeManager coffeeManager) { 328 | this.coffeeManager = coffeeManager; 329 | } 330 | 331 | @Override 332 | protected void runInContext() { 333 | try { 334 | if (LOGGER.isTraceEnabled()) { 335 | LOGGER.trace("Coffee GC task is running..."); 336 | } 337 | // Code to do based on Coffee TTL 338 | final Long ttl = CoffeeTTLInterval.value(); 339 | 340 | } catch (final Throwable t) { 341 | LOGGER.error("Error trying to run Coffee GC task", t); 342 | } 343 | } 344 | 345 | @Override 346 | public Long getDelay() { 347 | return CoffeeGCInterval.value(); 348 | } 349 | } 350 | ``` 351 | 352 | Finally, submit an instance of the background task to the BackgroundPollManager 353 | in the `configure` of your service/manager class: 354 | 355 | ```java 356 | @Inject 357 | private BackgroundPollManager backgroundPollManager; 358 | 359 | // ... code redacted ... 360 | public boolean configure(String name, Map params) throws ConfigurationException { 361 | super.configure(name, params); 362 | backgroundPollManager.submitTask(new CoffeeGCTask(this)); 363 | return true; 364 | } 365 | ``` 366 | 367 | ## Exercises 368 | 369 | 1. Implement the manager module, interface and impl class. 370 | 371 | 2. Add APIs from previous exercise, build and run the management server and 372 | verify that you're able to call those APIs using `cmk`. 373 | 374 | 3. Define a Coffee resource interface and handlers for the APIs in the service 375 | class. 376 | 377 | 4. Define the global settings for the feature. 378 | 379 | 5. Define the logic for the GC task. 380 | 381 | Challenge: Attempt to find and fix service layer issue(s) from: https://github.com/apache/cloudstack/issues?q=is%3Aissue+is%3Aopen+label%3Atype%3Abug+ 382 | -------------------------------------------------------------------------------- /hack/spec.md: -------------------------------------------------------------------------------- 1 | # Specification 2 | 3 | ## Requirements 4 | 5 | Coffee is a beverage that CloudStack needs to support, a typical coffee life 6 | cycle involves it to be created, updated, listed, removed etc. Coffee can be 7 | brewed by hypervisors however a framework should allow developers to write 8 | custom coffee/brewing plugins. The coffee should have a maximum time to live, 9 | a garbage-collection thread to remove stale coffees. The feature should be 10 | available through UI. 11 | 12 | ## Functional Spec 13 | 14 | Your task now is to read and understand the above requirements and then proceed 15 | with the implementation based on a functional specification. 16 | 17 | References: 18 | - https://cwiki.apache.org/confluence/display/CLOUDSTACK/Adding+new+features+and+design+documents 19 | - https://cwiki.apache.org/confluence/display/CLOUDSTACK/Design+Document+Template 20 | - https://cwiki.apache.org/confluence/display/CLOUDSTACK/Design 21 | 22 | FS references and examples: 23 | - CloudStack Design docs (FS): https://cwiki.apache.org/confluence/display/CLOUDSTACK/Design 24 | - FS case study (Dynamic Roles): https://cwiki.apache.org/confluence/display/CLOUDSTACK/Dynamic+Role+Based+API+Access+Checker+for+CloudStack 25 | 26 | The feature can be implemented as a CloudStack plugin at 27 | `plugins/hackerbook/feature` which should also implement a framework and two 28 | coffee machine plugins: 29 | - `default`: the in-built cloudstack brewer machine, implemented at 30 | `plugins/hackerbook/default` 31 | - `kvm`: the brewer machine that brews coffee on a KVM host, implemented at 32 | `plugins/hackerbook/kvm` 33 | 34 | A `Coffee` should follow a finite-state-machine (FSM) with following states: 35 | 36 | - Created: Coffee order created 37 | - Brewing: Coffee is order is sent 38 | - Brewed: Coffee is brewed 39 | - Discarded: Coffee is put in trash 40 | 41 | The `size` of coffee can be as follows: 42 | - `SMALL` 43 | - `MEDIUM` 44 | - `LARGE` 45 | 46 | The following coffee `offering` should be supported: 47 | - Espresso 48 | - Cappuccino 49 | - Mocha 50 | - Latte 51 | 52 | The following sections document high-level implementation details. 53 | 54 | ### APIs 55 | 56 | | API | Arguments | Description | 57 | | --- | --------- | ----------- | 58 | | createCoffee | name, offering, size, details | Creates coffee | 59 | | listCoffees | id, offering, size | Lists coffee by id, type and/or size | 60 | | updateCoffee | id, size, details | Updates coffee order that is not ready yet | 61 | | removeCoffee | id, ids | Removes coffee by id or ids | 62 | 63 | The details are string->string map of any arbitrary key/value to note any 64 | details of each coffee order. 65 | 66 | ### DB Schema 67 | 68 | Following tables should be added with appropriate DAO and VO classes: 69 | 70 | - coffee: 71 | - id: auto_increment ID for the table 72 | - uuid: the UUID for a coffee order 73 | - name: name on the coffee order 74 | - offering: the offering/type of the coffee 75 | - size: size of coffee 76 | - state: the state of Coffee 77 | - account_id: the user account ID 78 | - domain_id: the domain ID 79 | - created: the creation timestamp 80 | - removed: the removal timestamp 81 | 82 | - coffee_details: 83 | - id: auto_increment ID for the table 84 | - coffee_id: the coffee ID 85 | - name: the string key 86 | - value: the string value 87 | - display: whether to show/hide the detail in the API response 88 | 89 | ### Service Changes 90 | 91 | The feature would require a new service/business layer class that handles 92 | various API requests along with exports APIs, global settings and schedules 93 | background task(s). This may also implement the framework and support discovery 94 | and usage of plugins. 95 | 96 | Global settings: 97 | 98 | - `coffee.ttl.interval`: The max time in seconds after which coffee becomes 99 | stale, default 600 seconds. 100 | - `coffee.gc.interval`: The interval in seconds for the GC thread to run, 101 | default 10s. 102 | - `coffee.machine.plugin`: The name of coffee machine plugin, default is `inbuilt`. 103 | 104 | Background task: Run based on GC interval and scan coffees across all accounts 105 | and remove coffee that have exceeded the coffee TTL interval. 106 | 107 | ### Pluggable CoffeeMaker 108 | 109 | Implement a framework that allows plugin creation based on a `CoffeeMaker` 110 | interface that implements a `brew()` method. Implement two plugins, an inbuilt 111 | default coffeemaker that performs no-operation (maybe just log) and a separate 112 | maven project based lazy coffeemaker that randomly sleeps in `brew()`. 113 | 114 | ### Message Passing: IPC and RPC 115 | 116 | On creation of any new CloudStack account, a fresh default coffee must be 117 | created and on removal of an account all coffees of that account must be purged. 118 | On coffee creation and removal, publish event on the message bus. On coffee 119 | discarded by GC thread, send an alert. 120 | 121 | Create a `CoffeeMachine` plugin that brews coffee via a random agent such as the 122 | CPVM/SSVM agent or the KVM agent. 123 | 124 | ### Coffee Usage Records 125 | 126 | Introduce a new coffee usage record type and implement usage parser for the 127 | coffee resource that generates usage records aggregated per account by the 128 | number of coffees ordered. 129 | 130 | ### UI 131 | 132 | The feature should be implemented as a UI plugin which shows up as a new 133 | `coffee` tab in the UI. In the tab, users should see a table of coffee orders 134 | with name, offering, size, state and account. On clicking a coffee, it should 135 | show a details tab that lists the attributes of the coffee. The UI should 136 | provide appropriate buttons to create, update and remove coffee. 137 | 138 | ### Testing 139 | 140 | Create Marvin test for the API and features. 141 | 142 | ### Packaging 143 | 144 | Make the coffee feature and any of its plugins packaged separately as a separate 145 | rpm/deb package. 146 | -------------------------------------------------------------------------------- /hack/testing.md: -------------------------------------------------------------------------------- 1 | # Testing 2 | 3 | **Recommended reading list**: 4 | - https://cwiki.apache.org/confluence/display/CLOUDSTACK/Marvin+-+Testing+with+Python 5 | - https://cwiki.apache.org/confluence/display/CLOUDSTACK/Automated+Tests+Rules+and+Guidelines 6 | 7 | ## Writing Marvin Test 8 | 9 | A marvin based integration test is a Python `unittest` framework based class 10 | that extends `cloudstackTestCase`, and run using the `nose` test runner. The 11 | test can take input from the marvin (json) config file, the inbuilt-test 12 | `test_data.py` map or any custom internal/external service map. 13 | 14 | Here is an illustrated example on writing a Marvin based integration test: 15 | 16 | ```python 17 | # Local and marvin related imports (tip: import specifics than all) 18 | from marvin.cloudstackTestCase import cloudstackTestCase 19 | from marvin.cloudstackAPI import (listCoffees) 20 | from marvin.lib.utils import (cleanup_resources) 21 | from marvin.lib.base import * # import all resources like Account 22 | from marvin.lib.common import * # import all utility methods like get_zone etc. 23 | from marvin.sshClient import SshClient 24 | 25 | # Import runner related 26 | from nose.plugins.attrib import attr 27 | 28 | # Import system/additional libraries here 29 | import os 30 | 31 | class TestCoffee(cloudstackTestCase): 32 | 33 | # Define class globals here 34 | 35 | # Class's setUp method, runs only once per test class before test cases 36 | @classmethod 37 | def setUpClass(cls): 38 | testClient = super(TestCoffee, cls).getClsTestClient() 39 | cls.apiclient = testClient.getApiClient() 40 | cls.services = testClient.getParsedTestDataConfig() 41 | 42 | # Get attributes 43 | cls.zone = get_zone(cls.apiclient, testClient.getZoneForTests()) 44 | cls.domain = get_domain(cls.apiclient) 45 | cls.hypervisor = testClient.getHypervisorInfo() 46 | 47 | # Define/create other resources 48 | cls.account = Account.create( 49 | cls.apiclient, 50 | cls.services["account"], 51 | domainid=cls.domain.id 52 | ) 53 | cls.cleanup = [ 54 | cls.account 55 | ] 56 | 57 | # Class's tearDown method, runs only once per test class after test cases 58 | @classmethod 59 | def tearDownClass(cls): 60 | try: 61 | cleanup_resources(cls.apiclient, cls.cleanup) 62 | except Exception as e: 63 | raise Exception("Warning: Exception during cleanup : %s" % e) 64 | 65 | # Define setUp that runs before each test case 66 | def setUp(self): 67 | # Define probes here 68 | # Probe: apiclient to make API calls 69 | self.apiclient = self.testClient.getApiClient() 70 | # Probe: DB client to query the database 71 | self.dbclient = self.testClient.getDbConnection() 72 | # Probe: SSH client to run remote commands 73 | self.sshclient = SshClient( 74 | ipaddress, 75 | 22, # port 76 | username, 77 | password, 78 | retries=1, 79 | log_lvl=logging.INFO 80 | ) 81 | # Get hypervisor detail 82 | self.hypervisor = self.testClient.getHypervisorInfo() 83 | # Command to get the default test_data config 84 | self.services = self.testClient.getParsedTestDataConfig() 85 | # List to hold any resources requiring cleanup 86 | self.cleanup = [] 87 | 88 | # Define tearDown that runs after each test case 89 | def tearDown(self): 90 | try: 91 | cleanup_resources(self.apiclient, self.cleanup) 92 | except Exception as e: 93 | raise Exception("Warning: Exception during cleanup : %s" % e) 94 | 95 | # Define test cases with tags 96 | @attr(tags = ["advanced", "basic"], required_hardware="false") 97 | def test_list_coffee(self): 98 | # Use the inbuilt logger using `self.debug` 99 | self.debug("Test list coffee") 100 | 101 | # CloudStack build auto-generates API cmd/response python classes 102 | # All API command classes have the API name + `Cmd` suffix 103 | # Define the cmd object and its attributes 104 | cmd = listCoffees.listCoffeesCmd() 105 | # cmd.id = ...xyz... 106 | 107 | # Probe: Execute API by using `apiclient.(cmd_obj)`: 108 | response = self.apiclient.listCoffees(cmd) 109 | 110 | # Probe: db example: 111 | row = self.dbclient.execute("select * from coffee where id=1")[0] 112 | 113 | # Probe: ssh client example: 114 | result = sshclient.runCommand("uname -a") 115 | 116 | # Tip: Add any resources created by test case to `self.cleanup.append(resource)` 117 | 118 | # Utility example: using wait_until to poll for a resource state/expectation 119 | def checkCoffeeBrewed(): 120 | # Perform API call to list a coffee resource 121 | # Return true/false and any additional fields 122 | return coffee.state == "Brewed", None 123 | # Definition: wait_until(sleep_seconds, retries, callback, *callbackArgs) 124 | res, _ = wait_until(2, 30, checkCoffeeBrewed) 125 | if not res: 126 | self.fail("Failed to get Coffee in Brewed state") 127 | 128 | # Assertion examples: 129 | self.assertTrue(obj.attribute == conditional, "Some description") 130 | self.assertFalse(obj.attribute == conditional, "Some description") 131 | self.assertIsNotNone(obj, "Some description") 132 | self.assertEqual( 133 | isinstance(list_response, list), 134 | True, 135 | msg="Some message about response" 136 | ) 137 | with self.assertRaises(Exception): # Assert failure on API 138 | self.apiclient.runSomeApi(cmd) 139 | 140 | # Skip/fail test examples: 141 | self.skipTest("Some message) 142 | self.fail("Some message") 143 | ``` 144 | 145 | ## Exercises 146 | 147 | 1. Write test cases to verify positive/negative inputs for the CRUD APIs. 148 | 149 | 2. Create coffee and verify that the GC thread garbage collects it based on 150 | the `coffee` TTL global setting. 151 | 152 | 3. Verify brewing operation for addition plugins wherever possible. 153 | 154 | Challenge: Attempt to fix any failing marvin integration test. 155 | -------------------------------------------------------------------------------- /hack/ui.md: -------------------------------------------------------------------------------- 1 | # CloudStack UI Development 2 | 3 | For the modern UI, starting 4.15, refer to the following: 4 | - https://github.com/apache/cloudstack/tree/main/ui#development 5 | - https://github.com/apache/cloudstack/blob/main/ui/docs/development.md 6 | - (Optional) https://github.com/apache/cloudstack/blob/main/ui/docs/customize.md 7 | 8 | Learn VueJS: 9 | - https://vuejs.org/guide/introduction.html 10 | - https://www.youtube.com/results?search_query=Vue.js+3+tutorial 11 | 12 | ## Exercises 13 | 14 | 1. Implement support for Coffee feature in the UI 15 | 16 | 2. Implement list and details views along with CRUD action for all the APIs 17 | 18 | 3. Implement a customised Vue component for say the Coffee Create form. 19 | 20 | Challenge: Attempt and fix CloudStack UI issue(s) 21 | https://github.com/apache/cloudstack/issues?q=is%3Aissue+is%3Aopen+label%3Acomponent%3AUI 22 | -------------------------------------------------------------------------------- /hack/usage.md: -------------------------------------------------------------------------------- 1 | # Usage Development 2 | 3 | The CloudStack usage server is responsible for event processing, parsing and 4 | creation of usage records for various usage types. The usage server has only 5 | access to the `cloud_usage` database where it has several helper tables to 6 | hold metadata about a resource, it's usage information and added/removed 7 | timestamps. 8 | 9 | Related references: 10 | - http://docs.cloudstack.apache.org/en/4.15.0.0/adminguide/usage.html 11 | - http://docs.cloudstack.apache.org/en/4.15.0.0/plugins/quota.html 12 | 13 | The usage server can be started using maven using: 14 | 15 | mvn -P usage -Drun -Dpid=$$ -pl usage 16 | 17 | CloudStack usage server creates usage jobs in `cloud_usage.usage_job` table 18 | based on the `usage.stats.job.aggregation.range` global setting. To create an 19 | adhoc usage job, the API `generateUsageRecords` can be called. 20 | 21 | Note: 22 | - To enable debug logging, replace the `@USAGELOG@` to something like 23 | `usage.log` and `INFO` to `DEBUG` in 24 | `usage/target/transformed/log4j-cloud_usage.xml` and copy/rename it as 25 | `log4j.xml` to `usage/target/classes`. 26 | - To run usage jobs quickly lower the usage aggregation interval. 27 | 28 | In this exercise you'll learn how to add usage parsing and processing for a new 29 | resource, `Coffee`. First define the usage type for the new resource in 30 | `UsageTypes` and confirm it in the `listUsageTypes` response: 31 | 32 | ```java 33 | public static final int COFFEE = 40; 34 | // .. code redacted .. 35 | responseList.add(new UsageTypeResponse(COFFEE, "COFFEE", "Coffee usage")); 36 | ``` 37 | 38 | Ensure that usage events are emitted by your service layer code when (for 39 | example) a resource is created/removed etc. using 40 | `UsageEventUtils::publishUsageEvent`. 41 | 42 | Define a helper table such as `cloud_usage.usage_coffee` where you store the 43 | `coffee_id`, `accountId`, `created` and `removed`, and any additional metadata 44 | you want to export in the `listUsageRecords` response. Implement appropriate VO 45 | and Dao classes in `engine/schema`. The `Dao` would at least require a method 46 | to get usage records by accountId, start and end dates, and two helper methods 47 | to save and remove usage record. 48 | 49 | For example: 50 | 51 | ```sql 52 | CREATE TABLE IF NOT EXISTS `cloud_usage`.`usage_coffee` ( 53 | `id` bigint(20) unsigned NOT NULL AUTO_INCREMENT, 54 | `account_id` bigint(20) unsigned NOT NULL, 55 | `domain_id` bigint(20) unsigned NOT NULL, 56 | `coffee_id` bigint(20) unsigned NOT NULL, 57 | `size` bigint(20) DEFAULT 0, 58 | `created` datetime NOT NULL, 59 | `removed` datetime, 60 | PRIMARY KEY (`id`), 61 | INDEX `i_usage_coffee` (`account_id`,`coffee_id`,`created`) 62 | ) ENGINE=InnoDB CHARSET=utf8; 63 | ``` 64 | 65 | From the API exercise, the create and remove APIs would emit create/remove 66 | events that can be used by the usage server to parse and create usage records. 67 | 68 | In `UsageManagerImpl`, write the parser and handlers such as: 69 | 70 | ```java 71 | @Inject 72 | private UsageCoffeeDao usageCoffeeDao; 73 | 74 | // In parseHelperTables: 75 | parsed = CoffeeUsageParser.parse(account, currentStartDate, currentEndDate); 76 | 77 | // In createHelperRecord: 78 | if (isCoffeeEvent(eventType)) { 79 | createCoffeeEvent(event); 80 | } 81 | 82 | // define isCoffeeEvent method 83 | private boolean isCoffeeEvent(String eventType) { 84 | return eventType != null && (eventType.equals(EventTypes.EVENT_COFFEE_CREATE) || eventType.equals(EventTypes.EVENT_COFFEE_DELETE)); 85 | } 86 | 87 | // define createCoffeeEvent method 88 | private void createCoffeeEvent(final UsageEventVO event) { 89 | Long coffeeId = event.getResourceId(); 90 | Long accountId = event.getAccountId(); 91 | Date created = event.getCreateDate(); 92 | Account account = _accountDao.findByIdIncludingRemoved(event.getAccountId()); 93 | if (EventTypes.EVENT_COFFEE_CREATE.equals(event.getType())) { 94 | // Create UsageCoffeeVO and save it 95 | } else if (EventTypes.EVENT_COFFEE_DELETE.equals(event.getType())) { 96 | // Remove UsageCoffeeVO by the coffeeId 97 | } 98 | } 99 | ``` 100 | 101 | Note: 102 | - The UsageCoffeeDao would need to only interact with the `cloud_usage` database. 103 | - Transaction.execute can be used to ensure your db logic only gets executed 104 | against the `cloud_usage` db, see `QuotaUsageDaoImpl` as a modern example and 105 | `UsageVMInstanceDaoImpl` as an old-styled example. 106 | 107 | Finally, you'll need to add a `CoffeeUsageParser` that will aggegrate coffee 108 | usage by the account and persist usage record. For example: 109 | 110 | ```java 111 | @Component 112 | public class CoffeeUsageParser { 113 | private static UsageDao s_usageDao; 114 | private static UsageCoffeeDao s_usageCoffeeDao; 115 | 116 | @Inject 117 | private UsageDao usageDao; 118 | @Inject 119 | private UsageCoffeeDao usageCoffeeDao; 120 | 121 | @PostConstruct 122 | void init() { 123 | s_usageDao = usageDao; 124 | s_usageCoffeeDao = usageCoffeeDao; 125 | } 126 | 127 | public static boolean parse(AccountVO account, Date startDate, Date endDate) { 128 | if ((endDate == null) || endDate.after(new Date())) { 129 | endDate = new Date(); 130 | } 131 | final List usageCoffees = s_usageCoffeeDao.getUsageRecords(account.getId(), startDate, endDate); 132 | if (usageCoffees == null || usageCoffees.isEmpty()) { 133 | LOGGER.debug("No Coffee usage for this period"); 134 | return true; 135 | } 136 | for (final UsageCoffeeVO usageCoffee : usageCoffees) { 137 | // Logic to aggregate coffee usage by the account id 138 | // Use a hashmap for this 139 | } 140 | // Loop through the items in hashmap 141 | // Create UsageVO for aggregate coffee consumption per account 142 | final UsageVO usageRecord = new UsageVO(zoneId, account.getAccountId(), 143 | account.getDomainId(), description, usageDisplay, 144 | UsageTypes.COFFEE, rawUsage, null, null, null, null, null, 145 | coffeeCount, null, startDate, endDate); 146 | s_usageDao.persist(usageRecord); 147 | return true; 148 | } 149 | } 150 | ``` 151 | 152 | ## Exercises 153 | 154 | 1. Implement the Coffee usage resource, its parser, helper tables, VO and Dao 155 | classes. To avoid maven/spring dependency hell, you may either implement a 156 | a new maven project `coffee-schema` and use that as dependency for your 157 | feature and the usage server, or simply add the schema related classes in the 158 | engine-schema project. 159 | 160 | 2. Validate that usage records are created by scheduled job or adhoc job created 161 | using `generateUsageRecords` API. Verify the usage records in the 162 | `listUsageRecords` response. 163 | 164 | 3. Verify that usage records stop generating when coffee is removed. 165 | -------------------------------------------------------------------------------- /primer/capc.md: -------------------------------------------------------------------------------- 1 | # CloudStack Cluster API Provider (CAPC) 2 | 3 | ## What is the Cluster API ? 4 | 5 | [Kubernetes Cluster API](https://cluster-api.sigs.k8s.io/) is a Kubernetes project focused on providing declarative APIs and tooling to simplify provisioning, upgrading, and operating multiple Kubernetes clusters. 6 | 7 | If this has too much jargon, it pretty much means that everyone has their own version of Kubernetes as a Service and the way they operate depends on the provider. CAPI aims to standardize and fix that! 8 | 9 | CAPI defines the common CRUD operations on the Kubernetes Clusters along with default implementations for them. It aims to simplify the management of the cluster lifecycle and provide a single pane of glass to view and manage your Kubernetes clusters. 10 | 11 | ## How does it work ? 12 | 13 | Have you seen inception ? If you have you'll have a pretty good idea! It's Kubernetes managing Kubernetes! 14 | 15 | It starts off with running a Kubernetes cluster. It can run anywhere, on the cloud, in your infra or locally on your laptop! 16 | This cluster is called your management cluster. It is this cluster that manages all your other Kubernetes Clusters. 17 | 18 | With the `clusterctl` tool, you can initialize your cluster into a management cluster with the specific infrastructure provider. 19 | This results in a bunch of pods running on your management cluster : 20 | - CAPI Controller : This is the CAPI orchestrator that is the brain of the operation 21 | - Control Plane Manager : This manages the control planes of your workload clusters and checks their status and if they are up and running 22 | - Bootstrap Provider : This provides the relevant userdata which is used to convert your VM into a Kubernetes node 23 | - Infrastructure Provider : This is the interface between CAPI and your underlying infrastructure. It defines Custom Resources that are specific to your target infrastructure. Eg: CloudStackCluster, CloudStakcMachine, etc 24 | 25 | Now that you have a management cluster, it's time to create your workload clusters! It's these clusters that run on the specific infrastructure and can be exposed to the end-user. 26 | 27 | Have a look at the presentation here: https://github.com/shapeblue/hackerbook/blob/main/primer/capi-capc.pdf 28 | 29 | Also have a look at the documentation [here](https://cluster-api-cloudstack.sigs.k8s.io/) to get started 30 | 31 | ## CloudStack Cluster API Provider (CAPC) 32 | 33 | With that introduction, let's move on to CAPC. CAPC is the CloudStack provider for CAPI. It provides an interface for CAPI to communicate with CloudStack 34 | 35 | It consists of the following main components : 36 | - `/api` : Contains the API definitions of the Custom resources used to create your cluster 37 | - `/config` : Contains the yaml with replacements which later on becomes the CRDs 38 | - `/controllers` : Contains the core of CAPC. It consists of the cluster reconciler and the machine reconciler. More on this later 39 | - `/docs`: Contains the documentation for the project 40 | - `/pkg` : Custom package to abstract the code used to communicate with CloudStack. Uses the cloudstack-go package 41 | - `/tests` : Does pretty much what it says 42 | - `/templates` : Contains the template YAMLs which are used as flavours while generating the workload cluster definition 43 | 44 | Now that you've got a good overview, head on over to the official repo on how to get started! https://github.com/kubernetes-sigs/cluster-api-provider-cloudstack 45 | 46 | ## Internals 47 | 48 | Coming to how it works, CAPI communicates with CloudStack to set up the workload cluster. It does so via `controllers` 49 | There are two main files you need to care about. The `cloudstackcluster_controller.go` and `cloudstackmachine_controller.go` 50 | 51 | ### cloudstackcluster_controller : 52 | 53 | This is file sets up all the resources required to bring up a cluster and cleans up the resources after a cluster has been deleted : 54 | It in turn communicates with other reconcilers to create the necessary resources such as the networks, affinity groups, etc. 55 | 56 | This functionality is handled by the `CloudStackClusterReconciler` in the `Reconcile` method. Based on whether the cluster is created or to be deleted, it takes the appropriate workflow. It runs initially when the cluster is created to set up the required resources. Only when it returns success, does it move on to setting up the nodes of the cluster. 57 | 58 | ### cloudstackmachine_controller : 59 | 60 | This file deals with all the machines or VMs that become worker nodes. That includes : 61 | - The VMs 62 | - Adding control plane nodes to the Load Balancer rules 63 | - Adding VMs to the Port Forwarding rules 64 | 65 | This functionality is handled by the `CloudStackMachineReconciler` in the `Reconcile` method. Based on whether the machine is created or to be deleted, it takes the appropriate workflow. This runs periodically to ensure that the VM is up and a functional node. Based on how health checks are configured, it can delete a node if it is unhealthy and create a new one. It also brings up new nodes and removes old nodes in case of upgrades, scaling, etc. 66 | 67 | ### cloudstackaffinitygroup_controller : 68 | 69 | This manages the required affinity groups for the cluster. Depending on the requirement, it either creates the affinity group, fetches it if already exists and deletes it when no longer required. 70 | 71 | ### cloudstackisolatednetwork_controller : 72 | 73 | This manages the lifecycle of the isolated networks in ACS which power the CAPC cluster. Similar to other controllers, it creates or fetches (if network already exists) and deletes the network if created as part of the cluster. 74 | 75 | 76 | ### /api : 77 | 78 | This directory contains the definitions of the Custom resources. These resources are CloudStack specific and used to internally keep track of the CloudStack resources mapped onto the CAPI definitions. 79 | 80 | The main components are : 81 | 82 | - CloudStackCluster : This is the definition of a CloudStack cluster. It contains the `CloudStackClusterSpec` which contains all the required information to get a cluster up and running such as the zone, network, endpoint, etc. The `CloudStackClusterStatus` keeps track of the CloudStack resources which are created and used in the workload cluster 83 | 84 | - CloudStackMachine : This is the definition of a CloudStack VM. It contains the `CloudStackMachineSpec` which contains all the required information to get a VM up and running such as the template, offering, ssh key, etc. The `CloudStackMachineStatus` keeps track of certain details of the VM used by CAPI 85 | 86 | - Webhooks : These act as controls and run checks to ensure that the specs provided are valid. Examples can be ensuring that the zone, network, etc are not null and that they do not change during the lifecycle of the cluster 87 | 88 | 89 | ### Templates 90 | 91 | CAPI requires that the VMs which become the Cluster nodes have all the necessary kuberntes binaries as well as the default images required baked into them. For creating these VM templates, we use the [image builder](https://github.com/kubernetes-sigs/image-builder) repo. 92 | There are prebuilt images available [here](http://packages.shapeblue.com/cluster-api-provider-cloudstack/images/) 93 | 94 | As of now CAPC supports RHEL 8, Rocky Linux 8 and Ubuntu 20.04 based images 95 | 96 | ## Exercise : 97 | 98 | Follow the development steps mentioned [here](https://cluster-api-cloudstack.sigs.k8s.io/development/index.html) to get started with developing CAPC 99 | Here are a simple exercises to get started : 100 | 101 | 1. When a VM is created in CAPC, ensure that it is created with the suffix '-my-VM'. 102 | > This can get tricky as when the same VM needs to be queried from CAPC, it must match the name in ACS. Ensure that CAPC not only creates the VM with the suffix, but is also able to fetch it as well 103 | 104 | Hint : Go ahead and follow the logical flow of the `cloudstackmachine_controller` (and related code in `pkg/`) and identify where the VM is created and fetched and make the appropriate changes 105 | 106 | 2. Ensure that the offering specified has at least 2 Cores and 2048 MB. 107 | 108 | Hint : This can be checked when the offering is resolved. 109 | 110 | Bigger Hint : Follow the logical flow of the `cloudstackmachine_controller` (and related code `pkg/`) and identify where the offering is fetched and verify the details of the offering, and throw an error if the requirements are not met 111 | 112 | ## Documentation 113 | 114 | - [Kubernetes Cluster API](https://cluster-api.sigs.k8s.io/) 115 | - [Cluster API Provider CloudStack](https://cluster-api-cloudstack.sigs.k8s.io/) 116 | -------------------------------------------------------------------------------- /primer/capi-capc.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/shapeblue/hackerbook/a7e66697bdddff069b1abe668547d6539b9a71bc/primer/capi-capc.pdf -------------------------------------------------------------------------------- /primer/dp.md: -------------------------------------------------------------------------------- 1 | # Design Patterns 2 | 3 | Practice and make yourself familiar with GoFs design patterns. Some popular 4 | patterns in CloudStack: 5 | 6 | | Design Pattern | Usage in CloudStack | 7 | | --- | --- | 8 | | AbstractFactory and Factory | Several abstract classes and building blocks | 9 | | Adapter | Resource and managers | 10 | | Bridge | Frameworks and their provider plugins | 11 | | Builder | Several building blocks | 12 | | Command | RPC to talk to agents | 13 | | Facade | Various subsystems | 14 | | Observer and State | FSM and events | 15 | | Decorator, Visitor and Flyweight | Network [programming](https://cwiki.apache.org/confluence/display/CLOUDSTACK/Refactoring+Redundant+Virtual+Router+Implementation) | 16 | | Singleton | Managers and Providers instantiated as Spring beans | 17 | | Strategy | Plugins, algorithms, Service Layers | 18 | 19 | Example Design Patterns implementations in Java: 20 | - https://github.com/rhtyd/hacklab/tree/master/designpatterns/java 21 | 22 | ## Orchestration Patterns 23 | 24 | Recommended reading list (from one of the original CloudStack architects): 25 | - Idempotent operations: https://cloudierthanthou.wordpress.com/2017/02/22/design-patterns-in-orchestrators-part-1/ 26 | - Southbound APIs: https://cloudierthanthou.wordpress.com/2017/02/23/design-patterns-in-orchestrators-part-2/ 27 | - Transfer of desired state: https://cloudierthanthou.wordpress.com/2017/06/22/design-patterns-in-orchestrators-transfer-of-desired-state-part-3n/ 28 | - Network and firewall management: 29 | - https://cloudierthanthou.wordpress.com/2015/04/09/how-to-manage-a-million-firewalls-part-1/ 30 | - https://cloudierthanthou.wordpress.com/2015/04/10/how-to-manage-a-million-firewalls-part-2/ 31 | -------------------------------------------------------------------------------- /primer/dsys.md: -------------------------------------------------------------------------------- 1 | # Distributed Systems Primer 2 | 3 | Topics: 4 | - Scalability, Availability 5 | - Latency, fault tolerance, performance 6 | - Partition and replication 7 | - Client-Server, agent based 8 | - CAP, Time and Order 9 | - Rebalancing, claim ownership 10 | - Reconciliation, eventual consistency 11 | 12 | References and readings: 13 | - Distributed Systems for Fun and Profit: http://book.mixu.net/distsys/single-page.html 14 | - System Design Primer: https://github.com/donnemartin/system-design-primer/blob/master/README.md 15 | - Consistency models: https://jepsen.io/consistency 16 | - Jepsen Analyses: https://jepsen.io/analyses 17 | - CAP and network partition: https://drive.google.com/file/d/15nxAaVXZwNFnJNVvgtKonNbzxNgTUCxP/view 18 | - Further readings: 19 | - http://henryr.github.io/distributed-systems-readings/ 20 | - http://christophermeiklejohn.com/distributed/systems/2013/07/12/readings-in-distributed-systems.html 21 | - High Scalability Blog: http://highscalability.com 22 | - AOSA: http://aosabook.org/en/index.html 23 | - DIY OS: https://github.com/danistefanovic/build-your-own-x/blob/master/README.md#build-your-own-operating-system 24 | -------------------------------------------------------------------------------- /primer/efficiency.md: -------------------------------------------------------------------------------- 1 | # Developer Efficiency 2 | 3 | ## Fast CloudStack Build 4 | 5 | Consider following maven strategies: 6 | 7 | - Use `-T {number of max.cores}` to build codebase in parallel. 8 | - If you're confident, use `-DskipTests=true` to skip unit tests during building. 9 | - Use `-Dnoredist` only when you need to build/test/something against VMware and 10 | other non-oss plugins. 11 | - Use incremental builds using -pl ,client. Always include client as it is where all the 13 | jars are put together to create the cloudstack management far jar. 14 | - For UI changes, edit relevant files in the `client/target/classes/META-INF/` 15 | path and refresh browser. When done, copy and commit the changes. 16 | -------------------------------------------------------------------------------- /primer/git.md: -------------------------------------------------------------------------------- 1 | # Git Version Control System 2 | 3 | ## Configuration 4 | 5 | Syntax: 6 | 7 | git config --global 8 | 9 | Frequently used: 10 | 11 | git config --global user.name "[name]" 12 | git config --global user.email "[email address]" 13 | git config --global color.ui auto 14 | 15 | ## Repositories 16 | 17 | Create Repo: 18 | 19 | mkdir project && cd project 20 | git init 21 | 22 | Clone repo: 23 | 24 | git clone 25 | 26 | ## Changes 27 | 28 | git rm 29 | git rm --cached 30 | git mv 31 | 32 | git status 33 | 34 | git diff 35 | git diff --staged 36 | 37 | git add 38 | 39 | git reset 40 | git reset --hard 41 | 42 | git commit -s -m "some message" 43 | 44 | ## Branching and Synchronization 45 | 46 | git branch 47 | 48 | git branch 49 | 50 | git branch -d 51 | 52 | git checkout 53 | 54 | git merge 55 | 56 | git fetch 57 | 58 | git push 59 | 60 | git pull 61 | 62 | ## Forward Merging 63 | 64 | # Checkout base branch and get the changes 65 | git checkout 66 | git pull --rebase origin 67 | # Checkout the next branch and update it before forward merging changes 68 | git checkout 69 | git pull --rebase origin 70 | git merge origin/ 71 | # Fix conflicts, merge/commit changes and then push the changes 72 | git push origin 73 | 74 | ## Stashing 75 | 76 | git stash 77 | git stash list 78 | git stash 79 | git stash drop 80 | 81 | ## History 82 | 83 | git log 84 | 85 | git log --follow 86 | 87 | git diff ... 88 | 89 | git show 90 | 91 | ## Searching 92 | 93 | git grep 94 | git grep -i 95 | 96 | ## Squashing Changes 97 | 98 | To squash all the changes in your branch against a target branch say `master` 99 | you can do the following: 100 | 101 | git checkout your-branch 102 | git reset $(git merge-base master your-branch) 103 | git add -A 104 | git commit -s # add pretty commit message and description 105 | 106 | ## Rebasing Branch 107 | 108 | This is the most common and challenging task that requires say a PR branch to 109 | be rebased against a new/different branch than say `master`: 110 | 111 | 0. Easiest way when commits are melded: 112 | 113 | # Note the SHA of the commit on your PR branch 114 | git log 115 | # Reset the PR branch against the new branch 116 | git reset --hard origin/4.11 117 | # Cherry-pick and fix conflicts 118 | git cherry-pick 119 | # Fix conflicts, add/commit it 120 | git commit --amend -s 121 | 122 | 1. When new branch is forward branch (say from 4.11 to master): 123 | 124 | 2. When new branch is a backward (say from master to 4.11): 125 | 126 | 3. General solution: (say to 4.11) 127 | 128 | git rebase origin/4.11 129 | -------------------------------------------------------------------------------- /primer/index.md: -------------------------------------------------------------------------------- 1 | # Appendix: Primers 2 | 3 | | # | Tool | 4 | | - | ---- | 5 | | 0 | [Git](git.md) | 6 | | 1 | [Design Patterns](dp.md) | 7 | | 2 | [Distributed Systems](dsys.md) | 8 | | 3 | [Developer Efficiency](efficiency.md) | 9 | | 4 | [Cluster API Provider CloudStack](capc.md) | 10 | -------------------------------------------------------------------------------- /primer/networking-refresh.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/shapeblue/hackerbook/a7e66697bdddff069b1abe668547d6539b9a71bc/primer/networking-refresh.pdf --------------------------------------------------------------------------------