├── .gitignore ├── .gitmodules └── README.md /.gitignore: -------------------------------------------------------------------------------- 1 | out/ 2 | out-br/ 3 | toolchains/ 4 | *.diff 5 | *.dump 6 | *.patch 7 | *bake/ 8 | *.[0-9] 9 | -------------------------------------------------------------------------------- /.gitmodules: -------------------------------------------------------------------------------- 1 | [submodule "qemu"] 2 | path = qemu 3 | url = git@github.com:TwinVisor/qemu.git 4 | [submodule "guest-linux"] 5 | path = guest-linux 6 | url = git@github.com:TwinVisor/guest-linux.git 7 | [submodule "edk2"] 8 | path = edk2 9 | url = https://github.com/tianocore/edk2.git 10 | [submodule "edk2-platforms"] 11 | path = edk2-platforms 12 | url = https://github.com/tianocore/edk2-platforms.git 13 | [submodule "mbedtls"] 14 | path = mbedtls 15 | url = https://github.com/ARMmbed/mbedtls.git 16 | [submodule "buildroot"] 17 | path = buildroot 18 | url = https://github.com/buildroot/buildroot.git 19 | [submodule "grub"] 20 | path = grub 21 | url = https://git.savannah.gnu.org/git/grub.git 22 | [submodule "trusted-firmware-a"] 23 | path = trusted-firmware-a 24 | url = git@github.com:TwinVisor/trusted-firmware-a.git 25 | [submodule "linux"] 26 | path = linux 27 | url = git@github.com:TwinVisor/linux.git 28 | [submodule "build"] 29 | path = build 30 | url = git@github.com:TwinVisor/build.git 31 | [submodule "s-visor"] 32 | path = s-visor 33 | url = git@github.com:TwinVisor/s-visor.git 34 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # TwinVisor Functional Prototype 2 | 3 | ## Introduction 4 | 5 | This is a functional prototype of the SOSP'21 paper **TwinVisor: Hardware-isolated Confidential Virtual Machines for ARM** based on ARM Fixed Virtual Platform (FVP). You can find our paper [here](https://dl.acm.org/doi/abs/10.1145/3477132.3483554). 6 | 7 | Due to the confidentiality agreement, we cannot release the proprietary code of the performance testing prototype on Hisilicon Kirin 990 hardware. 8 | 9 | The copyright of TwinVisor and its prototype belongs to Institute of Parallel and Distributed Systems (IPADS) from Shanghai Jiao Tong University (SJTU). 10 | 11 | The code is based on [OP-TEE project](https://github.com/OP-TEE/manifest/blob/master/fvp.xml) and tested on Ubuntu 20.04 & Ubuntu 18.04 (x86-64). 12 | 13 | If you encounter any questions, please contact us via e-mail: 14 | 15 | > Dingji Li: dj_lee@sjtu.edu.cn, Zeyu Mi: yzmizeyu@sjtu.edu.cn 16 | 17 | ## Prerequesites 18 | 19 | 1. Update the package managers database. 20 | 21 | ```bash 22 | sudo apt-get update 23 | ``` 24 | 25 | 2. Install the following packages. 26 | 27 | ```bash 28 | sudo apt-get install android-tools-adb android-tools-fastboot autoconf \ 29 | automake bc bison build-essential cmake ccache codespell \ 30 | cscope curl device-tree-compiler \ 31 | expect flex ftp-upload gdisk iasl libattr1-dev libcap-dev \ 32 | libfdt-dev libftdi-dev libglib2.0-dev libgmp-dev libhidapi-dev \ 33 | libmpc-dev libncurses5-dev libpixman-1-dev libssl-dev libtool make \ 34 | mtools netcat ninja-build python-crypto python3-crypto python-pyelftools \ 35 | python3-pycryptodome python3-pyelftools python3-serial python \ 36 | rsync unzip uuid-dev xdg-utils xterm xz-utils zlib1g-dev 37 | ``` 38 | 39 | ## Setup 40 | 41 | Clone this repo and update submodules. 42 | 43 | ```bash 44 | export TV_ROOT=$(pwd) 45 | git submodule update --init --recursive 46 | ``` 47 | 48 | Download [FVP Base](https://ipads.se.sjtu.edu.cn:1313/f/73e2572b19a24b32817c/?dl=1) and extract to ``FVP_Base_RevC-2xAEMv8A`` in ``$TV_ROOT`` directory. 49 | 50 | ```bash 51 | # MD5: 0bd25ec5005c600d6f9b8ebc41aff0ab FVP_Base_RevC-2xAEMv8A.tar.gz 52 | wget -c https://ipads.se.sjtu.edu.cn:1313/f/73e2572b19a24b32817c/?dl=1 -O $TV_ROOT/FVP_Base_RevC-2xAEMv8A.tar.gz 53 | 54 | tar xzf $TV_ROOT/FVP_Base_RevC-2xAEMv8A.tar.gz -C $TV_ROOT/ 55 | ``` 56 | 57 | **NOTE:** Following steps should be done in the ``build`` directory. 58 | 59 | ```bash 60 | cd $TV_ROOT/build 61 | ``` 62 | 63 | First, get the toolchains for aarch64. 64 | 65 | ```bash 66 | make toolchains -j2 67 | 68 | # Test toolchains 69 | ls ../toolchains 70 | ``` 71 | 72 | Next, download the disk image of the prototype. For simplicity, this disk image already contains QEMU, kernel images and rootfs for S-VMs. You can also compile the source code of QEMU and guest kernel, and copy them into the disk image by yourself. 73 | 74 | ```bash 75 | mkdir -p ../out 76 | # The tarball is about 8GB, and the disk image after decompressed is about 40GB 77 | # MD5: c16d78505fa16a8520ee08a05d1debf7 boot.tar.gz 78 | wget -c https://ipads.se.sjtu.edu.cn:1313/f/73350e5ff3e440a98081/?dl=1 -O ../out/boot.tar.gz 79 | 80 | tar xzf ../out/boot.tar.gz -C ../out 81 | 82 | # Test boot.img and the guest kernel image 83 | ls ../out/boot.img 84 | ls ../out/Image # for md5 integrity checking 85 | ``` 86 | 87 | Then, compile all of them. 88 | 89 | ```bash 90 | make all -j$(nproc) 91 | ``` 92 | 93 | Finally, run it. 94 | 95 | ```bash 96 | make run-only 97 | ``` 98 | 99 | ## Run 100 | 101 | 1. Login the host Linux (N-visor) with username ``root`` 102 | 103 | 2. Mount the rootfs and chroot in host Linux 104 | 105 | ```bash 106 | mount /dev/vda2 /root 107 | chroot /root bash 108 | ./init.sh 109 | ``` 110 | 111 | 3. Run the guest Linux (S-VM) 112 | 113 | ```bash 114 | cd /test 115 | ./s-vm0.sh 116 | ``` 117 | 118 | 4. Mount the rootfs and chroot in guest Linux 119 | 120 | ```bash 121 | mount /dev/vda /root 122 | chroot /root bash 123 | ``` 124 | 125 | 5. Some workload examples are under ``/test/`` in guest rootfs 126 | 127 | ```bash 128 | # Take FileIO as an example 129 | ./fileio.sh 130 | ``` 131 | 132 | ## Guest QEMU 133 | 134 | To build QEMU for S-VMs: 135 | 136 | ```bash 137 | # For cross compile, take ubuntu **focal** as an example 138 | dpkg --add-architecture arm64 139 | echo "deb [arch=arm64] http://ports.ubuntu.com/ubuntu-ports/ focal main multiverse restricted universe" >> /etc/apt/sources.list 140 | echo "deb [arch=arm64] http://ports.ubuntu.com/ubuntu-ports/ focal-updates main multiverse restricted universe" >> /etc/apt/sources.list 141 | apt update 142 | apt install crossbuild-essential-arm64 pkg-config-aarch64-linux-gnu 143 | apt install libpixman-1-dev:arm64 libglib2.0-dev:arm64 144 | 145 | mkdir -p $TV_ROOT/qemu/build 146 | cd $TV_ROOT/qemu/build 147 | 148 | # Cross compile with static link 149 | ../configure --target-list=aarch64-softmmu \ 150 | --cross-prefix=aarch64-linux-gnu- \ 151 | --static 152 | make -j$(nproc) 153 | 154 | # Test output 155 | ../aarch64-softmmu/qemu-system-aarch64 --version 156 | 157 | # Copy the QEMU into the disk image ``$TV_ROOT/out/boot.img`` 158 | ``` 159 | 160 | ## Guest Linux 161 | 162 | To build guest kernel image: 163 | 164 | ```bash 165 | # For cross compile 166 | export ARCH=arm64 167 | export CROSS_COMPILE=aarch64-linux-gnu- 168 | 169 | cd $TV_ROOT/guest-linux 170 | cp prototype-config .config 171 | make all -j$(nproc) 172 | 173 | # Test output 174 | ls arch/arm64/boot/Image 175 | 176 | # Copy the kernel image to ``$TV_ROOT/out/Image``, 177 | # then re-compile the S-visor under ``$TV_ROOT/build/`` with ``make s-visor`` 178 | cp arch/arm64/boot/Image $TV_ROOT/out/ 179 | # Also copy the kernel image into the disk image ``$TV_ROOT/out/boot.img`` 180 | ``` 181 | --------------------------------------------------------------------------------