└── README.md /README.md: -------------------------------------------------------------------------------- 1 | # ELISA: Exit-Less, Isolated, and Shared Access 2 | 3 | This document complements the paper "[Exit-Less, Isolated, and Shared Access for Virtual Machines](https://doi.org/10.1145/3582016.3582042)" presented at the International Conference on Architectural Support for Programming Languages and Operating Systems in March 2023 (ASPLOS'23). 4 | 5 | ## WARNING 6 | 7 | - **The authors do not guarantee perfect security of the ELISA-relevant prototype implementations.** 8 | - **The ELISA-relevant prototype implementations can break your entire system.** 9 | - **The authors will not bear any responsibility if the implementations, provided by the authors, cause any problems.** 10 | 11 | ## Table of contents 12 | 13 | - [Materials](#materials) 14 | - [List of the ELISA prototype repositories](#list-of-the-elisa-prototype-repositories) 15 | - [Requirements](#requirements) 16 | - [Setup](#setup) 17 | - [Trying an example application : elisa-app-nop](#trying-an-example-application--elisa-app-nop) 18 | - [Commentary](#commentary) 19 | - [Section 4.1 : Anywhere Page Table (APT)](#section-41--anywhere-page-table-apt) 20 | - [Section 4.2 : Gate EPT Context](#section-42--gate-ept-context) 21 | - [Section 5.1 : ELISA-Specific Hypercalls](#section-51--elisa-specific-hypercalls) 22 | - [Section 5.2 : libelisa](#section-52--libelisa) 23 | - [Section 5.3 : Negotiation Steps](#section-53--negotiation-steps) 24 | - [Section 5.4 : Code for the Sub EPT Context](#section-54--code-for-the-sub-ept-context) 25 | - [Section 5.5 : Shared Memory Management](#section-55--shared-memory-management) 26 | - [Section 5.6 : Interrupt Setting](#section-56--interrupt-setting) 27 | - [Section 6.1 : Context Switch Overhead](#section-61--context-switch-overhead) 28 | - [Section 7.1 : VM Networking System](#section-71--vm-networking-system) 29 | 30 | ## Materials 31 | 32 | - Paper: [https://doi.org/10.1145/3582016.3582042](https://doi.org/10.1145/3582016.3582042) 33 | - Slides: [https://yasukata.github.io/presentation/2023/03/asplos2023/asplosc23main-p814-slides.pdf](https://yasukata.github.io/presentation/2023/03/asplos2023/asplosc23main-p814-slides.pdf) 34 | - Lightning talk slides: [https://yasukata.github.io/presentation/2023/03/asplos2023/asplosc23main-p814-slides-LT.pdf](https://yasukata.github.io/presentation/2023/03/asplos2023/asplosc23main-p814-slides-LT.pdf) 35 | - Lightning talk video: [https://www.youtube.com/watch?v=oLatL6TIIa4](https://www.youtube.com/watch?v=oLatL6TIIa4) 36 | - Poster: [https://yasukata.github.io/presentation/2023/03/asplos2023/asplosc23main-p814-poster.pdf](https://yasukata.github.io/presentation/2023/03/asplos2023/asplosc23main-p814-poster.pdf) 37 | 38 | ## List of the ELISA prototype repositories 39 | 40 | ### Core components 41 | 42 | - [libelisa](https://github.com/yasukata/libelisa): the core library for ELISA 43 | - [libelisa-extra](https://github.com/yasukata/libelisa-extra): supplemental utilities of libelisa 44 | - [kvm-elisa](https://github.com/yasukata/kvm-elisa): KVM modification for ELISA 45 | 46 | ### Utilities 47 | 48 | - [elisa-util-exec](https://github.com/yasukata/elisa-util-exec): a simple ELISA application launcher 49 | 50 | ### Applications 51 | 52 | - [elisa-app-nop](https://github.com/yasukata/elisa-app-nop): a minimal ELISA application 53 | - [elisa-app-vmnet](https://github.com/yasukata/elisa-app-vmnet): an ELISA-based VM networking system 54 | - [rvs](https://github.com/yasukata/rvs): a virtual switch implementation employed for the ELISA-based VM networking system 55 | - [librte_pmd_rvif](https://github.com/yasukata/librte_pmd_rvif): a DPDK poll mode driver for rvif that is the vNIC of rvs 56 | 57 | ## Requirements 58 | 59 | The ELISA prototype implementations assume the following platform. 60 | 61 | - CPU: VMFUNC-capable Intel CPU 62 | - OS: Linux 63 | - Hypervisor: QEMU 64 | 65 | The authors use Ubuntu 20.04/22.04, and recommend using Ubuntu for a relatively easy setup. 66 | 67 | If you do not have QEMU on your machine, you can install it by the following command (on Ubuntu). 68 | 69 | ``` 70 | sudo apt install qemu-system-x86 71 | ``` 72 | 73 | ## Setup 74 | 75 | **As mentioned in [the WARNING section](#warning), the ELISA prototype implementations can break your entire system. Therefore, we strongly recommend trying the ELISA prototype implementations on a computer that is OK to get damaged.** 76 | 77 | ### General information 78 | 79 | To use ELISA, we need to install a modified KVM kernel module. 80 | 81 | Please find the ELISA patch for KVM [here](https://github.com/yasukata/kvm-elisa/blob/0fadc257ca8a99365ba8db09d03eed431881cdd8/elisa.patch); this patch is generated for linux-source-5.15.0 which is distributed by the apt repository of Ubuntu. 82 | 83 | We are primarily assuming this patch for KVM is applied **by hand** along with manual source code changes for a Linux version installed in the user's environment; this is because: 84 | - the definitions of data structures and functions, used in the original KVM implementation, are changed quite often, and minor version changes lead to incompatibility between the KVM module and the Linux kernel, therefore, we thought it is difficult for a single repository to cover a large set of Linux versions. 85 | - we do not want developers, who think of trying ELISA, to compile the full Linux kernel code base just for ELISA because kernel compilation is troublesome; we assume to use a Linux kernel provided by a distributor (e.g., Ubuntu apt repository). 86 | 87 | Patching by hand sounds complicated, but we believe it is not too much because we have minimized the size of the patch: the code that needs to be added is just around 120 lines of a single code block, and a single line has to be replaced. 88 | 89 | The goal of the patch is to add the hypercalls, newly implemented for ELISA. 90 | 91 | - [line 137 ~ 138](https://github.com/yasukata/kvm-elisa/blob/0fadc257ca8a99365ba8db09d03eed431881cdd8/elisa.patch#L137-L138) applies the newly implemented hypercall handler as the callback function for VMCALL. 92 | - [line 8 ~ 128](https://github.com/yasukata/kvm-elisa/blob/0fadc257ca8a99365ba8db09d03eed431881cdd8/elisa.patch#L8-L128) is the hypercall implementation. 93 | 94 | ### Example on Ubuntu 22.04 with Linux 5.15 95 | 96 | While ELISA does not depend on a specific Linux kernel version, we do not want to compile the full Linux kernel source code because it is troublesome. 97 | 98 | This example shows how to use ELISA on Ubuntu 22.04 without fully replacing the Linux kernel provided by Ubuntu. 99 | 100 | #### 1. download the Linux source code 101 | 102 | First, please download the Linux kernel source code by the following command; you will get it at ```/usr/src/linux-source-5.15.0.tar.bz2```. 103 | 104 | ``` 105 | sudo apt install linux-source-5.15.0 106 | ``` 107 | 108 | #### 2. install the Linux kernel for the downloaded source 109 | 110 | To install a self-compiled KVM module, the source code of it has to be compatible with the Linux kernel running on the machine, and the versions of the Linux source code and the running Linux kernel have to be the same or very close; otherwise, a compilation of the KVM source code often fails because of undefined symbols. 111 | 112 | To cope with this issue, let's first check which version of the Linux source is installed in the previous step [1. download the Linux source code](1-download-the-linux-source-code). 113 | 114 | ``` 115 | $ sudo apt install linux-source-5.15.0 116 | Reading package lists... Done 117 | Building dependency tree... Done 118 | Reading state information... Done 119 | linux-source-5.15.0 is already the newest version (5.15.0-67.74). 120 | ``` 121 | 122 | As seen above, in this case, we have the Linux source for 5.15.0-67. (**in the following steps, please change the version number according to the linux-source version you got on your environment.**) 123 | 124 | To minimize the version mismatch between the source code and the installed Linux kernel, we install Linux 5.15.0-67 (offered by Ubuntu) by the following command. After finishing the following procedures, you need to reboot (and you may need to explicitly specify it in the GRUB menu) to load Linux 5.15.0-67. 125 | 126 | ``` 127 | export KVER=5.15.0-67-generic; sudo apt install linux-image-$KVER linux-modules-$KVER linux-modules-extra-$KVER linux-headers-$KVER 128 | ``` 129 | 130 | After reboot, you can confirm Linux 5.15.0-67 is properly loaded by the ```uname``` command. 131 | 132 | ``` 133 | $ uname -a 134 | Linux ubuntu 5.15.0-67-generic #74-Ubuntu SMP Wed Feb 22 14:14:39 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux 135 | ``` 136 | 137 | #### 3. extract the KVM module files 138 | 139 | Then, let's make a new working directory; this time, we name it ```./kvm-elisa-patch```. 140 | 141 | ``` 142 | mkdir kvm-elisa-patch 143 | ``` 144 | 145 | Please extract the downloaded Linux source in ```./kvm-elisa-patch``` by the following command. 146 | 147 | ``` 148 | tar xvf /usr/src/linux-source-5.15.0.tar.bz2 -C ./kvm-elisa-patch 149 | ``` 150 | 151 | For easy source code maintenance, let's copy KVM-relevant files to ```./kvm-elisa-patch/kvm```. (It is also fine to directly use the source in ```./kvm-elisa-patch/linux-source-5.15.0```) 152 | 153 | ``` 154 | mkdir -p kvm-elisa-patch/kvm/virt 155 | ``` 156 | 157 | ``` 158 | mkdir -p kvm-elisa-patch/kvm/arch/x86 159 | ``` 160 | 161 | ``` 162 | cp -r kvm-elisa-patch/linux-source-5.15.0/virt/kvm kvm-elisa-patch/kvm/virt 163 | ``` 164 | 165 | ``` 166 | cp -r kvm-elisa-patch/linux-source-5.15.0/arch/x86/kvm kvm-elisa-patch/kvm/arch/x86 167 | ``` 168 | 169 | Now, we have the default KVM implementation in ```./kvm-elisa-patch/kvm```. 170 | 171 | It may be a good time to save the status of ```./kvm-elisa-patch/kvm``` by git so that we can easily roll back to the default KVM implementation. If you have no preference for gitting, you can use the following commands. 172 | 173 | ``` 174 | cd kvm-elisa-patch/kvm 175 | ``` 176 | 177 | ``` 178 | git init 179 | ``` 180 | 181 | ``` 182 | git add . 183 | ``` 184 | 185 | ``` 186 | git commit -m "first commit" 187 | ``` 188 | 189 | #### 4. test compilation for the default KVM module 190 | 191 | Before starting to apply the ELISA patch, let's confirm the KVM source code can be properly compiled. 192 | 193 | Please save the following as ```./kvm-elisa-patch/kvm/arch/x86/kvm/MyMakefile```. 194 | 195 | ```Makefile 196 | KDIR := /lib/modules/$(shell uname -r)/build 197 | 198 | srctree := $(dir $(abspath $(lastword $(MAKEFILE_LIST))))../../.. 199 | 200 | ccflags-y += -I$(srctree)/arch/x86 # for trace-relevant header 201 | 202 | include $(dir $(abspath $(lastword $(MAKEFILE_LIST))))Makefile 203 | 204 | EXTRA_CFLAGS += $(ccflags-y) 205 | 206 | SUBDIRS := $(dir $(abspath $(lastword $(MAKEFILE_LIST)))) 207 | COMMON_OPS = -C $(KDIR) M='$(SUBDIRS)' EXTRA_CFLAGS='$(EXTRA_CFLAGS)' 208 | 209 | CLEANITEMS := *.ur-safe .cache.mk *.o *.ko *.cmd *.mod.c *.mod .*.cmd .tmp_versions modules.order Module.symvers *~ 210 | CLEANOBJS = $(addprefix $(dir $(abspath $(lastword $(MAKEFILE_LIST)))),$(CLEANITEMS)) \ 211 | $(addprefix $(dir $(abspath $(lastword $(MAKEFILE_LIST))))mmu/,$(CLEANITEMS)) \ 212 | $(addprefix $(dir $(abspath $(lastword $(MAKEFILE_LIST))))vmx/,$(CLEANITEMS)) \ 213 | $(addprefix $(dir $(abspath $(lastword $(MAKEFILE_LIST))))svm/,$(CLEANITEMS)) \ 214 | $(addprefix $(dir $(abspath $(lastword $(MAKEFILE_LIST))))../../../virt/kvm/,$(CLEANITEMS)) \ 215 | 216 | default: 217 | $(MAKE) $(COMMON_OPS) modules 218 | 219 | clean: 220 | @rm -f $(CLEANOBJS) 221 | ``` 222 | 223 | Then, please enter ```./kvm-elisa-patch/kvm/arch/x86/kvm```. 224 | 225 | ``` 226 | cd kvm-elisa-patch/kvm/arch/x86/kvm 227 | ``` 228 | 229 | Please type the following command to compile the KVM module. 230 | 231 | ``` 232 | make -f MyMakefile 233 | ``` 234 | 235 | If the compilation fails here and the compiler complains, for example, something is not declared, it would be because of the version mismatch between the source code and the Linux kernel you are currently using; in this case, please look back [2. install the Linux kernel for the downloaded source](#2-install-the-linux-kernel-for-the-downloaded-source). 236 | 237 | #### 5. install the KVM module for testing 238 | 239 | Once you could compile the KVM module, you can install it by the following commands. 240 | 241 | First, uninstall the current kvm_intel (vmx) kernel module. 242 | 243 | ``` 244 | sudo rmmod kvm_intel 245 | ``` 246 | 247 | Afterward, uninstall the current kvm kernel module. 248 | 249 | ``` 250 | sudo rmmod kvm 251 | ``` 252 | 253 | Then, install the newly compiled kvm kernel module. 254 | 255 | ``` 256 | sudo insmod ./kvm.ko 257 | ``` 258 | 259 | Finally, install the modified kvm_intel (vmx) kernel module. 260 | 261 | ``` 262 | sudo insmod ./kvm-intel.ko 263 | ``` 264 | 265 | Up to here, if you do not encounter any issue, it means that the installation of the default KVM module works properly, and if you find a problem after proceeding with the subsequent steps, it means that the ELISA patch adaptation causes the problem. 266 | 267 | #### 6. apply the ELISA patch for KVM 268 | 269 | Finally, let's start applying the patch for ELISA to KVM. 270 | 271 | Here, we assume that we are in ```./kvm-elisa-patch```. 272 | 273 | Please download the ELISA patch for KVM. 274 | 275 | ``` 276 | wget https://raw.githubusercontent.com/yasukata/kvm-elisa/0fadc257ca8a99365ba8db09d03eed431881cdd8/elisa.patch 277 | ``` 278 | 279 | First, please try to apply it by the following git command; if you are lucky (maybe using Linux 5.15), the command above successfully applies the ELISA patch to the KVM module implementation, otherwise, you need to modify the KVM source code manually. 280 | 281 | ``` 282 | git -C kvm apply ../elisa.patch 283 | ``` 284 | 285 | If the patch adaptation by the command above fails, please edit ```kvm-elisa-patch/kvm/arch/x86/kvm/vmx/vmx.c``` by yourself to: 286 | 287 | - set [```handle_vmx_hypercall```](https://github.com/yasukata/kvm-elisa/blob/0fadc257ca8a99365ba8db09d03eed431881cdd8/elisa.patch#L113) to the VMCALL handler (done in [line 137 ~ 138 in the patch](https://github.com/yasukata/kvm-elisa/blob/0fadc257ca8a99365ba8db09d03eed431881cdd8/elisa.patch#L137-L138)). 288 | - add the code block which is [line 8 ~ 128 in the patch](https://github.com/yasukata/kvm-elisa/blob/0fadc257ca8a99365ba8db09d03eed431881cdd8/elisa.patch#L8-L128); it may be better to put it just above the part setting ```handle_vmx_hypercall``` to the VMCALL handler. 289 | 290 | Afterward, please try to compile the KVM module that includes the ELISA patch by the following commands. 291 | 292 | ``` 293 | cd kvm/arch/x86/kvm 294 | ``` 295 | 296 | ``` 297 | make -f MyMakefile 298 | ``` 299 | 300 | If the compilation by the command above fails, it would be because the definition of the data structures and functions, used in the ELISA patch, would be different from the ones defined in the Linux kernel version that you use. 301 | 302 | In this case, please change the definitions in the ELISA patch accordingly by your hand; hopefully, it is not too complicated because the size of the ELISA patch is not too big. 303 | 304 | #### 7. install the modified KVM module 305 | 306 | After you could compile the modified KVM module, you can install it using the same commands shown in [5. install the KVM module for testing](#5-install-the-kvm-module-for-testing). 307 | 308 | ## Trying an example application : elisa-app-nop 309 | 310 | The [elisa-app-nop](https://github.com/yasukata/elisa-app-nop) application would be a good starting point. 311 | 312 | **As mentioned in [the WARNING section](#warning), the ELISA prototype implementations can break your entire system. Therefore, we strongly recommend trying the ELISA prototype implementations on a computer that is OK to get damaged.** 313 | 314 | ### Preparation for running an ELISA-based application 315 | 316 | The example application runs on VMs that are hosted by the host installing [the KVM module including the ELISA patch](#setup). 317 | 318 | Please prepare at least one VM image that QEMU can run, in a manner you wish (e.g., virt-manager); we also recommend using Ubuntu for the VM. 319 | 320 | ### Download the source code needed for elisa-app-nop 321 | 322 | Please type the following to download the source files; it is recommended to download and build the them on a VM which executes them. 323 | 324 | ``` 325 | git clone https://github.com/yasukata/elisa-app-nop.git 326 | ``` 327 | 328 | Then, please enter ```./elisa-app-nop```. 329 | 330 | ``` 331 | cd ./elisa-app-nop 332 | ``` 333 | 334 | First, please create directories in ```./elisa-app-nop``` to place the core library and utility for ELISA. 335 | 336 | ``` 337 | mkdir -p deps/elisa/lib 338 | ``` 339 | 340 | ``` 341 | mkdir -p deps/elisa/util 342 | ``` 343 | 344 | Please type the following commands to download the source code of libelisa, libelisa-extra, and elisa-util-exec. 345 | 346 | ``` 347 | git -C deps/elisa/lib clone https://github.com/yasukata/libelisa.git 348 | ``` 349 | 350 | ``` 351 | git -C deps/elisa/lib clone https://github.com/yasukata/libelisa-extra.git 352 | ``` 353 | 354 | ``` 355 | git -C deps/elisa/util clone https://github.com/yasukata/elisa-util-exec.git 356 | ``` 357 | 358 | Now, we have all assets needed to build the elisa-app-nop application. 359 | 360 | ### Compilation of elisa-app-nop 361 | 362 | Let's start compilation. 363 | 364 | The following produces ```./deps/elisa/lib/libelisa/libelisa.a```. 365 | 366 | ``` 367 | make -C deps/elisa/lib/libelisa 368 | ``` 369 | 370 | The following generates ```./deps/elisa/util/elisa-util-exec/a.out```. 371 | 372 | ``` 373 | make -C deps/elisa/util/elisa-util-exec 374 | ``` 375 | 376 | The following command generates ```./lib/libelisa-applib-nop/lib.so```. 377 | 378 | ``` 379 | make -C lib/libelisa-applib-nop 380 | ``` 381 | 382 | The following command generates ```./libelisa-app-nop.so```. 383 | 384 | ``` 385 | make 386 | ``` 387 | 388 | ### Run elisa-app-nop 389 | 390 | #### Note on the simplified example for elisa-app-nop 391 | 392 | - For the following example commands, to simplify the testing, we assume to use a single VM as the manager VM and the guest VM, meaning that the program made for the manager VM and the program for guest VMs run on the same VM. 393 | - If you have multiple VMs and wish to use one as the manager VM and the others as the guest VMs, please change the IP address passed to ```./deps/elisa/util/elisa-util-exec/a.out``` executed on a guest VM through the ```-s``` option, to specify the IP address of the manager VM. No change is needed for the command to be executed on the manager VM. 394 | 395 | #### The command for the manager VM 396 | 397 | The following command is for the manager VM. **WARNING: Do not terminate the process executing the following command while an ELISA-based application runs on a guest VM. The memory for the gate/sub EPT contexts of a guest VM is allocated from the process running the command on the manager VM; if you stop it, the memory for the gate/sub EPT contexts of the guest VM is automatically released and it results in a system failure because the guest VM accesses the released memory.** 398 | 399 | ``` 400 | sudo ELISA_APPLIB_FILE=./lib/libelisa-applib-nop/lib.so ./deps/elisa/util/elisa-util-exec/a.out -f ./libelisa-app-nop.so -p 10000 401 | ``` 402 | 403 | #### The command for a guest VM 404 | 405 | The following command is for a guest VM. 406 | 407 | ``` 408 | taskset -c 0 ./deps/elisa/util/elisa-util-exec/a.out -f ./libelisa-app-nop.so -p 10000 -s 127.0.0.1 409 | ``` 410 | 411 | #### Example output of elisa-app-nop 412 | 413 | The following is the example output seen on the guest VM. 414 | 415 | ``` 416 | $ taskset -c 0 ./deps/elisa/util/elisa-util-exec/a.out -f ./libelisa-app-nop.so -p 10000 -s 127.0.0.1 417 | enter sub EPT context and get a return value 418 | return value is 1 419 | ``` 420 | 421 | ### Explanation on elisa-app-nop 422 | 423 | Up to here, we have quickly run the elisa-app-nop application. 424 | 425 | Here, we explain what happened in the steps above. 426 | 427 | #### Binary files 428 | 429 | We have generated four binary files. 430 | 431 | ##### ./deps/elisa/lib/libelisa/libelisa.a 432 | 433 | ```./deps/elisa/lib/libelisa/libelisa.a``` is involved by ```./deps/elisa/util/elisa-util-exec/a.out``` and ```./libelisa-app-nop.so```. 434 | 435 | ##### ./lib/libelisa-applib-nop/lib.so 436 | 437 | ```./lib/libelisa-applib-nop/lib.so``` contains the code executed in the sub EPT context. 438 | 439 | ##### ./libelisa-app-nop.so 440 | 441 | ```./libelisa-app-nop.so``` implements the application-specific logic which is nop this time, and it includes the procedure for both the manager VM and guest VMs: 442 | - in the manager VM side, it [loads the code](https://github.com/yasukata/elisa-app-nop/blob/bc932930c0dd07fbee61a41c968f5476a8bde981/main.c#L42-L59) specified by an environment variable [```ELISA_APPLIB_FILE```](https://github.com/yasukata/elisa-app-nop/blob/bc932930c0dd07fbee61a41c968f5476a8bde981/main.c#L29) to the sub EPT context of a guest VM, and 443 | - in the guest VM side, it [enters the sub EPT context and executes the code in it](https://github.com/yasukata/elisa-app-nop/blob/bc932930c0dd07fbee61a41c968f5476a8bde981/main.c#L69-L75). 444 | 445 | ##### ./deps/elisa/util/elisa-util-exec/a.out 446 | 447 | ```./deps/elisa/util/elisa-util-exec/a.out``` is a generic launcher application and used for executing both the commands for the manager VM and guest VMs; we note that we made ```./deps/elisa/util/elisa-util-exec/a.out``` just for reducing redundant implementations and it is not mandatory to use. 448 | 449 | ```./deps/elisa/util/elisa-util-exec/a.out``` works either in the manager VM mode or the guest VM mode; if it has an argument for ```-s``` which specifies the IP address of the manager VM, it will be the guest VM mode, and otherwise, the manager VM mode. 450 | 451 | ```./deps/elisa/util/elisa-util-exec/a.out``` loads a library file specified through its ```-f``` option passes a pointer to a function named ```elisa__server_cb``` to libelisa if it is in the manager VM mode, and ```elisa__client_cb``` will be passed to libelisa if it is in the guest VM mode. The option ```-p``` specifies the port to listen on for the manager VM or connect to for a guest VM. 452 | 453 | #### Explanation on the commands 454 | 455 | The meaning of the command for the manager VM is: 456 | - ```sudo```: we need sudo for translating from GVA to GPA 457 | - ```ELISA_APPLIB_FILE=./lib/libelisa-applib-nop/lib.so```: requesting ```./libelisa-app-nop.so``` to load ```./lib/libelisa-applib-nop/lib.so``` to the sub EPT context of a guest VM. 458 | - ```./deps/elisa/util/elisa-util-exec/a.out```: the binary executed 459 | - ```-f ./libelisa-app-nop.so```: requesting ```./deps/elisa/util/elisa-util-exec/a.out``` to load ```./libelisa-app-nop.so``` 460 | - ```-p 10000```: requesting ```./deps/elisa/util/elisa-util-exec/a.out``` to listen on port 10000 461 | 462 | The meaning of the command for the guest VM is: 463 | - ```taskset -c 0```: we specify the CPU affinity of the process to ensure that the negotiation with the manager VM and the execution of the code in the sub EPT context are done on the same vCPU. (The negotiation (explained in Section 5.3 of the paper) with the manager VM has to be done for each vCPU, but this implementation, particularly on the guest VM side, only performs a single negotiation with the manager VM. By modifying the implementation for the guest VM to conduct negotiation for each vCPU, we can remove this restriction.) 464 | - ```./deps/elisa/util/elisa-util-exec/a.out```: the binary executed 465 | - ```-f ./libelisa-app-nop.so```: requesting ```./deps/elisa/util/elisa-util-exec/a.out``` to load ```./libelisa-app-nop.so``` 466 | - ```-p 10000```: requesting ```./deps/elisa/util/elisa-util-exec/a.out``` to connect to port 10000 of the manager VM 467 | - ```-s 127.0.0.1```: telling ```./deps/elisa/util/elisa-util-exec/a.out``` that the IP address of the manager VM is 127.0.0.1. (If you use different VMs for the manager VM and the guest VM, please change this part to specify the IP address of the manager VM.) 468 | 469 | #### Explanation of the implementations 470 | 471 | The output by printf on the guest VM comes from [line 70](https://github.com/yasukata/elisa-app-nop/blob/bc932930c0dd07fbee61a41c968f5476a8bde981/main.c#L70) and [line 74](https://github.com/yasukata/elisa-app-nop/blob/bc932930c0dd07fbee61a41c968f5476a8bde981/main.c#L74). 472 | 473 | The guest VM prints [the value obtained from the code executed in the sub EPT context](https://github.com/yasukata/elisa-app-nop/blob/bc932930c0dd07fbee61a41c968f5476a8bde981/main.c#L72). 474 | 475 | The implementation of the code for the sub EPT context is found in [elisa-app-nop/lib/libelisa-applib-nop/main.c](https://github.com/yasukata/elisa-app-nop/blob/bc932930c0dd07fbee61a41c968f5476a8bde981/lib/libelisa-applib-nop/main.c); especially, the current implementation of it [returns 1](https://github.com/yasukata/elisa-app-nop/blob/bc932930c0dd07fbee61a41c968f5476a8bde981/lib/libelisa-applib-nop/main.c#L26), therefore, we have got the output ```return value is 1```. 476 | 477 | In ```elisa-app-nop/lib/libelisa-applib-nop/main.c```, [line 71](https://github.com/yasukata/elisa-app-nop/blob/bc932930c0dd07fbee61a41c968f5476a8bde981/main.c#L71) disables interrupt and [line 73](https://github.com/yasukata/elisa-app-nop/blob/bc932930c0dd07fbee61a41c968f5476a8bde981/main.c#L73) enables interrupt; to find the reason why we need this, please refer to Section 5.6 of the paper. 478 | 479 | #### Quick experiment 480 | 481 | For a quick experiment, if you replace the return value in ```elisa-app-nop/lib/libelisa-applib-nop/main.c``` with a value that you like and recompile it, the next execution will print the value that you put. 482 | 483 | ## Commentary 484 | 485 | This commentary bridges the descriptions in the paper and the source code of the ELISA prototype. 486 | 487 | ### List of the sections complemented by the commentary 488 | 489 | - [Section 4.1 : Anywhere Page Table (APT)](#section-41--anywhere-page-table-apt) 490 | - [Section 4.2 : Gate EPT Context](#section-42--gate-ept-context) 491 | - [Section 5.1 : ELISA-Specific Hypercalls](#section-51--elisa-specific-hypercalls) 492 | - [Section 5.2 : libelisa](#section-52--libelisa) 493 | - [Section 5.3 : Negotiation Steps](#section-53--negotiation-steps) 494 | - [Section 5.4 : Code for the Sub EPT Context](#section-54--code-for-the-sub-ept-context) 495 | - [Section 5.5 : Shared Memory Management](#section-55--shared-memory-management) 496 | - [Section 5.6 : Interrupt Setting](#section-56--interrupt-setting) 497 | - [Section 6.1 : Context Switch Overhead](#section-61--context-switch-overhead) 498 | - [Section 7.1 : VM Networking System](#section-71--vm-networking-system) 499 | 500 | ### Section 4.1 : Anywhere Page Table (APT) 501 | 502 | The implementation of APT is found at [```configure_apt```](https://github.com/yasukata/libelisa/blob/e46242e5ecd854a807f9ea1816dae3f292d5a250/server.c#L304-L348), which assumes to be called at [the last step](https://github.com/yasukata/libelisa/blob/e46242e5ecd854a807f9ea1816dae3f292d5a250/server.c#L383) of [an EPT context setup](https://github.com/yasukata/libelisa/blob/e46242e5ecd854a807f9ea1816dae3f292d5a250/server.c#L356-L384); it fills all empty GPA entries in the EPT with the pointer to the HPA of the created page table root. 503 | 504 | The page table root for APT is allocated as [```uint64_t apt_mem[512]```](https://github.com/yasukata/libelisa/blob/e46242e5ecd854a807f9ea1816dae3f292d5a250/server.c#L299) which is part of [```struct vcpu_ept_ctx```](https://github.com/yasukata/libelisa/blob/e46242e5ecd854a807f9ea1816dae3f292d5a250/server.c#L298-L302) that represents an EPT context; we allocate ```struct vcpu_ept_ctx``` [on the stack](https://github.com/yasukata/libelisa/blob/e46242e5ecd854a807f9ea1816dae3f292d5a250/server.c#L427-L439), and [```__attribute__((aligned(0x1000)))```](https://github.com/yasukata/libelisa/blob/e46242e5ecd854a807f9ea1816dae3f292d5a250/server.c#L302) automatically instantiates ```uint64_t apt_mem[512]``` at a 4KB aligned memory address. 505 | 506 | ```configure_apt``` first [creates the mapping between the last GPA in the GPA range and HPA of ```uint64_t apt_mem[512]```](https://github.com/yasukata/libelisa/blob/e46242e5ecd854a807f9ea1816dae3f292d5a250/server.c#L328-L330); at the same time, and this will add an entry to each level of the EPT tree. 507 | 508 | Then, we [collect the added entry value at each layer of the EPT tree](https://github.com/yasukata/libelisa/blob/e46242e5ecd854a807f9ea1816dae3f292d5a250/server.c#L336-L344), in a 64-bit integer array. 509 | 510 | Afterward, we walk through the EPT tree, and if there is an empty entry at a specific level, we [fill this entry with the collected entry value of the same level](https://github.com/yasukata/libelisa/blob/e46242e5ecd854a807f9ea1816dae3f292d5a250/server.c#L315). 511 | 512 | ### Section 4.2 : Gate EPT Context 513 | 514 | The implementation of the EPT context transition point (shown by Figure 5 in the paper) is divided across the default, gate, and sub EPT contexts. 515 | 516 | #### The default EPT context 517 | 518 | A guest VM only needs to load a [12 KB code block](https://github.com/yasukata/libelisa/blob/e46242e5ecd854a807f9ea1816dae3f292d5a250/client.c#L22-L53) (4 KB x 3 : top, middle, and bottom) in its program to make an entry point. 519 | 520 | The [top 4 KB page](https://github.com/yasukata/libelisa/blob/e46242e5ecd854a807f9ea1816dae3f292d5a250/client.c#L23-L38) has the entry point to the gate EPT context, named ```elisa_gate_entry```, at the end of it. 521 | 522 | The [middle 4 KB page](https://github.com/yasukata/libelisa/blob/e46242e5ecd854a807f9ea1816dae3f292d5a250/client.c#L42-L43) is empty. (The shaded part in Figure 5.) 523 | 524 | When the execution returns from the sub EPT context, it lands at the top of the [bottom 4 KB page](https://github.com/yasukata/libelisa/blob/e46242e5ecd854a807f9ea1816dae3f292d5a250/client.c#L47-L53). 525 | 526 | #### The gate EPT context 527 | 528 | The guest VM enters the gate EPT context by executing [VMFUNC at the end of the top 4 KB page](https://github.com/yasukata/libelisa/blob/e46242e5ecd854a807f9ea1816dae3f292d5a250/client.c#L37), and lands at the top of the [middle 4 KB page](https://github.com/yasukata/libelisa/blob/e46242e5ecd854a807f9ea1816dae3f292d5a250/server.c#L48-L92) in the gate EPT context; the code shown in Figure 6 is [in this middle 4 KB page](https://github.com/yasukata/libelisa/blob/e46242e5ecd854a807f9ea1816dae3f292d5a250/server.c#L48-L60). 529 | 530 | The manager VM maps the EPTP list for the vCPU at the [bottom 4 KB page](https://github.com/yasukata/libelisa/blob/e46242e5ecd854a807f9ea1816dae3f292d5a250/server.c#L511-L518). 531 | 532 | The manager VM conducts the page table and EPT settings for the gate EPT context [here](https://github.com/yasukata/libelisa/blob/e46242e5ecd854a807f9ea1816dae3f292d5a250/server.c#L507-L532). 533 | 534 | #### The sub EPT context 535 | 536 | The middle 4 KB page of the gate EPT context contains VMFUNC and it brings the execution at [line 116](https://github.com/yasukata/libelisa/blob/e46242e5ecd854a807f9ea1816dae3f292d5a250/server.c#L116) in the [middle 4 KB page](https://github.com/yasukata/libelisa/blob/e46242e5ecd854a807f9ea1816dae3f292d5a250/server.c#L116-L147) of the sub EPT context. 537 | 538 | The manager VM maps the EPTP list for the vCPU at the [bottom 4 KB page](https://github.com/yasukata/libelisa/blob/e46242e5ecd854a807f9ea1816dae3f292d5a250/server.c#L471). 539 | 540 | ### Section 5.1 : ELISA-Specific Hypercalls 541 | 542 | The ELISA-specific hypercalls are implemented in [the patch for KVM](https://github.com/yasukata/kvm-elisa/blob/0fadc257ca8a99365ba8db09d03eed431881cdd8/elisa.patch#L113-L128). 543 | 544 | As mentioned in [general information](#general-information) for the setup, we tried to minimize the size of the patch for KVM as much as possible; for simplicity, in the current implementation, there is no clear separation between the manager and guest hypercalls, and all guest VMs can execute all ELISA-specific hypercalls. 545 | 546 | Therefore, if you wish to separate manager and guest hypercalls, you need to add some implementations for checking whether a VM, attempting a hypercall, is privileged or not. 547 | 548 | The implementations of the manager hypercall functionalities are: 549 | - [address translation from GPA to HPA](https://github.com/yasukata/kvm-elisa/blob/0fadc257ca8a99365ba8db09d03eed431881cdd8/elisa.patch#L42-L44); this is used by the manager VM to [translate from GVA to HPA](https://github.com/yasukata/libelisa/blob/e46242e5ecd854a807f9ea1816dae3f292d5a250/vmcall.c#L104-L110). 550 | - to [associate a created EPTP list with a vCPU of a guest VM](https://github.com/yasukata/kvm-elisa/blob/0fadc257ca8a99365ba8db09d03eed431881cdd8/elisa.patch#L84-L89). 551 | 552 | The implementations of the guest hypercall functionalities are: 553 | - to [obtain a unique identifier of its vCPU which is used in the negotiation with the manager VM](https://github.com/yasukata/kvm-elisa/blob/0fadc257ca8a99365ba8db09d03eed431881cdd8/elisa.patch#L94-L96). 554 | - to [trigger the activation of the EPTP list configured by the manager VM](https://github.com/yasukata/kvm-elisa/blob/0fadc257ca8a99365ba8db09d03eed431881cdd8/elisa.patch#L97-L104). 555 | 556 | ### Section 5.2 : libelisa 557 | 558 | In the current implementation, libelisa offers two primary functions: [```elisa_server```](https://github.com/yasukata/libelisa/blob/e46242e5ecd854a807f9ea1816dae3f292d5a250/server.c#L577-L579) for the manager VM programming and [```elisa_client```](https://github.com/yasukata/libelisa/blob/e46242e5ecd854a807f9ea1816dae3f292d5a250/client.c#L59) for the guest VM programming; the primary [task of elisa-util-exec described in the example section](#run-elisa-app-nop) is to execute either [```elisa_server```](https://github.com/yasukata/elisa-util-exec/blob/eb8a14a132fe7f9b0e7980c7a4ab3a30ef8a465f/main.c#L67) or [```elisa_client```](https://github.com/yasukata/elisa-util-exec/blob/eb8a14a132fe7f9b0e7980c7a4ab3a30ef8a465f/main.c#L72). 559 | 560 | ```elisa_server``` enters an infinite server loop, and ```elisa_client``` returns after the setup of the gate/sub EPT contexts has been completed. 561 | 562 | #### The arguments of [```elisa_server```](https://github.com/yasukata/libelisa/blob/e46242e5ecd854a807f9ea1816dae3f292d5a250/server.c#L577-L579) 563 | 564 | - ```const int server_port```: the port that the manager VM listens on. 565 | - ```int (*server_cb)(int, uint64_t *, struct elisa_map_req **, int *, uint64_t *)```: the callback function [called during the sub EPT context setup](https://github.com/yasukata/libelisa/blob/e46242e5ecd854a807f9ea1816dae3f292d5a250/server.c#L488). 566 | - ```int (*server_exit_cb)(uint64_t *)```: the callback function [called when a guest VM, that has a sub EPT context, exits](https://github.com/yasukata/libelisa/blob/e46242e5ecd854a807f9ea1816dae3f292d5a250/server.c#L572), and this is primarily preserved to implement application-specific destructors. 567 | 568 | [```server_cb```](https://github.com/yasukata/libelisa/blob/e46242e5ecd854a807f9ea1816dae3f292d5a250/server.c#L488) is the point where programmers can customize how a sub EPT context is configured, and it takes the following arguments: 569 | - ```int```: a socket file descriptor connected to a guest VM. 570 | - ```uint64_t *```: a pointer to a 64-bit variable that a programmer can use for any purpose, and passed to ```server_exit_cb```; primarily assuming to assign an identifier in ```server_cb```. 571 | - ```struct elisa_map_req **```: a pointer to an array of [```struct elisa_map_req```](https://github.com/yasukata/libelisa/blob/e46242e5ecd854a807f9ea1816dae3f292d5a250/include/libelisa.h) page mapping requests that libelisa assumes to be prepared by ```server_cb```; subsequently, libelisa makes page table and EPT mappings according to the requests put in this array. 572 | - ```int *```: number of mapping requests in the array of ```struct elisa_map_req```, set by ```server_cb```. 573 | - ```uint64_t *```: a virtual address of the entry point of the core function executed in the sub EPT context; this value is [embedded](https://github.com/yasukata/libelisa/blob/e46242e5ecd854a807f9ea1816dae3f292d5a250/server.c#L490) to [the code of the sub EPT context](https://github.com/yasukata/libelisa/blob/e46242e5ecd854a807f9ea1816dae3f292d5a250/server.c#L129). 574 | 575 | Programmers can map pages allocated in a user-space process to a sub EPT context of a guest VM, by creating an array of [```struct elisa_map_req```](https://github.com/yasukata/libelisa/blob/e46242e5ecd854a807f9ea1816dae3f292d5a250/include/libelisa.h#L61-L69). 576 | 577 | libelisa-extra has [```elisa_create_program_map_req```](https://github.com/yasukata/libelisa-extra/blob/a3ccab11ffb65c7cb5be741bdd5c0f041c40c343/include/libelisa_extra/map.h#L35-L40) which helps programmers to load a shared library file to a sub EPT context of a guest VM in ```server_cb```; elisa-app-nop [uses this](https://github.com/yasukata/elisa-app-nop/blob/bc932930c0dd07fbee61a41c968f5476a8bde981/main.c#L48-L53) to load the program to a sub EPT context. 578 | 579 | ```elisa_create_program_map_req``` takes the following arguments: 580 | - ```const char *program_filename```: the name of the shared library file to be loaded in the sub EPT context. 581 | - ```uint64_t base_gpa```: the lowest GPA where the code will be located in the sub EPT context; GVA is the same as the one allocated in the process executing this on the manager VM. 582 | - ```void **handle```: the handle [opened by ```dlmopen```](https://github.com/yasukata/libelisa-extra/blob/a3ccab11ffb65c7cb5be741bdd5c0f041c40c343/include/libelisa_extra/map.h#L88); this is mainly used for finding the address of the entry point of the core function. elisa-app-nop [uses this](https://github.com/yasukata/elisa-app-nop/blob/bc932930c0dd07fbee61a41c968f5476a8bde981/main.c#L57) to find the address of the entry function named [```entry_function``` in the code for the sub EPT context](https://github.com/yasukata/elisa-app-nop/blob/bc932930c0dd07fbee61a41c968f5476a8bde981/lib/libelisa-applib-nop/main.c#L19). 583 | - ```struct elisa_map_req **map_req```: array of the memory mapping request, particularly for the code, and [manipulated in 584 | ```elisa_create_program_map_req```](https://github.com/yasukata/libelisa-extra/blob/a3ccab11ffb65c7cb5be741bdd5c0f041c40c343/include/libelisa_extra/map.h#L70-L77). 585 | - ```int *map_req_cnt```: number of the mapping request made by ```elisa_create_program_map_req```. 586 | - ```int *map_req_num```: the current length of the ```map_req``` array, that is [dynamically extended in ```elisa_create_program_map_req```](https://github.com/yasukata/libelisa-extra/blob/a3ccab11ffb65c7cb5be741bdd5c0f041c40c343/include/libelisa_extra/map.h#L63-L68). 587 | 588 | #### The arguments of [```elisa_client```](https://github.com/yasukata/libelisa/blob/e46242e5ecd854a807f9ea1816dae3f292d5a250/client.c#L59) 589 | - ```const char *server_addr_str```: the string specifying the address of the manager VM. 590 | - ```const int server_port```: the port the manager VM listens on. 591 | - ```int client_cb(int)```: a callback function [called during the negotiation with the manager VM](https://github.com/yasukata/libelisa/blob/e46242e5ecd854a807f9ea1816dae3f292d5a250/client.c#L75); this is preserved so that the negotiation steps can be extended for passing application-specific information through the socket file descriptor passed as the argument (int). 592 | 593 | ### Section 5.3 : Negotiation Steps 594 | 595 | To initiate a negotiation, a user-space process on a guest VM sends two integers: [a vCPU-specific id](https://github.com/yasukata/libelisa/blob/e46242e5ecd854a807f9ea1816dae3f292d5a250/client.c#L66-L67) and [the GVA where the user-space process wishes to instantiate an entry point to the gate EPT context](https://github.com/yasukata/libelisa/blob/e46242e5ecd854a807f9ea1816dae3f292d5a250/client.c#L70-L72). 596 | 597 | The vCPU-specific id can be obtained by [the guest hypercall](https://github.com/yasukata/libelisa/blob/e46242e5ecd854a807f9ea1816dae3f292d5a250/vmcall.c#L148-L155) which is [handled in the host](https://github.com/yasukata/kvm-elisa/blob/5b2e2e07dfb797d6d96474b12d57af3a40745794/elisa.patch#L94-L96), and the GVA for the entry point of the gate EPT context should be the GVA of the middle page shown in Figure 5; it is [```gate_entry_page```](https://github.com/yasukata/libelisa/blob/e46242e5ecd854a807f9ea1816dae3f292d5a250/client.c#L24) (the top page in Figure 5) + 0x1000. 598 | 599 | When a user-space process on the manager VM [receives these two integer values](https://github.com/yasukata/libelisa/blob/e46242e5ecd854a807f9ea1816dae3f292d5a250/server.c#L416-L417), it configures [the gate EPT entry point at the requested GVA](https://github.com/yasukata/libelisa/blob/e46242e5ecd854a807f9ea1816dae3f292d5a250/server.c#L509-L514) and [associates the created EPTP list with the vCPU through the received vCPU identifier](https://github.com/yasukata/libelisa/blob/e46242e5ecd854a807f9ea1816dae3f292d5a250/server.c#L544-L547) using [the manager hypercall](https://github.com/yasukata/libelisa/blob/7d11cf7c4bb447825bca67072d0ce8d56a329bf0/vmcall.c#L132-L146) which is [handled in the host](https://github.com/yasukata/kvm-elisa/blob/5b2e2e07dfb797d6d96474b12d57af3a40745794/elisa.patch#L84-L89). NOTE: this part of KVM patch implementation is [very lazy](https://github.com/yasukata/kvm-elisa/blob/5b2e2e07dfb797d6d96474b12d57af3a40745794/elisa.patch#L76) so that [we can minimize the size of the KVM patch](#general-information). 600 | 601 | After the manager VM completes the gate/sub EPT configuration, it [acknowledges to the guest VM](https://github.com/yasukata/libelisa/blob/e46242e5ecd854a807f9ea1816dae3f292d5a250/server.c#L550-L551). 602 | 603 | The guest VM, that has [received the acknowledgment](https://github.com/yasukata/libelisa/blob/e46242e5ecd854a807f9ea1816dae3f292d5a250/client.c#L79), [activates the created EPTP list](https://github.com/yasukata/libelisa/blob/e46242e5ecd854a807f9ea1816dae3f292d5a250/client.c#L82) using the [guest hypercall](https://github.com/yasukata/libelisa/blob/e46242e5ecd854a807f9ea1816dae3f292d5a250/vmcall.c#L157-L162) that is [handled by the host](https://github.com/yasukata/kvm-elisa/blob/5b2e2e07dfb797d6d96474b12d57af3a40745794/elisa.patch#L97-L103). 604 | 605 | ### Section 5.4 : Code for the Sub EPT Context 606 | 607 | As seen in elisa-app-nop, the code for the sub EPT context [can be implemented as a shared library](https://github.com/yasukata/elisa-app-nop/blob/bc932930c0dd07fbee61a41c968f5476a8bde981/lib/libelisa-applib-nop/main.c). 608 | 609 | To load a shared library file to a sub EPT context of a guest VM, we [use dlmopen](https://github.com/yasukata/libelisa-extra/blob/a3ccab11ffb65c7cb5be741bdd5c0f041c40c343/include/libelisa_extra/map.h#L88) in a user-space process on the manager VM so that we can avoid undesired symbol association. 610 | 611 | ### Section 5.5 : Shared Memory Management 612 | 613 | The following patch extends elisa-app-nop for the demo of the shared memory management. 614 | 615 | ```diff 616 | diff --git a/lib/libelisa-applib-nop/main.c b/lib/libelisa-applib-nop/main.c 617 | index dc5be09..e366883 100644 618 | --- a/lib/libelisa-applib-nop/main.c 619 | +++ b/lib/libelisa-applib-nop/main.c 620 | @@ -16,6 +16,8 @@ 621 | * 622 | */ 623 | 624 | +void *shm_ptr; 625 | + 626 | long entry_function(long rdi __attribute__((unused)), 627 | long rsi __attribute__((unused)), 628 | long rdx __attribute__((unused)), 629 | @@ -23,5 +25,6 @@ long entry_function(long rdi __attribute__((unused)), 630 | long r8 __attribute__((unused)), 631 | long r9 __attribute__((unused))) 632 | { 633 | - return 1; 634 | + unsigned long *val = (unsigned long *) shm_ptr, one = 1; 635 | + return __atomic_add_fetch(val, one, __ATOMIC_ACQ_REL); 636 | } 637 | diff --git a/main.c b/main.c 638 | index 2a7e9f0..085c4bb 100644 639 | --- a/main.c 640 | +++ b/main.c 641 | @@ -24,10 +24,14 @@ 642 | #include 643 | #include 644 | 645 | +#include 646 | + 647 | #define BASE_GPA (1UL << 37) /* TODO: ensure no-overlap with others */ 648 | 649 | #define ENV_APPLIB_FILE_STR "ELISA_APPLIB_FILE" 650 | 651 | +static void *__shm_ptr; 652 | + 653 | int elisa__server_exit_cb(uint64_t *user_any __attribute__((unused))) 654 | { 655 | return 0; 656 | @@ -51,6 +55,25 @@ int elisa__server_cb(int sockfd __attribute__((unused)), 657 | map_req, 658 | map_req_cnt, 659 | &map_req_num)); 660 | + { 661 | + uintptr_t *shm_ptr; 662 | + assert((shm_ptr = dlsym(handle, "shm_ptr")) != NULL); 663 | + *shm_ptr = (uintptr_t) __shm_ptr; 664 | + } 665 | + { 666 | + if (*map_req_cnt == map_req_num) { 667 | + map_req_num *= 2; 668 | + assert((*map_req = realloc(*map_req, sizeof(struct elisa_map_req) * map_req_num)) != NULL); 669 | + } 670 | + (*map_req)[*map_req_cnt].dst_gpa = 0x1000 * *map_req_cnt + BASE_GPA; 671 | + (*map_req)[*map_req_cnt].dst_gva = (uint64_t) __shm_ptr; 672 | + (*map_req)[*map_req_cnt].src_gxa = (uint64_t) __shm_ptr; 673 | + (*map_req)[*map_req_cnt].flags = 0; 674 | + (*map_req)[*map_req_cnt].level = 1; 675 | + (*map_req)[*map_req_cnt].pt_flags = PT_P | PT_W | PT_U; 676 | + (*map_req)[*map_req_cnt].ept_flags = EPT_R | EPT_W | /* EPT_X |*/ EPT_U | EPT_MT; 677 | + (*map_req_cnt)++; 678 | + } 679 | } 680 | } 681 | assert(handle); 682 | @@ -66,16 +89,20 @@ int elisa__client_cb(int sockfd __attribute__((unused))) 683 | 684 | int elisa__client_work(void) 685 | { 686 | - long ret; 687 | - printf("enter sub EPT context and get a return value\n"); 688 | - elisa_disable_irq_if_enabled(); 689 | - ret = elisa_gate_entry(0, 0, 0, 511 /* rcx */, 0, 0); 690 | + long ret, cnt = 0; 691 | + while (cnt++ < 10) { 692 | + printf("%ld: enter sub EPT context and get a return value\n", cnt); 693 | + elisa_disable_irq_if_enabled(); 694 | + ret = elisa_gate_entry(0, 0, 0, 511 /* rcx */, 0, 0); 695 | + printf("%ld: return value is %ld\n", cnt, ret); 696 | + sleep(1); 697 | + } 698 | vmcall_sti(); 699 | - printf("return value is %ld\n", ret); 700 | return 0; 701 | } 702 | 703 | int elisa__exec_init(void) 704 | { 705 | + assert((__shm_ptr = mmap(NULL, 0x1000, PROT_READ | PROT_WRITE, MAP_SHARED | MAP_ANONYMOUS | MAP_POPULATE, -1, 0)) != MAP_FAILED); 706 | return 0; 707 | } 708 | ``` 709 | 710 | The point in the patch above is that ```__shm_ptr``` is allocated by mmap with MAP_SHARED in ```elisa__exec_init``` which is called in the parent process in the manager VM, and this will be the shared memory among sub EPT contexts of guest VMs because it is exposed to the sub EPT contexts of guest VMs by child processes [forked](https://github.com/yasukata/libelisa/blob/e46242e5ecd854a807f9ea1816dae3f292d5a250/server.c#L591) from the parent process on the manager VM. 711 | 712 | The code for the sub EPT context accesses the shared memory through a variable named ```shm_ptr```, and the manager VM associates ```__shm_ptr``` with it by finding the pointer to it using dlsym. 713 | 714 | The manager VM exposes the memory pointed by ```__shm_ptr``` to the sub EPT context by adding an entry to the array of ```map_req``` in ```elisa__server_cb```. 715 | 716 | #### Trying the shared memory demo 717 | 718 | To apply the patch above, please save this a file named ```./shm-demo.patch``` in the top directory of elisa-app-nop (commit id : ffb1598) and type the following commands; here, we are assuming all steps shown in the [Trying an example application : elisa-app-nop](#trying-an-example-application--elisa-app-nop) section have been completed. 719 | 720 | ``` 721 | git apply ./shm-demo.patch 722 | ``` 723 | 724 | ``` 725 | make -C lib/libelisa-applib-nop 726 | ``` 727 | 728 | ``` 729 | make 730 | ``` 731 | 732 | Please launch the process for the manager VM; the following is the same as the one shown in the [Run elisa-app-nop section](#the-command-for-the-manager-vm). 733 | 734 | ``` 735 | sudo ELISA_APPLIB_FILE=./lib/libelisa-applib-nop/lib.so ./deps/elisa/util/elisa-util-exec/a.out -f ./libelisa-app-nop.so -p 10000 736 | ``` 737 | 738 | Then, please open a new console and launch a process for a guest VM; this is also the same as the one shown in the [Run elisa-app-nop section](#the-command-for-a-guest-vm). 739 | 740 | ``` 741 | taskset -c 0 ./deps/elisa/util/elisa-util-exec/a.out -f ./libelisa-app-nop.so -p 10000 -s 127.0.0.1 742 | ``` 743 | 744 | Please open a new console and execute, on a guest VM, the following command which is mostly similar to the command above, but the only difference is the number passed to ```taskset -c``` to specify a vCPU id which is different from the one used for the process above. **WARNING: pay attention to this vCPU specification if you run the commands above and below on the same VM; if you specify a vCPU id that is already used by a different process, the guest VM will crash because of [the lazy implementation in the KVM patch](#general-information) missing a check in the EPTP list activation.** We note that the commands above assume the use of a single VM, but the proper usage is to dedicate a VM for each of the two commands above and the one below. 745 | 746 | ``` 747 | taskset -c 1 ./deps/elisa/util/elisa-util-exec/a.out -f ./libelisa-app-nop.so -p 10000 -s 127.0.0.1 748 | ``` 749 | 750 | What you supposedly see from the commands above is that each process on a guest VM increments a shared variable in the sub EPT contexts every 1 second; since the variable is shared by the two processes on the guest VM(s), in each second, the variable will be incremented by 2. 751 | 752 | When a process on a guest VM exits, the memory reserved for its gate/sub EPT contexts (e.g., the memory for the code in the sub EPT context) is released by the kernel in the manager VM; but, the memory for ```__shm_ptr```, which is the shared memory, is not released until the parent process in the manager VM exits. 753 | 754 | ### Section 5.6 : Interrupt Setting 755 | 756 | To control the interrupt flag, the [current implementation](https://github.com/yasukata/libelisa/blob/e46242e5ecd854a807f9ea1816dae3f292d5a250/vmcall.c#L185-L194) employs [hypercalls which request wvmcs/rvmcs](https://github.com/yasukata/libelisa/blob/e46242e5ecd854a807f9ea1816dae3f292d5a250/vmcall.c#L166-L183) on an arbitrary field, to [the host](https://github.com/yasukata/kvm-elisa/blob/0fadc257ca8a99365ba8db09d03eed431881cdd8/elisa.patch#L31-L37); this could be too powerful and dangerous, but we think it is OK as long as for prototyping purpose. 757 | 758 | We assume that [```elisa_disable_irq_if_enabled```](https://github.com/yasukata/libelisa-extra/blob/a3ccab11ffb65c7cb5be741bdd5c0f041c40c343/include/libelisa_extra/irq.h#L24) in libelisa-extra is called before the every entry to the gate EPT context, for [checking the current flag register state](https://github.com/yasukata/libelisa-extra/blob/a3ccab11ffb65c7cb5be741bdd5c0f041c40c343/include/libelisa_extra/irq.h#L28-L34) and [disabling interrupts if enabled](https://github.com/yasukata/libelisa-extra/blob/a3ccab11ffb65c7cb5be741bdd5c0f041c40c343/include/libelisa_extra/irq.h#L34-L35); the typical code path only involves [pushf, pop, and cmp instructions](https://github.com/yasukata/libelisa-extra/blob/a3ccab11ffb65c7cb5be741bdd5c0f041c40c343/include/libelisa_extra/irq.h#L28-L34) and it is unlikely to [trigger the costly hypercall](https://github.com/yasukata/libelisa-extra/blob/a3ccab11ffb65c7cb5be741bdd5c0f041c40c343/include/libelisa_extra/irq.h#L35) to disable interrupts because the flag register is preserved by the kernel in the guest VM. 759 | 760 | ### Section 6.1 : Context Switch Overhead 761 | 762 | #### Measuring the context round-trip time of ELISA 763 | 764 | To measure the context switch overhead of ELISA, we can use the following program which is the patch for elisa-app-nop. 765 | 766 | ```diff 767 | diff --git a/main.c b/main.c 768 | index 2a7e9f0..62bf9b1 100644 769 | --- a/main.c 770 | +++ b/main.c 771 | @@ -20,6 +20,8 @@ 772 | #define _GNU_SOURCE 773 | #endif 774 | 775 | +#include 776 | + 777 | #include 778 | #include 779 | #include 780 | @@ -66,12 +68,26 @@ int elisa__client_cb(int sockfd __attribute__((unused))) 781 | 782 | int elisa__client_work(void) 783 | { 784 | - long ret; 785 | - printf("enter sub EPT context and get a return value\n"); 786 | - elisa_disable_irq_if_enabled(); 787 | - ret = elisa_gate_entry(0, 0, 0, 511 /* rcx */, 0, 0); 788 | - vmcall_sti(); 789 | - printf("return value is %ld\n", ret); 790 | + if (getenv("ELISA_APP_NOP_LOOPCNT")) { 791 | + unsigned long cnt = 0, i, loop = atol(getenv("ELISA_APP_NOP_LOOPCNT")); 792 | + printf("loop: %lu\n", loop); 793 | + { 794 | + unsigned long t = ({ struct timespec ts; assert(!clock_gettime(CLOCK_REALTIME, &ts)); ts.tv_sec * 1000000000UL + ts.tv_nsec; }); 795 | + for (i = 0; i < loop; i++) { 796 | + elisa_disable_irq_if_enabled(); 797 | + cnt += elisa_gate_entry(0, 0, 0, 511 /* rcx */, 0, 0); 798 | + } 799 | + { 800 | + unsigned long _t = ({ struct timespec ts; assert(!clock_gettime(CLOCK_REALTIME, &ts)); ts.tv_sec * 1000000000UL + ts.tv_nsec; }); 801 | + printf("duration %lu nsec (%lu nsec for each)\n", 802 | + _t - t, 803 | + (_t - t) / loop); 804 | + } 805 | + } 806 | + vmcall_sti(); 807 | + printf("cnt: %lu\n", cnt); 808 | + } else 809 | + printf("env val ELISA_APP_NOP_LOOPCNT is not specified, just exit\n"); 810 | return 0; 811 | } 812 | ``` 813 | 814 | Please save the code above as ```./nop-bench.patch``` and type the following command to apply it. (If you have tried the [shared memory experiment](#section-55--shared-memory-management), please restore the state of the elisa-app-nop repository before typing the following command so that it will not involve the patch for the shared memory experiment.) 815 | 816 | ``` 817 | git apply ./nop-bench.patch 818 | ``` 819 | 820 | After the patch is applied, please type the following to compile it. 821 | 822 | ``` 823 | make 824 | ``` 825 | 826 | Then, please launch the process on the manager VM by the following command; the following is the same as the one shown in the [Run elisa-app-nop section](#the-command-for-the-manager-vm). 827 | 828 | ``` 829 | sudo ELISA_APPLIB_FILE=./lib/libelisa-applib-nop/lib.so ./deps/elisa/util/elisa-util-exec/a.out -f ./libelisa-app-nop.so -p 10000 830 | ``` 831 | 832 | On the guest VM, please type the following; this is almost the same as the one shown in the [Run elisa-app-nop section](#the-command-for-a-guest-vm), but the only difference is that this has ```ELISA_APP_NOP_LOOPCNT=100000``` to specify the number of loops. 833 | 834 | ``` 835 | ELISA_APP_NOP_LOOPCNT=100000 taskset -c 0 ./deps/elisa/util/elisa-util-exec/a.out -f ./libelisa-app-nop.so -p 10000 -s 127.0.0.1 836 | ``` 837 | 838 | The command above executes 100000 times of context round-trips between the default and sub EPT contexts and shows the average time spent on each. The following is the example output which shows it takes 198 ns for a context round-trip. 839 | 840 | ``` 841 | $ ELISA_APP_NOP_LOOPCNT=100000 taskset -c 0 ./deps/elisa/util/elisa-util-exec/a.out -f ./libelisa-app-nop.so -p 10000 -s 127.0.0.1 842 | loop: 100000 843 | duration 19858557 nsec (198 nsec for each) 844 | cnt: 100000 845 | ``` 846 | 847 | We note that 100000 is small for the number of loops, and it should be increased a bit for more accurate measurement. 848 | 849 | #### Measuring the speed of VMCALL 850 | 851 | To measure the speed of a VMCALL invocation, we can use the following program. 852 | 853 | ```c 854 | #include 855 | #include 856 | #include 857 | #include 858 | 859 | void ____asm_impl_vmcall(void) 860 | { 861 | asm volatile( 862 | ".globl do_vmcall \n\t" 863 | "do_vmcall: \n\t" 864 | "endbr64 \n\t" 865 | "push %rbp \n\t" 866 | "movq %rsp, %rbp \n\t" 867 | 868 | "push %rbx \n\t" 869 | "push %rcx \n\t" 870 | "push %rdx \n\t" 871 | "push %rsi \n\t" 872 | 873 | /* adjust calling convention */ 874 | "push %r8 \n\t" // a3 875 | "push %rcx \n\t" // a2 876 | "push %rdx \n\t" // a1 877 | "push %rsi \n\t" // a0 878 | "push %rdi \n\t" // nr 879 | "pop %rax \n\t" // rax: nr 880 | "pop %rbx \n\t" // rbx: a0 881 | "pop %rcx \n\t" // rcx: a1 882 | "pop %rdx \n\t" // rdx: a2 883 | "pop %rsi \n\t" // rsi: a3 884 | 885 | "vmcall \n\t" 886 | 887 | "pop %rsi \n\t" 888 | "pop %rdx \n\t" 889 | "pop %rcx \n\t" 890 | "pop %rbx \n\t" 891 | 892 | "leaveq \n\t" 893 | "retq \n\t" 894 | ); 895 | } 896 | 897 | extern long do_vmcall(long, ...); 898 | 899 | int main(void) 900 | { 901 | if (getenv("ELISA_APP_NOP_LOOPCNT")) { 902 | unsigned long cnt = 0, i, loop = atol(getenv("ELISA_APP_NOP_LOOPCNT")); 903 | printf("loop: %lu\n", loop); 904 | { 905 | unsigned long t = ({ struct timespec ts; assert(!clock_gettime(CLOCK_REALTIME, &ts)); ts.tv_sec * 1000000000UL + ts.tv_nsec; }); 906 | for (i = 0; i < loop; i++) 907 | cnt += do_vmcall(1000 /* KVM_HC_ELISA_UTIL */, 1 /* HC_ELISA_UTIL_DEBUG */); 908 | { 909 | unsigned long _t = ({ struct timespec ts; assert(!clock_gettime(CLOCK_REALTIME, &ts)); ts.tv_sec * 1000000000UL + ts.tv_nsec; }); 910 | printf("duration %lu nsec (%lu nsec for each)\n", 911 | _t - t, 912 | (_t - t) / loop); 913 | } 914 | } 915 | printf("cnt: %lu\n", cnt); 916 | } else 917 | printf("env val ELISA_APP_NOP_LOOPCNT is not specified, just exit\n"); 918 | return 0; 919 | } 920 | ``` 921 | 922 | Please save the program above as ```./vcbench.c``` and please compile it by the following command. 923 | 924 | ``` 925 | gcc -O3 vcbench.c 926 | ``` 927 | 928 | The following command executes VMCALL 100000 times and shows the average time for each execution. 929 | 930 | ``` 931 | ELISA_APP_NOP_LOOPCNT=100000 ./a.out 932 | ``` 933 | 934 | The following is the example output and it shows 707 ns is spent on a VMCALL execution. 935 | 936 | ``` 937 | $ ELISA_APP_NOP_LOOPCNT=100000 ./a.out 938 | loop: 100000 939 | duration 70723471 nsec (707 nsec for each) 940 | cnt: 18446744073609551616 941 | ``` 942 | 943 | ### Section 7.1 : VM Networking System 944 | 945 | The implementation of the ELISA-based VM networking system is [elisa-app-vmnet](https://github.com/yasukata/elisa-app-vmnet), and it consists of the following components. 946 | 947 | - The vNIC is made over the shared memory between the default and sub EPT context, and this setup is explained [below](#ivshmem-configuration-on-qemu). 948 | - The program run by the manager VM is [here](https://github.com/yasukata/elisa-app-vmnet/blob/4a9800ea488aa4ac9b057c3d98f04da35faefab1/server/main.c). 949 | - The code for the sub EPT context is [here](https://github.com/yasukata/elisa-app-vmnet/blob/4a9800ea488aa4ac9b057c3d98f04da35faefab1/server/lib/libelisa-applib-vmnet/main.c), and this involves a virtual switch implementation called [rvs](https://github.com/yasukata/rvs), which is exported for better modularity. 950 | - The program run by a guest VM is [the DPDK driver for rvif](https://github.com/yasukata/librte_pmd_rvif) which is a vNIC made for rvs and [a modular extension which triggers VMFUNC](https://github.com/yasukata/elisa-app-vmnet/blob/4a9800ea488aa4ac9b057c3d98f04da35faefab1/client/main.c). 951 | 952 | The behavior of the the ELISA-based VM networking system is as follows: 953 | - to transmit a packet, a guest VM requests the DPDK driver ([librte_pmd_rvif](https://github.com/yasukata/librte_pmd_rvif)) to transmit it, and this is done through ```rte_eth_tx_burst```, which is part of the DPDK API, invoked by a DPDK application running on the guest VM. 954 | - the DPDK driver [puts the packet on the I/O buffer of the vNIC](https://github.com/yasukata/librte_pmd_rvif/blob/9fbd375484cef415ea61bb6b436d509dca707c87/main.c#L276-L278). 955 | - then, it [triggers VMFUNC](https://github.com/yasukata/elisa-app-vmnet/blob/4a9800ea488aa4ac9b057c3d98f04da35faefab1/client/main.c#L55) to enter the sub EPT context, and this is applied through [a hook point](https://github.com/yasukata/librte_pmd_rvif/blob/9fbd375484cef415ea61bb6b436d509dca707c87/main.c#L285) in librte_pmd_rvif. 956 | - the execution of the sender guest VM reaches [the entry point of the virtual switch (rvs) execution](https://github.com/yasukata/elisa-app-vmnet/blob/4a9800ea488aa4ac9b057c3d98f04da35faefab1/server/lib/libelisa-applib-vmnet/main.c#L68) in the sub EPT context. 957 | - the virtual switch [looks up the destination of the packet](https://github.com/yasukata/rvs/blob/76a6b5d34a19af881964ef6c91155cb5ddec5847/rvs.c#L50-L58), and [copies it to the I/O buffer of the destination NIC](https://github.com/yasukata/rvs/blob/76a6b5d34a19af881964ef6c91155cb5ddec5847/rvs.c#L79-L88); this part is conceptionally similar to [VALE](https://dl.acm.org/doi/10.1145/2413176.2413185)/[mSwitch](https://dl.acm.org/doi/10.1145/2774993.2775065) that is a [netmap](https://www.usenix.org/conference/atc12/technical-sessions/presentation/rizzo)-based virtual switch, and a bit more explanation for this is found at [https://github.com/yasukata/rvs](https://github.com/yasukata/rvs). 958 | - afterward, the context of the sender guest VM returns to the default EPT context. 959 | - on the receiver side, the DPDK driver will [find the incoming packet by the vNIC index forwarded by the sender](https://github.com/yasukata/librte_pmd_rvif/blob/9fbd375484cef415ea61bb6b436d509dca707c87/main.c#L237) and let the application know it through ```rte_eth_rx_burst```, which is part of the DPDK API, invoked in the DPDK application on the guest VM. 960 | 961 | #### Files needed for elisa-app-vmnet testing 962 | 963 | For testing elisa-app-vmnet, please download the following files. 964 | 965 | ``` 966 | git clone https://github.com/yasukata/elisa-app-vmnet.git 967 | ``` 968 | 969 | ``` 970 | cd elisa-app-vmnet 971 | ``` 972 | 973 | The next step is similar to the one described in the section [Download the source code needed for elisa-app-nop](#download-the-source-code-needed-for-elisa-app-nop). 974 | 975 | ``` 976 | mkdir -p deps/elisa/lib 977 | ``` 978 | 979 | ``` 980 | mkdir -p deps/elisa/util 981 | ``` 982 | 983 | ``` 984 | git -C deps/elisa/lib clone https://github.com/yasukata/libelisa.git 985 | ``` 986 | 987 | ``` 988 | git -C deps/elisa/lib clone https://github.com/yasukata/libelisa-extra.git 989 | ``` 990 | 991 | ``` 992 | git -C deps/elisa/util clone https://github.com/yasukata/elisa-util-exec.git 993 | ``` 994 | 995 | Additionally, please download a DPDK driver for the ELISA virtual interface. 996 | 997 | ``` 998 | git -C deps clone https://github.com/yasukata/librte_pmd_rvif.git 999 | ``` 1000 | 1001 | #### Compilation of elisa-app-vmnet 1002 | 1003 | First, please build librte_pmd_rvif by the following command, which downloads the source code of the DPDK library and the virtual switch implementation [rvs](https://github.com/yasukata/rvs). For details, please see [https://github.com/yasukata/librte_pmd_rvif](https://github.com/yasukata/librte_pmd_rvif). 1004 | 1005 | ``` 1006 | make -C deps/librte_pmd_rvif 1007 | ``` 1008 | 1009 | Please compile libelisa and elisa-util-exec. 1010 | 1011 | ``` 1012 | make -C deps/elisa/lib/libelisa 1013 | ``` 1014 | 1015 | ``` 1016 | make -C deps/elisa/util/elisa-util-exec 1017 | ``` 1018 | 1019 | The following compiles the code to be executed on the manager VM. 1020 | 1021 | ``` 1022 | make -C server 1023 | ``` 1024 | 1025 | The following produces the shared library loaded onto the sub EPT context of a guest VM. 1026 | 1027 | ``` 1028 | make -C server/lib/libelisa-applib-vmnet 1029 | ``` 1030 | 1031 | At last, the following command generates the binary to be executed by a guest VM. 1032 | 1033 | ``` 1034 | make -C client 1035 | ``` 1036 | 1037 | #### ivshmem configuration on QEMU 1038 | 1039 | A virtual interface is made over the shared memory between the manager VM and a guest VM. 1040 | 1041 | To create the shared memory, we use the ivshmem feature of QEMU. 1042 | 1043 | First, please create two of 32 MB shared memory files by the following commands on the host. 1044 | 1045 | ``` 1046 | dd if=/dev/zero of=/dev/shm/rvs_shm00 bs=1M count=0 seek=32 1047 | ``` 1048 | 1049 | ``` 1050 | dd if=/dev/zero of=/dev/shm/rvs_shm01 bs=1M count=0 seek=32 1051 | ``` 1052 | 1053 | For the QEMU setup, we wish that the manager VM can see both of the shared memory files, and the guest VM 0 can see only rvs_shm00 and guest VM 1 can only access rvs_shm01. 1054 | 1055 | To achieve this setup, please add the following lines for the argument of the QEMU (```qemu-system-x86_64```) command for the manager VM, guest VM 0, guest VM 1, respectively. 1056 | 1057 | - manager VM 1058 | - /dev/shm/rvs_shm00 : ```-object memory-backend-file,size=32M,share=on,mem-path=/dev/shm/rvs_shm00,id=rvs_shm00 -device ivshmem-plain,memdev=rvs_shm00 \``` 1059 | - /dev/shm/rvs_shm01 : ```-object memory-backend-file,size=32M,share=on,mem-path=/dev/shm/rvs_shm01,id=rvs_shm01 -device ivshmem-plain,memdev=rvs_shm01 \``` 1060 | - guest VM 0 1061 | - /dev/shm/rvs_shm00 : ```-object memory-backend-file,size=32M,share=on,mem-path=/dev/shm/rvs_shm00,id=rvs_shm00 -device ivshmem-plain,memdev=rvs_shm00 \``` 1062 | - guest VM 1 1063 | - /dev/shm/rvs_shm01 : ```-object memory-backend-file,size=32M,share=on,mem-path=/dev/shm/rvs_shm01,id=rvs_shm01 -device ivshmem-plain,memdev=rvs_shm01 \``` 1064 | 1065 | 1066 | #### Assumed IP address setting 1067 | 1068 | The current implementation employs the IP address of a guest VM as the identifier of the VM. 1069 | 1070 | In this example, we assume the following IP address configuration, and please change them according to your environment. 1071 | 1072 | - manager VM: 10.100.0.10 1073 | - guest VM 0: 10.100.0.100 1074 | - guest VM 1: 10.100.0.101 1075 | 1076 | #### The command on the manager VM for elisa-app-vmnet 1077 | 1078 | Before launching the ELISA program on the manager VM, on the manager VM, please type the following command to look up the PCI IDs for the attached shared memory files. 1079 | 1080 | ``` 1081 | lspci -vvv 1082 | ``` 1083 | 1084 | The following is the example output: in our case, the manager VM sees them at ```0000:00:02.0``` and ```0000:00:03.0```. 1085 | 1086 | ``` 1087 | ... 1088 | 00:02.0 RAM memory: Red Hat, Inc. Inter-VM shared memory (rev 01) 1089 | Subsystem: Red Hat, Inc. Inter-VM shared memory (QEMU Virtual Machine) 1090 | Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx- 1091 | Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- TAbort- SERR- TAbort- SERR- TAbort- SERR-