├── README.md ├── screenshot.png ├── screenshot2.png └── timing.patch /README.md: -------------------------------------------------------------------------------- 1 | # BetterTiming 2 | This is a small project of mine aiming to improve CPU timing in KVM SVM implementation to bypass certain anti-VM checks. It saves vm-exit times and then uses them to offest TSC. 3 | 4 | ![screenshot](screenshot.png) 5 | *[Pafish](https://github.com/a0rtega/pafish) passing CPU timing checks* 6 | ![screenshot2](screenshot2.png) 7 | *[vmcheck kernel-mode driver](https://github.com/SamuelTulach/vmcheck) passing checks* 8 | 9 | **Disclaimer:** Testing was done only in an isolated environment. Doing such a change might introduce unwanted side effects to the guest OS. 10 | 11 | ## How to apply the patch 12 | You will need to recompile Linux kernel from source. If you are on a distribution that has a custom build system, it might be easier to use it for building the kernel. The patch is currently for version 5.9.2, but you should be able to easily modify it for any new version of the kernel. 13 | 14 | 1.) Download and extract kernel source 15 | ``` 16 | wget https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.9.2.tar.xz 17 | tar -xf linux-5.9.2.tar.xz 18 | cd linux-5.9.2 19 | ``` 20 | 2.) Download the patch 21 | ``` 22 | git clone https://github.com/SamuelTulach/BetterTiming 23 | mv BetterTiming/timing.patch timing.patch 24 | ``` 25 | 3.) Apply patch 26 | ``` 27 | patch -s -p0 < timing.patch 28 | ``` 29 | 4.) Build and install the kernel 30 | 31 | ## How it works 32 | The concept of this is extremely simple. I have added additional variables to `kvm_vcpu` struct. 33 | ``` 34 | u64 last_exit_start; 35 | u64 total_exit_time; 36 | ``` 37 | Then created a wrapper function around `vcpu_enter_guest` (which is where VM-exit is handled). This wrapper would then save the time it took for the vcpu to exit if the exit reason matches specified value. 38 | ``` 39 | static int vcpu_enter_guest(struct kvm_vcpu *vcpu) 40 | { 41 | int result; 42 | u64 differece; 43 | 44 | vcpu->last_exit_start = rdtsc(); 45 | 46 | result = vcpu_enter_guest_real(vcpu); 47 | 48 | if (vcpu->run->exit_reason == 123) 49 | { 50 | differece = rdtsc() - vcpu->last_exit_start; 51 | vcpu->total_exit_time += differece; 52 | } 53 | 54 | return result; 55 | } 56 | ``` 57 | Added a VM-exit handler for RDTSC instruction which takes those values. 58 | ``` 59 | static int handle_rdtsc_interception(struct vcpu_svm *svm) 60 | { 61 | u64 differece; 62 | u64 final_time; 63 | u64 data; 64 | 65 | differece = rdtsc() - svm->vcpu.last_exit_start; 66 | final_time = svm->vcpu.total_exit_time + differece; 67 | 68 | data = rdtsc() - final_time; 69 | 70 | svm->vcpu.run->exit_reason = 123; 71 | svm->vcpu.arch.regs[VCPU_REGS_RAX] = data & -1u; 72 | svm->vcpu.arch.regs[VCPU_REGS_RDX] = (data >> 32) & -1u; 73 | 74 | skip_emulated_instruction(&svm->vcpu); 75 | 76 | return 1; 77 | } 78 | ``` 79 | And of course modified `kvm_get_msr_common` to also utilise this patch. 80 | ``` 81 | case MSR_IA32_TSC: { 82 | differece = rdtsc() - vcpu->last_exit_start; 83 | final_time = vcpu->total_exit_time + differece; 84 | 85 | msr_info->data = rdtsc() - final_time; 86 | 87 | vcpu->run->exit_reason = 123; 88 | break; 89 | } 90 | ``` 91 | Then added custom exit reason to other instruction with unconditional exit reason (such as INVD and XSETBV). 92 | 93 | ## Troubleshooting 94 | - **Q:** VM does not seem to boot after the patch! 95 | - **A:** Double check if the patch was applied correctly and that your CPU supports RDTSC instruction. 96 | - **Q:** I am still getting detected by some rootkits! 97 | - **A:** Check you are on AMD processor. Check that you have CPU pinning enabled and that you have free cores that are not pinned to the VM. In case CPU pinning is already in place, you can try increasing the difference in `vcpu_enter_guest`. Also you might want to specify exit reason for other instructions then just CPUID. 98 | - **Q:** My PC does not boot after applying the patch! 99 | - **A:** The patch does not change anything unless you boot a VM. Please check that you have compiled the kernel with correct settings. 100 | - **Q:** I installed the new kernel, but I can see only blinking cursor! 101 | - **A:** If you have NVIDIA GPU, make sure that the driver did not fuck itself after installation of a custom kernel. If you are on Arch Linux for example, you can use package `nvidia-dkms` to install the proprietary driver. 102 | -------------------------------------------------------------------------------- /screenshot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SamuelTulach/BetterTiming/3d95a8fe8a9e6f072c7732e83c6cca43023aa4a7/screenshot.png -------------------------------------------------------------------------------- /screenshot2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SamuelTulach/BetterTiming/3d95a8fe8a9e6f072c7732e83c6cca43023aa4a7/screenshot2.png -------------------------------------------------------------------------------- /timing.patch: -------------------------------------------------------------------------------- 1 | diff -ruN linux-5.9.2/arch/x86/kvm/svm/svm.c linux-5.9.2-patch/arch/x86/kvm/svm/svm.c 2 | --- linux-5.9.2/arch/x86/kvm/svm/svm.c 2020-10-29 10:12:22.000000000 +0100 3 | +++ linux-5.9.2-patch/arch/x86/kvm/svm/svm.c 2020-10-30 22:38:08.893100407 +0100 4 | @@ -1041,6 +1041,7 @@ 5 | svm_set_intercept(svm, INTERCEPT_XSETBV); 6 | svm_set_intercept(svm, INTERCEPT_RDPRU); 7 | svm_set_intercept(svm, INTERCEPT_RSM); 8 | + svm_set_intercept(svm, INTERCEPT_RDTSC); 9 | 10 | if (!kvm_mwait_in_guest(svm->vcpu.kvm)) { 11 | svm_set_intercept(svm, INTERCEPT_MONITOR); 12 | @@ -2083,6 +2084,8 @@ 13 | 14 | static int wbinvd_interception(struct vcpu_svm *svm) 15 | { 16 | + svm->vcpu.run->exit_reason = 123; 17 | + 18 | return kvm_emulate_wbinvd(&svm->vcpu); 19 | } 20 | 21 | @@ -2091,6 +2094,8 @@ 22 | u64 new_bv = kvm_read_edx_eax(&svm->vcpu); 23 | u32 index = kvm_rcx_read(&svm->vcpu); 24 | 25 | + svm->vcpu.run->exit_reason = 123; 26 | + 27 | if (kvm_set_xcr(&svm->vcpu, index, new_bv) == 0) { 28 | return kvm_skip_emulated_instruction(&svm->vcpu); 29 | } 30 | @@ -2170,6 +2175,7 @@ 31 | 32 | static int cpuid_interception(struct vcpu_svm *svm) 33 | { 34 | + svm->vcpu.run->exit_reason = 123; 35 | return kvm_emulate_cpuid(&svm->vcpu); 36 | } 37 | 38 | @@ -2185,6 +2191,8 @@ 39 | 40 | static int invd_interception(struct vcpu_svm *svm) 41 | { 42 | + svm->vcpu.run->exit_reason = 123; 43 | + 44 | /* Treat an INVD instruction as a NOP and just skip it. */ 45 | return kvm_skip_emulated_instruction(&svm->vcpu); 46 | } 47 | @@ -2739,6 +2747,24 @@ 48 | return nop_interception(svm); 49 | } 50 | 51 | +static int handle_rdtsc_interception(struct vcpu_svm *svm) 52 | +{ 53 | + u64 differece; 54 | + u64 final_time; 55 | + u64 data; 56 | + 57 | + differece = rdtsc() - svm->vcpu.last_exit_start; 58 | + final_time = svm->vcpu.total_exit_time + differece; 59 | + 60 | + data = rdtsc() - final_time; 61 | + 62 | + svm->vcpu.arch.regs[VCPU_REGS_RAX] = data & -1u; 63 | + svm->vcpu.arch.regs[VCPU_REGS_RDX] = (data >> 32) & -1u; 64 | + 65 | + svm->vcpu.run->exit_reason = 123; 66 | + return nop_interception(svm); 67 | +} 68 | + 69 | static int (*const svm_exit_handlers[])(struct vcpu_svm *svm) = { 70 | [SVM_EXIT_READ_CR0] = cr_interception, 71 | [SVM_EXIT_READ_CR3] = cr_interception, 72 | @@ -2805,6 +2831,7 @@ 73 | [SVM_EXIT_RSM] = rsm_interception, 74 | [SVM_EXIT_AVIC_INCOMPLETE_IPI] = avic_incomplete_ipi_interception, 75 | [SVM_EXIT_AVIC_UNACCELERATED_ACCESS] = avic_unaccelerated_access_interception, 76 | + [SVM_EXIT_RDTSC] = handle_rdtsc_interception, 77 | }; 78 | 79 | static void dump_vmcb(struct kvm_vcpu *vcpu) 80 | diff -ruN linux-5.9.2/arch/x86/kvm/x86.c linux-5.9.2-patch/arch/x86/kvm/x86.c 81 | --- linux-5.9.2/arch/x86/kvm/x86.c 2020-10-29 10:12:22.000000000 +0100 82 | +++ linux-5.9.2-patch/arch/x86/kvm/x86.c 2020-10-30 21:29:50.866180190 +0100 83 | @@ -3163,6 +3163,9 @@ 84 | 85 | int kvm_get_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info) 86 | { 87 | + u64 differece; 88 | + u64 final_time; 89 | + 90 | switch (msr_info->index) { 91 | case MSR_IA32_PLATFORM_ID: 92 | case MSR_IA32_EBL_CR_POWERON: 93 | @@ -3232,10 +3235,12 @@ 94 | * operate L1's TSC value to ensure backwards-compatible 95 | * behavior for migration. 96 | */ 97 | - u64 tsc_offset = msr_info->host_initiated ? vcpu->arch.l1_tsc_offset : 98 | - vcpu->arch.tsc_offset; 99 | + differece = rdtsc() - vcpu->last_exit_start; 100 | + final_time = vcpu->total_exit_time + differece; 101 | 102 | - msr_info->data = kvm_scale_tsc(vcpu, rdtsc()) + tsc_offset; 103 | + msr_info->data = rdtsc() - final_time; 104 | + 105 | + vcpu->run->exit_reason = 123; 106 | break; 107 | } 108 | case MSR_MTRRcap: 109 | @@ -8368,7 +8373,7 @@ 110 | * exiting to the userspace. Otherwise, the value will be returned to the 111 | * userspace. 112 | */ 113 | -static int vcpu_enter_guest(struct kvm_vcpu *vcpu) 114 | +static int vcpu_enter_guest_real(struct kvm_vcpu *vcpu) 115 | { 116 | int r; 117 | bool req_int_win = 118 | @@ -8665,6 +8670,24 @@ 119 | return r; 120 | } 121 | 122 | +static int vcpu_enter_guest(struct kvm_vcpu *vcpu) 123 | +{ 124 | + int result; 125 | + u64 differece; 126 | + 127 | + vcpu->last_exit_start = rdtsc(); 128 | + 129 | + result = vcpu_enter_guest_real(vcpu); 130 | + 131 | + if (vcpu->run->exit_reason == 123) 132 | + { 133 | + differece = rdtsc() - vcpu->last_exit_start; 134 | + vcpu->total_exit_time += differece; 135 | + } 136 | + 137 | + return result; 138 | +} 139 | + 140 | static inline int vcpu_block(struct kvm *kvm, struct kvm_vcpu *vcpu) 141 | { 142 | if (!kvm_arch_vcpu_runnable(vcpu) && 143 | diff -ruN linux-5.9.2/include/linux/kvm_host.h linux-5.9.2-patch/include/linux/kvm_host.h 144 | --- linux-5.9.2/include/linux/kvm_host.h 2020-10-29 10:12:22.000000000 +0100 145 | +++ linux-5.9.2-patch/include/linux/kvm_host.h 2020-10-30 21:30:33.539340889 +0100 146 | @@ -286,6 +286,9 @@ 147 | unsigned int halt_poll_ns; 148 | bool valid_wakeup; 149 | 150 | + u64 last_exit_start; 151 | + u64 total_exit_time; 152 | + 153 | #ifdef CONFIG_HAS_IOMEM 154 | int mmio_needed; 155 | int mmio_read_completed; 156 | diff -ruN linux-5.9.2/.vscode/settings.json linux-5.9.2-patch/.vscode/settings.json 157 | --- linux-5.9.2/.vscode/settings.json 1970-01-01 01:00:00.000000000 +0100 158 | +++ linux-5.9.2-patch/.vscode/settings.json 2020-10-30 21:29:37.269606033 +0100 159 | @@ -0,0 +1,5 @@ 160 | +{ 161 | + "files.associations": { 162 | + "ioapic.h": "c" 163 | + } 164 | +} 165 | \ No newline at end of file 166 | --------------------------------------------------------------------------------