Since my post last week, a good amount of progress has been made. The original creator of IntroVirt (Steve!) is back on to help, quickly updating the KVM patch for a much newer Linux kernel (6.18) as well as getting started with testing on Windows 11 (which turns out to kind-of work). We should be seeing a push or PR for those changes shortly.

While he’s been doing that, I’ve been working on some cleanup, testing, bug fixes, automation, and patch improvements. libmspdb has a new version with some cleanup, and bug fixes as well as automated builds, testing, and releases using GitHub actions. kvm-introvirt has a bunch of fixes to the patch, some cleanup, automation in GitHub actions and support for 2 Proxmox (PVE) kernels. We also have a discord now and a roadmap!! It’s been a busy couple of weeks.

Finishing The VMCALL Example

As I was going through, cleaning things up, and planning out next steps, I noticed an examples directory in the IntroVirt repository with 2 example. One practically blank, and the other 99% complete. So naturally, we delete the blank one as if it never existed, and finish the other one: vmcall_interface.cc. I don’t want to just finish it and move on though, I want to make it cooler.

What is the point of vmcall_interface.cc? It’s probably best to understand it first. The vmcall_interface.cc example demonstrates how to create an IntroVirt tool that can receive commands from processes in the guest to perform actions on behalf of those processes that wouldn’t normally be possible without VMI. For example, you could make a command to elevate an unprivileged guest process to admin while completely bypassing all security controls. Or you could give guest processes a way to request additional protection from the hypervisor: preventing termination, or debugging. We’re going to go with the later for this example.

The fundamental thing at play here is the vmcall instruction. It’s a regular old assembly instruction like any other, but when it runs, it triggers a VM exit and then the hypervisor decides what to do. Hyper-V has them, KVM has them, Xen has them too, they all have them. What makes this so powerful is that we can write a tool that enables custom handling of vmcall instructions that work at any privilege level without recompiling or modifying the hypervisor (since the kvm patch for IntroVirt exposes that functionality already).

Let’s start with the in-guest component. We just need a simple user-mode application in C with an assembly stub to perform the vmcall instruction. Here’s a simple assembly stub that implements 2 capabilities (the one on GitHub has way more comments and features):

.code
; Reverse a NULL-terminated string
HypercallReverseCString PROC
    mov rax, 0FACEh
    mov rdx, rcx  
    mov rcx, 0F000h   
    vmcall           
    ret
HypercallReverseCString ENDP

; Protect a process from termination, debug, and modification
HypercallProtectProcess PROC
    mov rax, 0FACEh
    mov rcx, 0F002h
    vmcall
    ret
HypercallProtectProcess ENDP

Then we can use it with a little C program:

#include <stdio.h>
#include <stdint.h>
#include <string.h>
#include <windows.h>

extern uint64_t HypercallReverseCString(char *c_str);
extern uint64_t HypercallProtectProcess();

int main(int argc, char** argv) {
    uint64_t status = 0;
    char test_str[] = "Hello, IntroVirt!";

    printf("Original string: %s\n", test_str);
    status = HypercallReverseCString(test_str);
    if (status == 0) {
        printf("Reversed string: %s\n", test_str);
    } else {
        printf("Failed. Status code: %llu\n", status);
    }

    status = HypercallProtectProcess();
    if (status == 0) {
        while (1) {
            printf("This process is protected\n");
            Sleep(2000);
        }
    } else {
        printf("Failed. Status code: %llu\n", status);
    }
    return 0;
}

So now let’s explain what’s happening here. In our vmcall_interface tool we expose “service codes”. These are the “things” we can do and they map directly to the two assembly functions from above. We can reverse a null-terminated C-string and protect the calling process from debugging, termination, or injection. The service codes are defined in vmcall_interface.cc like this:

enum IVServiceCode { CSTRING_REVERSE = 0xF000, PROTECT_PROCESS = 0xF002 };

We can make as many service codes as we want, and we can implement them however we want. For this simple example we just use an enum to define our codes as arbitrary values. It is the responsibility of the in-guest process to execute the vmcall instruction with the appropriate service codes.

The complete version of vmcall_interface will have more service codes and the in-guest components may be more complicated. These snippets simply illustrate the core of what’s going on. Any additional service codes or functionality is just more of the same with tweaks to perform different actions or add additional protections.

To make a vmcall we simply need to get the vmcall instruction to run. That’s what our assembly stub is for. Then we need to pass in our service code and any other arguments in appropriate CPU registers. So, if that’s all, how does IntroVirt know about our vmcall? Well, there’s one more piece. We need to set a register to a special constant value that let’s IntroVirt know this vmcall is an event that should be sent to an IntroVirt tool instead of the default logic of the KVM hypervisor. We can see a simplified version of this logic in the kvm-introvirt KVM patch:

int kvm_emulate_hypercall(struct kvm_vcpu *vcpu)
{
    unsigned long nr, a0, a1, a2, a3, ret;
    int op_64_bit;
    nr = kvm_rax_read(vcpu);
    if(nr == 0xFACE) {
        const uint64_t original_rip = kvm_rip_read(vcpu);
        if (vcpu->vmcall_hook_enabled) {
            kvm_deliver_vmcall_event(vcpu);
        }
        ++vcpu->stat.hypercalls;
        if (original_rip == kvm_rip_read(vcpu)) {
            return kvm_skip_emulated_instruction(vcpu);
        }
        return 0;
    }
//...the rest of kvm_emulate_hypercall() is just the normal
// non-introvirt code for handling hypercalls in KVM.

So we can see here, if we ensure the RAX register is set to 0xFACE when we make our vmcall then the handling will divert away from stock KVM into IntroVirt. In addition to the 0xFACE value we also need to pass in the service code and any arguments for the service we’re implementing. To reverse a C-String, we just need the service code and the pointer to the string. For the process protection we just need the service code. If we look at vmcall_interface.cc we can see the service code is expected to be in the RCX register.

switch (regs.rcx()) {
case CSTRING_REVERSE:
    return_code = service_string_reverse(event);
    break;
case PROTECT_PROCESS:
    return_code = service_protect_process(event);
    break;
}

and for the service_string_reverse we expect the pointer to the string to be in the RDX register. And with that, we now have enough information to actually understand the assembly stub from earlier:

.code
; Reverse a NULL-terminated string
HypercallReverseCString PROC
    mov rax, 0FACEh
    mov rdx, rcx  
    mov rcx, 0F000h   
    vmcall           
    ret
HypercallReverseCString ENDP

; Protect a process from termination, debug, and modification
HypercallProtectProcess PROC
    mov rax, 0FACEh
    mov rcx, 0F002h
    vmcall
    ret
HypercallProtectProcess ENDP

In both cases we put 0xFACE (0FACEh in this assembly syntax) in the RAX register. Then we put the service codes 0xF000 and 0xF002 in RCX. The only catch is in HypercallReverseCString where we do mov rdx, rcx. This requires some understanding of calling conventions in Windows, but to skip the full explanation, when we call HypercallReverseCString(test_str), the address of test_str ends up in RCX, and since we actually need that to be in RDX and the service code to be in RCX, we do a quick switch to re-arrange things so it’s all prepped for the vmcall.

As we’ve seen up to this point, when the hypervisor sees these vmcall instructions with those values in those registers, and event will be sent to any running IntroVirt tools attached to that guest VM. In this case, it will be our vmcall_interface tool which will parse the service code in the switch, case above and call one of service_string_reverse or service_protect_process, both of which are fairly straightforward:

int service_string_reverse(Event& event) {
    auto& vcpu = event.vcpu();
    auto& regs = vcpu.registers();
    try {
        // RDX has a pointer to the string to reverse
        guest_ptr<void> pStr(event.vcpu(), regs.rdx());

        // Map it and reverse it
        guest_ptr<char[]> str = map_guest_cstring(pStr);
        reverse(str.begin(), str.end());
    } catch (VirtualAddressNotPresentException& ex) {
        cout << ex;
        return -1;
    }
    cout << '\t' << "String reversed successfully\n";
    return 0;
}

int service_protect_process(Event& event) {
    auto& task = event.task();
    lock_guard lock(mtx_);
    protected_pids_.insert(task.pid());
    return 0;
}

The service_reverse_string function is the simplest example since it runs and completes everything it needs to do right there and when it returns the string is reversed.

The service_protect_process is harder to show in one snippet since it involves tracking the process and then monitoring system calls. In the snippet above we add the PID to our protected_pids_ variable. Then we need to handle the NtOpenProcess system call:

case SystemCallIndex::NtOpenProcess: {
    auto* handler = static_cast<nt::NtOpenProcess*>(wevent.syscall().handler());
    auto desired_access = handler->DesiredAccess();
    auto* client_id = handler->ClientId();
    const uint64_t target_pid = client_id->UniqueProcess();

    lock_guard lock(mtx_);
    if (protected_pids_.count(target_pid)) {
        if (desired_access.has(nt::PROCESS_TERMINATE) ||
            desired_access.has(nt::PROCESS_VM_WRITE) ||
            desired_access.has(nt::PROCESS_VM_OPERATION) ||
            desired_access.has(nt::PROCESS_CREATE_THREAD) ||
            desired_access.has(nt::PROCESS_CREATE_PROCESS) ||
            desired_access.has(nt::PROCESS_SET_INFORMATION))
        {    
            handler->ClientIdPtr(guest_ptr<void>());
        }
    }
    break;
}

NtOpenProcess is the precursor to basically anything that can be done to a process. It’s not possible to debug a process, read its memory, write its memory, terminate it, inject into it, or anything without first opening a handle to it. So this snippet shows that all we need to do is look for processes opening our protected process, check the access rights they are requesting, and if we don’t like it, simply change the ClientId paramter to a NULL pointer, which will result in an invalid parameter at the kernel level and the call will fail. Once that’s handled, we just have a snippet for catching terminate and we’re basically done:

case SystemCallIndex::NtTerminateProcess: {
    auto* handler = static_cast<nt::NtTerminateProcess*>(wevent.syscall().handler());
    if (!handler->will_return() || handler->target_pid() == wevent.task().pid()) {
        lock_guard lock(mtx_);
        protected_pids_.erase(wevent.task().pid())
        break;
    } else {
        lock_guard lock(mtx_);
        if (protected_pids_.count(handler->target_pid()) != 0) {
            // This process is protected. Deny termination
            handler->ProcessHandle(0xFFFFFFFFFFFFFFFF);
            break;
        }
    }
}

This way the process can still exit on it’s own but any other process trying to terminate it will fail. We achieve this by intercepting the NtTerminateProcess call and setting the process handle to an invalid handle value which will cause the call to fail. We check for self-terminate by checking for NtTerminateProcess calls that won’t return or who’s PID match the target process. Let’s see it in action:

The process can run and exit on its own, but trying to kill it in task manager fails. And, something unexpected happened while I was recording, it looks like MsMpEng.exe attempted to open the process and we blocked it. That’s kind-of funny since that’s Microsoft’s malware protection service. We really do have a lot of power to do whatever we want from the hypervisor.

Each one of the available projects is fairly outdated and/or lacking in one way or another. The core issue of the whole thing is just lack of mainline kernel adoption of VMI into KVM. I don’t know if I’m the right person to start that journey, I don’t know if I have the time….I’m not even sure I care enough. But I have some time today to start kicking some tires and I’m using this blog post as a place to keep notes more than anything else. I may get somewhere with this, I may not, I may abruptly stop and never return, but I started typing these paragraphs when I ran make -j$(nproc) bindeb-pkg and it’s STILL RUNNING! So we’re off to the races! ¯\_(ツ)_/¯ I don’t know what I’m trying to say. Let’s start with what I want:

In a perfect world:

• VMI functionality would be adopted into the mainline kernel, enabled by some config like KVM_INTROSPECTION
• VMI would be supported on 64-bit Intel, AMD, and ARM architectures (do we need 32-bit too?)
• VMI WOULDN’T be some half-completed academic project for research – and it would actually be some production/commercial ready thing.
• A user-land abstraction layer (WITH Python bindings) would exist to make tasks like getting system call traces, setting breakpoints, and reading and writing memory as easy as:

import vmi

def cb(*args):
    print("do syscall stuff idk...but the API should be stupid easy")

with vmi.attach("win10-x64") as iface:
    iface.set_syscall_hook(cb)
    iface.run_forever()  # catch cntr+c and exit or something...idk

Why though? (my kernel make command is still running by the way)

Am I the only one that thinks VMI is like the coolest thing ever? It enables you to do whatever you want to a guest and there’s not much the guest can do about it. You could make an out-of-guest debugger that is impossible to detect. You could strip encryption off practically anything (I’m thinking malware/ransomware). Imagine if we had a robust VMI library and ARM64 support. You could run Android in a VM, install your favorite app, and strip all of its protections and security and figure out how it really works! It’s a reverse engineers dream for anything from hacking to malware analysis with the right tool-set. With GPU pass-through and the right PV drivers you could even write undetectable cheats for games that run outside the guest. The possibilities are endless.

So with all that said, let’s start trying to revive VMI as best we can. I have some experience working on IntroVirt, so I’ll start there. The IntroVirt KVM patch is much smaller than kvmi so it’ll be an easier starting point. It doesn’t fully support AMD and doesn’t support ARM at all, but those are problems for future me (or possibly someone else….possibly YOU).

Start Simple

We’re going to start by identifying the latest kernel supported by kvm-introvirt (6.8.0-41) and try to build that kernel, unmodified on a fresh install of Ubuntu 24.04.3. But instead of building the Ubuntu kernel, we’ll go straight to the kvm source and work from there. The goal (I think) is to migrate away from maintaining a kernel patch for the Ubuntu kernel and instead maintain (and maybe someday get merged in) a patch in the most vanilla Linux kernel we can and then go from there. If we start from the most vanilla place, and distribute whole pre-built kernels and things – we can maybe get more people using it and can more easily port our patch into Ubuntu, Arch Linux, Proxmox, and others since those are all also based in part on the vanilla Linux kernel. It just feels like the right place to be.

So let’s go. Here’ some stuff I always install:

sudo apt-get install -y make cmake build-essential git vim tmux

And then we need some things to build the kernel

sudo apt-get install -y bc fakeroot flex bison libelf-dev libssl-dev dwarves debhelper

I like to keep things in a ~/git folder. Let’s put kvm there:

mkdir ~/git
cd ~/git
git clone git://git.kernel.org/pub/scm/virt/kvm/kvm.git
cd ./kvm
# Latest kvm-introvirt patch is for 6.8.0, this was the closest tag
git checkout tags/kvm-6.8-1

Now let’s copy our running kernel’s config as a starting point and then make sure some settings are on/off (I’m basing some of this on the kvm-vmi setup instructions with modifications for Ubuntu 24.04 and the fact that I’m not actually building the kvmi kernel.

# Copy our config
cp /boot/config-$(uname -r) .config
# disable kernel modules signature
./scripts/config --disable SYSTEM_TRUSTED_KEYS
./scripts/config --disable SYSTEM_REVOCATION_KEYS
# enable KVM
./scripts/config --module KVM
./scripts/config --module KVM_INTEL
./scripts/config --module KVM_AMD
# Set a version str so we can see it in grub easier as ours
./scripts/config --set-str CONFIG_LOCALVERSION -kvm6.8-1
# Stuff I disabled b/c of warnings like this:
# .config:11148:warning: symbol value 'm' invalid for ANDROID_BINDERFS
./scripts/config --disable ANDROID_BINDERFS
./scripts/config --disable ANDROID_BINDER_IPC
./scripts/config --disable SERIAL_SC16IS7XX_SPI
./scripts/config --disable SERIAL_SC16IS7XX_I2C
./scripts/config --disable HAVE_KVM_IRQ_BYPASS

Now let’s start the build and while it runs you should have time to live out the rest of your life and pass away.

make olddefconfig
make -j$(nproc) bindeb-pkg

Then, if we’re still living, we can install it

sudo dpkg -i ../linux-image-6.7.0-rc7-kvm6.8-1*deb

Let’s also update our grub settings so we get a menu at boot. This kernel won’t be the default when we reboot.

# Edit GRUB_TIMEOUT_STYLE to be menu
# Edit GRUB_TIMEOUT to be something like 5 or 10
sudo vim /etc/default/grub
sudo update-grub

Finally we can reboot (and make sure to pick the kernel we just built…make sure it boots and the computer works):

sudo reboot

And we’re back. Let’s confirm our kernel version and see if kvm works:

user@user-XPS-13-9340:~$ uname -r
6.7.0-rc7-kvm6.8-1+
user@user-XPS-13-9340:~$ sudo modprobe kvm-intel
user@user-XPS-13-9340:~$ sudo lsmod | grep kvm
kvm_intel             475136  0
kvm                  1409024  1 kvm_intel
irqbypass              12288  1 kvm

Yes? Seems so. Let’s make a Windows 10 VM (download an ISO from Microsoft).

sudo apt-get install -y virt-manager
sudo systemctl daemon-reload
sudo systemctl start libvirtd
sudo usermod -a -G libvirt $USER
newgrp libvirt
virt-manager

Using the UI, make a VM for the downloaded Windows 10 ISO and we’ll see that it boots at all and that will be good enough.

Huzzah! Good enough! I’ll go ahead and install Windows 10 just so I have a VM ready. But I think it’s safe to say this kernel works fine (at least for stock KVM). Now let’s see if the kvm-introvirt patch applies cleanly for this kernel version.

RECAP!

What have we done so far? – We built an older Linux kernel and booted into it. (Woooooow!)

What have we done so far for VMI? Nothing.

SecureBoot

All of the above assumes SecureBoot is off, since we didn’t sign the kernel we built. If you want SecureBoot on, let’s see what we can do about that (following this guide).

Change to your home directory and create a file called mokconfig.cnf with the contents (If you want to not store these files in the home directory then just make sure $HOME is set and remove the HOME line from the config file below):

# This definition stops the following lines failing if HOME isn't
# defined.
HOME                    = .
RANDFILE                = $ENV::HOME/.rnd 
[ req ]
distinguished_name      = req_distinguished_name
x509_extensions         = v3
string_mask             = utf8only
prompt                  = no

[ req_distinguished_name ]
countryName             = <YOURcountrycode>
stateOrProvinceName     = <YOURstate>
localityName            = <YOURcity>
0.organizationName      = <YOURorganization>
commonName              = Secure Boot Signing Key
emailAddress            = <YOURemail>

[ v3 ]
subjectKeyIdentifier    = hash
authorityKeyIdentifier  = keyid:always,issuer
basicConstraints        = critical,CA:FALSE
extendedKeyUsage        = codeSigning,1.3.6.1.4.1.311.10.3.6
nsComment               = "OpenSSL Generated Certificate"

Replace YOUR* with appropriate values. Then run:

# Make keys
openssl req -config ./mokconfig.cnf \
        -new -x509 -newkey rsa:2048 \
        -nodes -days 36500 -outform DER \
        -keyout "MOK.priv" \
        -out "MOK.der"
# Convert to PEM
openssl x509 -in MOK.der -inform DER -outform PEM -out MOK.pem

At this point, I’m going to reboot, enable secure boot, and boot into the stock Ubuntu kernel (not the one I built). And then I’ll continue.

Import the DER

# Choose any password, it's just to confirm key selection
sudo mokutil --import MOK.der

Restart the system and at the blue screen of the MOKManager, select “Enroll MOK” and then “View key”. Confirm the key, continue, enter the password you just chose, and boot.

Confirm the new key is there:

sudo mokutil --list-enrolled

Sign the kernel, make a copy of initrd for the signed kernel, and update-grub (re-do this step only when you have a new kernel installed to sign).

sudo sbsign --key MOK.priv --cert MOK.pem /boot/vmlinuz-6.7.0-rc7-kvm6.8-1+ --output /boot/vmlinuz-6.7.0-rc7-kvm6.8-1+.signed
sudo cp /boot/initrd.img-6.7.0-rc7-kvm6.8-1+{,.signed}
sudo update-grub

Reboot and at the grub menu, select the signed kernel and it should boot.

Applying a patch

Now that we’ve accomplished basically nothing for VMI, let’s see if we can apply the kvm-introvirt patch to the stock Linux kernel for the same version (I literally cannot imagine a scenario where this doesn’t work).

Start by installing quilt which we’ll need to apply the patch

sudo apt-get install -y quilt

Now we’ll need to clone kvm-introvirt and change to the folder containing the most up-to-date patch.

cd ~/git
git clone git@github.com:IntroVirt/kvm-introvirt.git
cd ~/git/kvm-introvirt/ubuntu/noble/hwe/6.8.0-41-generic

We are now existing inside of the folder with the most up-to-date patch for Ubuntu 24.04. Do I like how kvm-introvirt got restructured? No I do not. Blame me…it was me. But anyway, we now have to either move the kvm git repo to this folder we’re in now, or re-clone all of kvm here and checkout the right branch. I tried a symlink and quilt did not apply the patch. So anyways, let’s move it in:

mv ~/git/kvm ~/git/kvm-introvirt/ubuntu/noble/hwe/6.8.0-41-generic/kernel

Then we can try to apply the patch

quilt push -a

It WORKED! (I hope). It still needs to compile and then run.

So let’s do that. We’ve already done this so I won’t go into verbose detail:

# Remember we moved the kvm repo and named it kernel
# you could put it back now if you want
cd ./kernel
# So we can distinguish our kernel
./scripts/config --set-str CONFIG_LOCALVERSION -introvirt

# Make and install
make olddefconfig
make -j$(nproc) bindeb-pkg
cd ..
sudo dpkg -i linux-image-6.7.0-rc7-introvirt+*deb

# Sign for SecureBoot (if we have that on)
cd ~
sudo sbsign --key MOK.priv --cert MOK.pem /boot/vmlinuz-6.7.0-rc7-introvirt+ --output /boot/vmlinuz-6.7.0-rc7-introvirt+.signed
sudo cp /boot/initrd.img-6.7.0-rc7-introvirt+{,.signed}
sudo update-grub

# Reboot and select the signed IntroVirt kernel at grub
sudo reboot

What a ride. Now we actually have to install IntroVirt. Which we can do from source pretty easily. I’ll go quick:

# Clone
cd ~/git
git clone git@github.com:IntroVirt/libmspdb.git
git clone git@github.com:IntroVirt/IntroVirt.git

# MS PDB
cd ./libmspdb/build
sudo apt-get install -y cmake libcurl4-openssl-dev libboost-dev git
cmake ..
make -j$(nproc) package
sudo apt install ./*.deb

# IntroVirt
cd ~/git/IntroVirt/build
sudo apt-get install -y python3 python3-jinja2 cmake \
    make build-essential libcurl4-openssl-dev \
    libboost-dev libboost-program-options-dev \
    git clang-format liblog4cxx-dev libboost-stacktrace-dev \
    doxygen liblog4cxx15
cmake ..
make -j$(nproc) package
sudo apt install ./*.deb

Hopefully that all goes well and now we can boot up our VM and test things out.

# Show IntroVirt installed and KVM recongnized
sudo ivversion
# Get guest info (OS version etc...)
sudo ivguestinfo -Dwin10
# Systemcall trace
sudo ivsyscallmon -Dwin10

GREAT SUCCESS! And that’s it for today. The state of VMI is right where I left it. Hopefully I pick this up against next week.