Hooking an SGX ENCLS Leaf Function Call from KVM
Contents

Environment

1. ENCLS

1
2
3
4
5
6
...
if (in VMX non-root operation) and (Enable_ENCLS_EXITING = 1):
    if ((EAX < 63) and (ENCLS_EXITING_Bitmap[EAX] = 1)) or ((EAX > 62) and (ENCLS_EXITING_Bitmap[63] = 1)):
        set VMCS.EXIT_REASON = ENCLS;
        deliver VM exit;
...

2. Secondary Processor-based VM-Execution Control (IA32_VMX_PROCBASED_CTLS2)

3. ENCLS-exiting Bitmap

4. Implementing A ENCLS Hook on KVM

/linux/arch/x86/kvm/vmx.c:3269 static __init int setup_vmcs_config(struct vmcs_config *vmcs_conf)
/linux/arch/x86/kvm/vmx.c:4936 static int vmx_vcpu_setup()

(1) In setup_vmcs_config() (/linux/arch/x86/kvm/vmx.c:3269).

In terms of configuring secondary processor-based VM-exeuction controls MSR,

(2) In vmx_vcpu_setup() (/linux/arch/x86/kvm/vmx.c:4939).

Set an ENCLS0-exiting bitmap VMCS control field (encoding 0x0000202E). This indicates which ENCLS instruction should be hooked by a VMM. It is not set by default. Hence, set the VMCS control field as follows to hook an ENLS instruction call.

Note that each bit position of the control field represents the leaf number of ENCLS instruction, i.e. for EEXTEND, which has leaf number 0x06, 6th bit should be set.
Also note that ENCLS-exiting bitmap control field (0x0000202E) is not defined in Linux kernel by default. You also have to set it first, in /linux/arch/x86/kvm/include/asm/vmx.h.

1
2
3
4
5
6
7
8
9
/linux/arch/x86/kvm/include/asm/vmx.h
enum vmcs_field {
    VIRTUAL_PROCESSOR_ID        = 0x00000000,
    ...
    XSS_EXIT_BITMAP_HIGH        = 0x0000202D,
    ENCLS_EXITING_BITMAP        = 0x0000202E,
    ENCLS_EXITING_BITMAP_HIGH   = 0x0000202F,
    ...
};
1
2
3
4
5
6
7
8
9
/linux/arch/x86/kvm/vmx.c
...
if (cpu_has_secondary_exec_ctrls()) {
    vmcs_write32(SECONDARY_VM_EXEC_CONTROL, vmx_secondary_exec_control(vmx));

    // My custom ENCLS instruction has leat number 0x1A, hence set 26th bit as 1.
    vmcs_write64(ENCLS_EXITING_BITMAP, (1 << 26));
}
...

If you want to hook multiple ENCLS instructions, the value passed should be a or-ed combination of several numbers.

(3) In vmx_handle_exit() (/linux/arch/x86/kvm/vmx.c:8314).

Now the KVM will hook an ENCLS instruction that is set to be hooked. However, as the exit reason ENCLS is not included in Linux kernel by default, it prints vmx: unexpected exit reason 0x3C in dmesg.

Add an ENCLS exit reason in /linux/arch/x86/include/uapi/asm/vmx.h, as follows.

1
2
3
4
5
#define EXIT_REASON_EXCEPTION_NMI     0
...
#define EXIT_REASON_INVPCID           58
#define EXIT_REASON_ENCLS             60
...

To see the entire list of vm exit reasons, refer to Intel 64 and IA-32 Architectures Software Developer’s Manual Volume 3, Appendix C.

After adding the definition of ENCLS exit reason, implement an exit reason handler in /linux/arch/x86/kvm/vmx.c.
The list of vmx exit handlers is defined in /linux/arch/x86/kvm/vmx.c:7713. Add a name of handler at the last of the list as follows.

1
2
3
4
5
6
static int (* const kvm_vmx_exit_handlers[])(struct kvm_vcpu * vcpu) = {
    [EXIT_REASON_EXCEPTION_NMI]         = handle_exception,
    ...
    [EXIT_REASON_PCOMMIT]               = handle_pcommit,
    [EXIT_REASON_ENCLS]                 = handle_encls,
};

And implement a handler named handle_encls(), as follows.

1
2
3
4
5
6
static int handle_encls (struct kvm_vcpu *vcpu) {
    skip_emulated_instruction(vcpu);
    printk("%s: handling encls exit reason from vcpu%d\n"),
          __func__, vcpu->vcpu_id);
    return 1;
}

As commented right above the declaration of kvm_vmx_exit_handlers, the exit handlers should return 1 if the exit was handled fully and guest execution may resume.
Also I added skip_emulated_instruction() call because vmx_handle_exit() is called indefinitely without this function call.
Currently I just added a simple printk(), but you can implement it as you want.

If a guest virtual machine calls the following __eabc() function,

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
#define __encls(rax, rbx, rcx, rdx...)  \
    ({              \
    int ret;            \
    asm volatile("1: .byte 0x0f, 0x01, 0xcf;\n\t"   \
             " xor %%eax,%%eax;\n"      \
             "2: \n"                    \
             ".section .fixup,\"ax\"\n"         \
             "3: movq $-1,%%rax\n"          \
             "   jmp 2b\n"              \
             ".previous\n"              \
             _ASM_EXTABLE(1b, 3b)           \
             : "=a"(ret), "=b"(rbx), "=c"(rcx)      \
             : "a"(rax), "b"(rbx), "c"(rcx), rdx    \
             : "memory");               \
    ret;    \
    })

static inline int __eabc (void) {
    unsigned long rbx = 0;
    unsigned long rcs = 0;
    // The value of the RAX register will be 0x1A, the leaf number of a custom ENCLS isntruction EABC.
    return __encls(0x1A, rbx, rcx, "d"(0));
}

The result from the VMM dmesg is as follows.
handle_exit_reason_encls

References