VMware is a proprietary, closed-source, reduced privilege guest virtual machine emulator. It supports guest-to-host and host-to-guest communication. Since it relies on the underlying hardware for execution of instructions, it must relocate sensitive data structures, such as the Interrupt Descriptor Table and the Global Descriptor Table. VMware also makes use of the Local Descriptor Table which is not otherwise used by Windows. Thus, a simple detection method for VMware is to check for a non-zero Local Descriptor Table (LDT) base on Windows.

RedPill

The more common method for detecting VMware is to check the value of the Interrupt Descriptor Table, using the “RedPill” method. For the “RedPill” method, if the value of the Interrupt Descriptor Table base exceeds a certain value, a virtual machine emulator is assumed to be present. However, as the LDT paper shows, this method is unreliable on machines with multiple CPUs.

Scooby Doo

The “Scooby Doo” method uses the same basic idea as the RedPill method but it compares the Interrupt Descriptor Table base value to specific hard-coded values in order to identify VMware specifically. While the Scooby Doo method is less likely to trigger false positives, compared to the RedPill method, there is still the chance that some false positives will occur.

Source: Peter Ferrie