INSTRUCTIONS BY VERCETY,

ORIGINAL POST: https://forums.anandtech.com/threads/what-controls-turbo-core-in-xeons.2496647/

MANY THANKS TO “THE STILT” & “DUFUS” FROM ANANDTECH.COM

LET’S GO:

0.- Windows has to be in UEFI mode, doesnt work on Legacy mode. Format it if necessary.

Also, CLEAR CMOS your BIOS and reset it to default, just in case.

1. Download latest bios for your motherboard

2. Open downloaded bios with UBU (Uefi Bios Updater).

- Right click on UBU.dat, open as Admin:

- Press “7”

- Press “1”

- Select any value except “0/skip” for Broadwell-E, as this forces the update (I chose the latest
microcode)

- Select “0/skip” for Haswell-E (this causes there to be no microcode for V3s).

- Press “”Enter”

- Press “0/exit”

- If ASUS bios, it will ask you to rename the bios to Asus Flashback Compatible, accept!

4.- Use a recently formated USB stick (better if its usb 2.0, 32gb or less, and use it in a usb 2.0 slot at the
back of your motherboard):

To format clean and in GPT:

Open WIN+R > CMD

1. diskpart

2. list disk
3. Select the drive, and reformat it: select disk <disk number>
4. clean
5. convert gpt
6. exit

7. Close the command prompt window.

4. Flash Modded bios

- Instructions from this post for any mobo (http://www.win-raid.com/t455f16-Guide-How-to-flash-a-

modded-ASUS-ASRock-Gigabyte-AMI-UEFI-BIOS.html)
 - For MY ASUS Bios: Flashing an ASUS BIOS file with the extension ".CAP"
These types of ASUS' BIOS files are capsuled and write protected, which makes it
impossible to flash a modded .CAP BIOS the usual way.
There are at least 2 options how to solve this problem:

 a) Using the ASUS "USB Flashback" procedure

Many ASUS mainboards with an Intel 7- 8- or 9-Series chipset do support this very easy
to use feature to get a modded BIOS flashed. These boards have a special "Flashback
USB port" and a special "USB Flashback button". For details please look into your
mainboard manual or at >this< ASUS site (maybe the Smart Redirect Addon has to be
disabled).
Important for ASUS X99 Boards:
Users of an ASUS X99 mainboard should read >this< post written by sinders, before they
start flashing any modded BIOS by using the "USB Flashback" method.
Procedure:
1. Rename the modded BIOS file according to ASUS "USB BIOS Flashback Rules". The ASUS
support is offering a tool named "BIOS Renamer for USB BIOS Flashback" for all
mainboards, which support the USB Flashback feature. Furthermore the UBU tool offfers
the renaming procedure as well.
2. Copy the modded and renamed BIOS file onto a FAT32 formatted empty USB Flash drive
and insert it into the special "USB Flashback port".
3. Shut down the computer, but don't power it off.
4. Hit the USB BIOS Flashback button. A LED light will start blinking. Press the button for
some seconds (LED light should have blinked 3-4 times).
5. Wait until the LED light doesn't blink anymore. This means, that the BIOS has been
successfully flashed into the BIOS chip of the mainboard.
6. Then all is done. You can power off the computer and remove the USB Flash drive.

4. Installing the V3.EFI driver:

(Some BIOS already incorporate the UEFI shell as part of the BIOS but if we don't have that then
we can download one and put it on a FAT32 USB flash drive or other suitable boot medium and
boot that.)

- Download the shell (shell.efi).

https://github.com/tianocore/edk2/tree/master/ShellBinPkg/UefiShell/X64

- Rename shell.efi to BOOTx64.EFI then create a root folder named EFI and sub folder of that
named BOOT and copy the BOOTx64.EFI to there.

- Download 'V3.EFI' (V3.zip 633bytes) which can be copied to the previously formatted FAT32
USB flash drive. clear cmos and load defaults in your bios if you haven’t already.
https://www.sendspace.com/file/ck1mlr

- Copy V3.EFI to the ROOT folder of the USB.

- Get into the Bios, and Disable Secure Boot to be able to boot into the USB. Info for ASUS mobos
(http://www.technorms.com/45538/disable-enable-secure-boot-asus-motherboard-uefi-bios-
utility)
- Using the BIOS boot manager (F11 during BIOS boot on Asrock, F8 on main Menu in ASUS) we
select the flash drive with 'UEFI:' prefix.

- Here we press 'ESC' after shell is loaded to stop the 'startup.nsh' from running. 'startup.nsh' is
similar to the old DOS '.BAT' file, just a way of automating things on shell startup.
-

- Our USB flash has been mounted as 'FS0:' as shown in the mapping table. Being the only USB
device makes it an easy giveaway. So now we can test 'V3.EFI' which was copied earlier to the
USB flash drive root by typing 'load fs0:\V3.EFI'. Hint, the TAB key can be used for auto-
completion so sometimes we can save on a lot of typing. One should see 'V3 - All Turbo Set' if
successful, something else if not. In this instance I've finished of with the 'exit' command
 which takes us back to the BIOS boot manager so we can select the 'Windows OS'.
Alternatively we could have directly run Windows from the shell. For this system the Windows
system partition is FS1: so being GPT typing "FS1:\EFI\Microsoft\Boot\BOOTMGFW.EFI" would
do the trick.

- If we are happy with the driver and want to keep it we can get it to automatically load by
placing it on the EFI system drive.In this instance I have copied V3.EFI from the USB flash drive
'FS0:' to the EFI system boot folder on 'FS1:' 'cp fs0:\V3.EFI fs1:\EFI\Boot'. Now using the shell
boot configuration command we can add it to be executed before any OS with shell command
'bcfg driver add 0 fs1:\EFI\BOOT\V3.EFI "V3 Full Turbo"'. After that type 'reset' to restart the
PC as the EFI driver has not executed as yet.

5.- Boot into Windows and do the following:

- For Windows 7 - 8.1 (including their server variants) update KB3064209 must be uninstalled, in case it
is found in the system. This is a microcode update, which contains microcode version 0x2E for Haswell-
Ex.
- For Windows 10 meanwhile is distributed with microcode version 0x36. To remove it, file named
"mcupdate_GenuineIntel.dll" found in System32 folder must be renamed so that the system no longer
finds it

6.- Install vmware driver: - > 0x27 microcode for best performance

-> 0x39 driver for better stability

 - Download the VMWare tool: https://labs.vmware.com/flings/vmware-cpu-microcode-update-
driver
- Download the microcodes: version 0x27 & 0x39 microcodes for Haswell-Ex (0x306F2) in
VMWare driver / Linux compatible format:

https://1drv.ms/u/s!Ag6oE4SOsCmDhFnET3uw9wHeV4EA

- Copy the file 0x27.dat (or 0x39) to the same folder as the vmware utility and rename it to
microcode.dat.
- Download & copy these files to the same folder:
microcode_amd.bin
microcode_amd_fam15h.bin

- Execute install.bat as ADMIN, and reboot.

- Check with HWINFO (https://www.hwinfo.com/)

- if your turbos are working as expected, and post in the fórum your result! =)

MANY THANKS TO “THE STILT” & “DUFUS” FROM ANANDTECH.COM

Explanation
After few days of testing, here's what I've discovered:

The most essential thing is that the CPU is initialized WITHOUT a microcode. Allegedly it is possible to initialize the CPU
with an extremely old microcode version, but so far I haven't been able to find such version (hence allegedly). Microcode
version 0x1F (06/03/2014) is already too "new" to prevent this exploit from working. Since each and every motherboard
bios is supplied with a microcode present (for obvious reasons), initializing the CPU without a microcode mandates that
the microcode is completely removed from the bios binary. This naturally involves modifying the bios and updating it,
which in some cases can be little tricky.

After testing all of the different microcodes I could find, I've found out that there are rather large differences between them.
The most important thing is, that it appears that Intel has no direct or indirect means to completely prevent this exploit
from working. Technically they can reduce the "yield" (clocks) in certain workloads, but not prevent it completely as it is
too late when the CPU has already been initialized. Newer microcode builds generally contain workarounds for errata and
because of that it is generally recommended to use the newest build available. When using this exploit you'll need to
decide if you want to have the highest possible performance in all workloads, possibly at the expence of reliability or
alternatively slightly lower performance at the best known reliability (i.e with the most recent microcode update).

Haswell was the first "wide" core from Intel (256-bit FP). In order to preserve power, the Power Management Unit (PMU)
power gates the upper 128-bit of the FP when 256-bit instructions are not executed. In somewhere between August and
September of 2014 Intel changed the behavior of the Turbo on Haswell. Previously the Turbo behavior was identical
regardless if the upper 128-bit of the FP was executing or not (i.e same clocks for 128-bit and 256-bit workloads). In the
microcode released in September 2014 the Turbo behavior was changed significantly, from static to workload dependant.
In this microcode and all the newer ones the Turbo clocks are exactly the same for 128-bit workloads as before, but
significantly lower for 256-bit workloads. On my CPU the difference is 400MHz.

The newest microcode version for the Haswell-E/EP/EX/EN production stepping (CPUID 0x306F2) is version 0x39
(10/07/2016). This microcode can be used for this exploit, however it will result in lower yield (clocks) than the earlier
ones. This microcode is highly recommended if you are satisfied with a more modest boost, or require maximum reliability
(professional use). This microcode also has an additional advantage on systems, which lack both the "Power Limit" or
"CPU telemetry feature" (SVID) options in the bios. Version 0x39 microcode is one of the few versions, which doesn't
feature the bug I call as the "LFM bug". The best way to describe the "LFM bug" is that when you use this exploit, load a
newer microcode in flight and then try adjusting any of the CPU parameters (frequency, voltage, power limits, etc), the
CPU will lock to the LFM state (typically 800MHz).

I personally ended up using microcode version 0x27 (08/08/2014), and this is the version which offers the best
performance. This versions still features the static Turbo behavior (same for 128/256-bit workloads) and has some of the
most critical Haswell-Ex erratas (such as TSX) already fixed.

Additionally there appears to be some Turbo rules, which appear to be core configuration dependant and completely fixed.

These apply on my Haswell-E HCC, but they might be different on other variants:

- >= 10 cores == Maximum Turbo Ratio available

- >= 12 cores == Maximum Turbo Ratio - 100MHz
- >= 14 cores == Maximum Turbo Ratio - 200MHz
- >= 16 cores == Maximum Turbo Ratio - 400MHz
- >= 18 cores == Maximum Turbo Ratio - 500MHz

This means that when 0x27 microcode is used, I can run my 2699 at 3.6GHz (1-10 cores), 3.5GHz (with 12 cores), 3.4GHz
(with 14 cores), 3.2GHz (with 16 cores), 3.1GHz (with 18 cores), regardless of the workload.

Since the microcode can be updated in flight, controlling the microcode version in Windows might be slightly harder.
For Windows 7 - 8.1 (including their server variants) update KB3064209 must be uninstalled, in case it is found in the
system. This is a microcode update, which contains microcode version 0x2E for Haswell-Ex.
Windows 10 meanwhile is distributed with microcode version 0x36. To remove it, file named "mcupdate_GenuineIntel.dll"
found in System32 folder must be renamed so that the system no longer finds it. Note that I haven't tested this procedure
personally, since I'm still using Windows 7.
For Linux using a specific microcode version should be quite well documented else where.

The microcode in Windows can be updated with a driver released by VMWare: https://labs.vmware.com/flings/vmware-
cpu-microcode-update-driver

Here are version 0x27 & 0x39 microcodes for Haswell-Ex (0x306F2) in VMWare driver / Linux compatible
format: https://1drv.ms/u/s!Ag6oE4SOsCmDhFnET3uw9wHeV4EA
Rename the desired version to microcode.dat, and proceed as instructed by VMWare.

Personally I gained around 28% of performance with this exploit.

#39The Stilt, Jan 16, 2017
As said before, the most important thing is that you initialize the CPU without ANY
microcode present (i.e in it's factory state). This is achieved by removing the microcode
completely from the bios. When the CPU is initialized without any microcode, the errata
which makes this exploit possible is still present. As soon as any microcode which
features a workaround is loaded to the µROM, the errata will be repaired and this no
longer works. However if you make the desired changes before any microcode is
loaded, they will stick just fine even after loading any of the microcode versions. The
errata which makes this exploit possible will be repaired the same way it would be
normally, however the CPU cannot revert the changes you've made and therefore the
changes will stick.

If the bios lacks OC options, such as the "multicore enhancement" (MCE), then this
exploit will be harder to use but still not impossible. It can be done in UEFI, for example
with tools such as RU. Programming the MSRs is far from rocket science:

The MSRs which need to be programmed are the same regardless of the model used,
only the number of bytes you need to change depend on the core count.

The targeted registers are MSRs 0x1AD (Cores 0-7), 0x1AE (Cores 8-15), 0x1AF
(Cores 16-18 + Semaphore bit).

Example default configuration for 2698 v3 CPU (16C/32T, 3.6GHz maximum turbo):

MSR 0x1AD: 0x1D1E1F20 (EDX), 0x21222424 (EAX) -

2.9/3.0/3.1/3.2/3.3/3.4/3.6/3.6GHz (Core 7 - 0)
MSR 0x1AE: 0x1C1C1C1C (EDX), 0x1C1C1C1C (EAX), - 2.8GHz (Core 8-15)
MSR 0x1AF: 0x00000000 (EDX), 0x00000000 (EAX) - N/A

To:

MSR 0x1AD: 0x24242424 (EDX), 0x24242424 (EAX) - 3.6GHz (Core 7 - 0)

MSR 0x1AE: 0x24242424 (EDX), 0x24242424 (EAX), -3.6GHz (Core 8-15)
MSR 0x1AF: 0x00000000 (EDX), 0x00000000 (EAX) - N/A

Once the multiplier registers have been programmed, the changes must be initialized
by setting the semaphore bit (MSR 0x1AF, bit 63:63).
After that a microcode can be loaded and the CPU new frequency settings will stick.