Recently we had two customers seeing undefinable performance problems on recent Gen8 and Gen9 systems running either VDI under VMware vSphere or simple file copy jobs under a Windows Server 2012 R2 system.
The VDI problem was quite curious as the full processing power of the Gen8 systems was never used but storage latencies raise up to 500ms (the storage system itself reported <1ms response time since it was an all-flash array). The customer did everything to solve that problem but in the end the power management from the HP ProLiant systems was the problem. The reason behind this is the default setting for all new ProLiant systems to use a "balanced power mode" that throttles down server components when they are not fully used. The BIOS has to define when the throtteling has to be done but the BIOS often doesn't know or is unable to decide on a stable base if the performance can be reduced to save power or not. This can only be done by the OS, no matter if it is a hypervisor like ESXi or Windows or Linux. Therefore, the power management can be set to either balanced, static low, static high or OS control performance mode. As already said, balanced is the default value.
Static low will save much power but will render your super fast, super expensive server into a "laptop-performing system". Static high will fully ignore any power savings and will kepp the system always at peak performance. OS control will give the OS the control over power management.
VMware did an excellent benchmark over the various power settings and the results are clear: BIOS handled power management (or balanced mode) is the worst you can do from a performance perspective and will result in unpredictable performance. Best is to use either full power mode or OS control, especially with latest versions of vSphere and Windows. See the full report here. Additional information for vSphere 5.5 and above can be found here and here.
To cut a long story short, after changing the power mode to static high the VDI environment performed massively better and all performance problems were gone.
A second and much more impressive experience comes from a brand new DL380Gen9 loaded with Server 2012 R2 fully patched. We used it as a Veeam server and installed a 8GBit FC adapter and a P441 SAS controller. Attached to the FC adapter there was a P2000 with 36x4TB MDL SAS disks in RAID6 mode with 12 disk per RAID set. The P441 controlled two D3700 shelves with 50x 600GB SAS 10k disks. They were configured into RAID5 sets with 12 disks. The server was configured with default "balanced" power management.
When starting the OS, the task manager showed ~1,7GHz CPU speed and never exceeding this value even when Veeam jobs ran. The CPU itself is a 2,4GHz Xeon v3 CPU that turboboosts up to 2,6GHz. Beside the CPU, the whole system wasn't exceptional fast. File copying from large backup files (3-6TB each file) was done with ~300MB/s to and from the both storage systems. That's not bad but could be faster.
During one of these large file transfers I received the information mentioned above from a colleague and decided to give it a try. Power management can be switched online by using iLO functionality. So I simply switched from balanced mode to static high mode and instantly the performance of the file copy job jumped up to 380MB/s constantly. I let it ran for 1 or 2 minutes and switched back to "balanced mode" and again, instantly the copy speed falled back to 300MB/s. Switching back to static high: 380MB/s, switching back to balanced: 300MB/s. So even for simple file copy jobs the power mode makes a significant difference.
BTW, switching to OS control can't be done without rebooting the server OS so I decided to stop tests here and let the copy jobs run further (even with 380MB/s the copy job needed additional 6h to complete).
Probably the way power management works will be more or less the same on all server models and vendors so this is not a HP only "bug". So as a recommendation I can only tell you to either disable pweor management, set it to static high or, best way, set it to OS control and let the OS do the job for you.