Fusion-IO cards in DataCore servers

Recently I got the chance to gather some experience with Fusion-IO cards. One of our customers decided to add AutoTiering support to their DataCore environment and as he has HP hardware in place, he decided to buy some I/O Accelerator cards from HP. These cards are nothing else than Fusion-IO cards labeld with a HP sign.

So he bought four 1.2TB ioDriveDuo Gen1, two for each server. Mirroring one card to the other via DataCOre's MirrorDisk feature adds some pretty level of hardware redundancy.

So we installed the two cards in each server and booted them up. The cards were detected as "mass storage controller" with a yellow warning sign as we haven't installed the appropriate drivers from Fusion-IO. This is where the real fun starts. With Fusion-IO cards it is essential to have the proper firmware/driver combination in place. If you don't you won't get them online. Searching on the HP support site you will get two downloads, one is called VSL and one ioSphere (the names of both packages change quite often so in older versions VSL was called HP_IO_Accelerator_SNMPDefault_xxxx, ioSphere was called IO_Manager, so it's a bit tricky to find your way). VSL basically is the driver you load within the OS whereas ioSphere is a WebGUI that gives you the ability to configure the ioDrives. So we grabbed the latest version, installed it, rebooted the server and the cards were recognized as HP Accelerators in Windows device manager. The problem is, the drives were not seen by the logical disk manager. So we started the WebGUI and saw that the drives were in "minimal mode" which basically means you can't do anything with them until you match the firmware of the drives to the driver you use. We found that the firmware on the drives was really old so we flashed them to a later version. Not to the latest, this is not supported. Fusion-IO gives you an upgrade guide from which version to which version you can go. So in our case we had to flash twice until we got the latest firmware on our drives. By the way, upgrading from a old v2 to one of the first v3 firmware took nearly 3h for two ioDrive duo.....

Next funny thing is to find the appropriate firmware. In older releases, the firmware was bundled with the ioSphere installer. Then Fusion-IO decided to move it from the ioSphere bundle to the VSL bundle. The problem now is, the ioSphere doesn't find the firmware and doesn't give you an option to define the firmware location. You can try putting the firmware in the ioSphere installation directory under \Firmware but that didn't work all the time. Easiest way was to use the CLI tools to update the firmware as you can provide the full path to the firmware file within the command line.

After a very long time we finally matched the firmware to the latest VSL and we could configure them in ioSphere. A low level format is mandatory after firmware upgrades so we decided to check DataCore's TBs which sector size to chose. DataCore recommends 512bytes so we used it. Formatting was quite fast, 1.2TB in less than 2minutes...

After the low level format we were able to "attach" the ioDrive to Windows which finally makes them visible to the Windows LVM.

Initializing the drives within DataCore was quite fast, ~500MB/s.

One really important thing to remember when using Fusion-IO cards: Fusion-IO needs to grap memory from the OS depending on the sector size you chose during low level format. So if you use 512bytes as recommended by DataCore you have to reserve ~20MB of RAM for each GB of storage on your ioDrive. Now take your calculator and type in: 20MB of RAM + 1200GB formatted capacity = 24GB of RAM. The problem is, we have two of these drives so just to get the two IO Accelerators to run you have to reserve ~48GB of RAM. This is a HUGE amount of RAM you simply can't use as DataCore cache just to be sure your Fusion-IO cards won't deplete all your ressources.... I'm not sure if this is related to all flash drives from all vendors (sTec for example doesn't need any additional RAM) but this is really a problem. Okay, the 48GB is worst case scenario but I have to be sure my DataCore node won't crash if this scenario happens so I simply have to reserve this amount of memory.

Another thing we learned is that some of the ioDrives (and especially the big ones) will draw a whole lot of power from your PCIe subsystem. Normally this is rated at 25 watts per slot but these cards need up to 55 watts. If the card doesn't get the power it will still work but with less performance. You won't spent thousands of dollars for those drives to accept a lower performance just becasue these drives don't get enough power. So remember to buy these special power cords that draw additional power from the power supply instead of the PCIe slots. This way your system will be much more stable.
Another option is to allow the drive to override the 25 watt limit on the PCIe bus but your hardware manufacturer has to support that. In the worst case you will grill your mainboard.

Here is the bottom line for all Fusion-IO cards in DataCore servers

  • Fusion-IO is DataCore ready certified so you can be pretty sure these cards are fast as hell and work perfectly with SANsymphony-V
  • It's a bit tricky to get your cards and drivers up-to-date especially if you have old cards like we had
  • Don't rely on the names for the drivers/firmware/tools as they change randomly
  • Size your RAM according to the requirements by the Fusion-IO cards and your SSY-V cache !!!!!!
  • Do not use the ioDrive Duos without additional power
  • Sometimes the WebGUI has problems (e.g. low-level-format), then use the CLI tools

Leave your comments

Post comment as a guest

0
Your comments are subjected to administrator's moderation.

People in this conversation

  • Hi, <br />Can you share some informations about the performance gains due to FusionIO implementation? <br />Can you somehow compare Datacore FusionIO with Datacore non-FusionIO SSD implementations?

  • Guest - Oliver

    Hi, <br />Can you share some informations about the performance gains due to FusionIO implementation? <br />Can you somehow compare Datacore FusionIO with Datacore non-FusionIO SSD implementations?
    <br />Hi Pavol,<br /><br />from the customer we got the feedback that it's Oracle installation (virtualized on vSphere 5) shows performance gains between 2-10x. File copies raised from 200GB/h to 1TB/h. Especially the file transfer speed will profit from the additional "standard" block storage as we added but you can see in the Allocation tab that most of the database storage is located on the SSDs and thus the performance gain will be nearly 100% from the flash storage.<br /><br />We also have a customer that has standard SSDs in a HP D2700 enclosure attached to P812 controllers. The gain here isn't that much as you can expect a standard SSD on a SAS controller to deliver not more than 2000-4000 IOPs regardless what the manufacturer tells you. In comparision to that, PCIe cards really offer 50.000-70.000 IOPs and this makes them really worth the money.<br />So my recommendation is to use PCIe flash devices over SAS SSDs whenever possible. The do cost a bit more but as you look at the IOPS/GB they are quite "cheap". Of course, you need to have enough empty PCIe slots....<br />Regards,<br />Oliver

  • Guest - Mark

    Hi there,<br /><br />thanks for the article.<br /><br />I ran into a strange issue with two 365 GB HP Accelerator cards (iodrive 2) in a DataCore environment.<br /><br />After updating the firmware and the driver to the latest VSL 3 version, the HP Accelerator Management Server service crashes right after the start. So access to the GUI is not possible anymore.<br /><br />It seems to be a known issue at HP:<br /><br />http://h20566.www2.hp.com/portal/site/hpsc/template.PAGE/public/psi/troubleshootDisplay/?javax.portlet.begCacheTok=com.vignette.cachetoken&javax.portlet.endCacheTok=com.vignette.cachetoken&javax.portlet.prp_efb5c0793523e51970c8fa22b053ce01=wsrp-navigationalState%3DdocId%253Demr_na-c03508648-3%257CdocLocale%253Den_US&javax.portlet.tpst=efb5c0793523e51970c8fa22b053ce01&sp4ts.oid=3900933&ac.admitted=1407479946434.876444892.199480143<br /><br />But a re-install of Windows Server 2008 R2 is not an option.<br /><br />Did you have the same issue at your customer? How did you solve it?<br /><br />Regards,<br /> Mark

  • Guest - Oliver

    Hi there,<br /><br />thanks for the article.<br /><br />I ran into a strange issue with two 365 GB HP Accelerator cards (iodrive 2) in a DataCore environment.<br /><br />After updating the firmware and the driver to the latest VSL 3 version, the HP Accelerator Management Server service crashes right after the start. So access to the GUI is not possible anymore.<br /><br />It seems to be a known issue at HP:<br /><br />http://h20566.www2.hp.com/portal/site/hpsc/template.PAGE/public/psi/troubleshootDisplay/?javax.portlet.begCacheTok=com.vignette.cachetoken&javax.portlet.endCacheTok=com.vignette.cachetoken&javax.portlet.prp_efb5c0793523e51970c8fa22b053ce01=wsrp-navigationalState%3DdocId%253Demr_na-c03508648-3%257CdocLocale%253Den_US&javax.portlet.tpst=efb5c0793523e51970c8fa22b053ce01&sp4ts.oid=3900933&ac.admitted=1407479946434.876444892.199480143<br /><br />But a re-install of Windows Server 2008 R2 is not an option.<br /><br />Did you have the same issue at your customer? How did you solve it?<br /><br />Regards,<br /> Mark
    <br />Hi Marc,<br /><br />we haven't touched the environment since installation time except from DataCore updates so I can't tell you if this will happen at our customer too. <br />Oliver

Powered by Komento
joomla templatesfree joomla templatestemplate joomla
2017  v-strange.de   globbers joomla template