Tape backup slow after upgrade to Veeam v8

A few days ago I upgraded one of our customers Veeam Backup&Replication v7 installation to v8. Normal backups and restores ran at nearly same speed. Problems started when I used tape to copy the backup files to a long-term storage medium.

Since the upgrade backup jobs to tape ran at exactly 18,4MB/s and didn't ever exceed or drop down from this speed. It seemed like any kind of bandwidth throtteling is in place. Restores also drop in speed from formerly 60-70MB/s (it's an LTO5 drive directly attached to the backup server via 6Gb SAS) to ~30-35MB/s. Clearly the hardware should do a better job and in v7 it did the same backups at 70-80MB/s in average with peaks up to 120MB/s.

Surely the load on the source disk system is a factor but even with no load on the storage the backups and restores keep this low speed.

I searched the Internet and especially the Veeam forum and found a post from someone having exactly the same issue but with the new tape role enabled thus transferring data from the backup storage to the tape server over LAN and then over SAS to tape. Surely this setup could be limited by the LAN performance but the reported 25MB/s should be no problem for a GBit LAN.

Checking for any bandwidth throtteling setup didn't show any misconfiguration as there was no throtteling enabled at all.

Veeam support checked the problem and found that the low speed was also related to a problem that v8 is handling the source backup files in a wrong way. This behavior is related to handling reverse incrementals in a wrong way thus copying more data to tape then needed. There is a KB article published in the meantime and a hotfix available. As you already may guess if you see the description of this KB article this hotfix won't be the solution to the performance problem. And you're right. Even after applying this hotfix, tape jobs remain at the same speed as before.

In my case, this will render tape integration more or less useless as I have to copy several TB to tape. No fun with <20MB/s.

I opened a case with Veeam but have to produce more logs to help Veeam support finding the problem. I expect (and hope) a solution to be available within the next week.

I post this article becasue I don't think this performance problem is related to the used hardware (IBM server with IBM tapelibrary and IBM storage) but will also hit users with other hardware. So it seems to be more general problem or a misconfiguration that is introduced during upgrade. This is possible as especially the tape support is changed through the implementation of the tape proxy role.

I will keep you informed about any news on that topic. In the meantime I would recommend to test your setup BEFORE the upgrade, get some logs that proves your tape installation to be correct and working fast and test your setup after upgrade. If you encounter the same performance hits as I did consider a rollback if there hasn't been a solution published in the meantime.

Update 24.11.2014

As soon as I uploaded the required logs to Veeam support I got nearly instantly (okay, after 30min but that's really fast) an update to my call. Veeam support checked the logs and found out that the limitation to 18MB/s was a result of enabled network encryption. Strange as copy to tape should be only local in my environment as I have backup to disk storage and tape drive attached to the same host. BUT in my environment as a result of reusing a server that has formerly other services installed, there was a second NIC configured using an official IP address. I don't know why this was done but it forced the Veeam installation to think it has to transfer data from an official IP to the private IP the server really used to do communication. In that case, Veeam encrypts network traffic by default. This setting is currenlty hard-coded and can't be disabled within the Veeam GUI. Veeam support send me a registry key that forces encryption to be disabled globally:

HKLM\Software\Veeam\Veeam Backup and Replication\DisablePublicIPTrafficEncryption = 1 where DisablePublicIPTrafficEncryption is a new DWORD to be manually set.

After restart of the Veeam backup Server service the copy to tape runs at ~90MB/s what is the vaule I've seen before upgrade to v8.

Leave your comments

Post comment as a guest

0
Your comments are subjected to administrator's moderation.

People in this conversation

  • Guest - Dax

    I have exactly the same issue. I have logged a support request with Veeam and they have acknowledged that this is a bug in the software. No ETA of a fix yet, which is frustrating. The only similarities in the hardware config to yours is that its LTO5, but is direct connect via FC and is HP hardware.

  • Guest - Sinan

    Thanks. Fix my problem.. Now appr.130MB to tape..

  • Guest - andy70

    Great find - I can confirm this reg fix do the trick. I went from around 25 mb/s to +130 mb/s just by applying this DWORD<br /> :-)

  • Guest - Sven

    Thanks a million for sharing this!

  • Guest - Peter

    First, Thanks a lot for your great Post.
    I was happy to find this article, because i have the same issues with my LTO-6 SAS Drive. Some time ago we upgraded our Backup Server, but Backup Storage and Tape Drive are still the same. Before i had ~120MB/s - 130MB/s transfer rates from our synology NAS (connected via iscsi to the backup server) to the Tape Drive. Since we moved everything to the new Backup Server i only got rates about 90MB/s

    Now i tried several things, driver, firmware of tape drive etc. nothing changed. now i entered the registry key, but there is no improvement either. its still the same speed. i know the synolgy and the tape drive can perform much faster (due to CrystalDisk Benchmark and Copying Files). Veeam also shows the Target is the bottleneck, but this could not be right.

    Hence i am going to open a call at veeam, maybe they are going to find something.

Powered by Komento
joomla templatesfree joomla templatestemplate joomla
2017  v-strange.de   globbers joomla template