Tape backup slow after upgrade to Veeam v8

User Rating: 0 / 5

Star InactiveStar InactiveStar InactiveStar InactiveStar Inactive
Published: Friday, 21 November 2014 20:09

A few days ago I upgraded one of our customers Veeam Backup&Replication v7 installation to v8. Normal backups and restores ran at nearly same speed. Problems started when I used tape to copy the backup files to a long-term storage medium.

Since the upgrade backup jobs to tape ran at exactly 18,4MB/s and didn't ever exceed or drop down from this speed. It seemed like any kind of bandwidth throtteling is in place. Restores also drop in speed from formerly 60-70MB/s (it's an LTO5 drive directly attached to the backup server via 6Gb SAS) to ~30-35MB/s. Clearly the hardware should do a better job and in v7 it did the same backups at 70-80MB/s in average with peaks up to 120MB/s.

Surely the load on the source disk system is a factor but even with no load on the storage the backups and restores keep this low speed.

I searched the Internet and especially the Veeam forum and found a post from someone having exactly the same issue but with the new tape role enabled thus transferring data from the backup storage to the tape server over LAN and then over SAS to tape. Surely this setup could be limited by the LAN performance but the reported 25MB/s should be no problem for a GBit LAN.

Checking for any bandwidth throtteling setup didn't show any misconfiguration as there was no throtteling enabled at all.

Veeam support checked the problem and found that the low speed was also related to a problem that v8 is handling the source backup files in a wrong way. This behavior is related to handling reverse incrementals in a wrong way thus copying more data to tape then needed. There is a KB article published in the meantime and a hotfix available. As you already may guess if you see the description of this KB article this hotfix won't be the solution to the performance problem. And you're right. Even after applying this hotfix, tape jobs remain at the same speed as before.

In my case, this will render tape integration more or less useless as I have to copy several TB to tape. No fun with <20MB/s.

I opened a case with Veeam but have to produce more logs to help Veeam support finding the problem. I expect (and hope) a solution to be available within the next week.

I post this article becasue I don't think this performance problem is related to the used hardware (IBM server with IBM tapelibrary and IBM storage) but will also hit users with other hardware. So it seems to be more general problem or a misconfiguration that is introduced during upgrade. This is possible as especially the tape support is changed through the implementation of the tape proxy role.

I will keep you informed about any news on that topic. In the meantime I would recommend to test your setup BEFORE the upgrade, get some logs that proves your tape installation to be correct and working fast and test your setup after upgrade. If you encounter the same performance hits as I did consider a rollback if there hasn't been a solution published in the meantime.

Update 24.11.2014

As soon as I uploaded the required logs to Veeam support I got nearly instantly (okay, after 30min but that's really fast) an update to my call. Veeam support checked the logs and found out that the limitation to 18MB/s was a result of enabled network encryption. Strange as copy to tape should be only local in my environment as I have backup to disk storage and tape drive attached to the same host. BUT in my environment as a result of reusing a server that has formerly other services installed, there was a second NIC configured using an official IP address. I don't know why this was done but it forced the Veeam installation to think it has to transfer data from an official IP to the private IP the server really used to do communication. In that case, Veeam encrypts network traffic by default. This setting is currenlty hard-coded and can't be disabled within the Veeam GUI. Veeam support send me a registry key that forces encryption to be disabled globally:

HKLM\Software\Veeam\Veeam Backup and Replication\DisablePublicIPTrafficEncryption = 1 where DisablePublicIPTrafficEncryption is a new DWORD to be manually set.

After restart of the Veeam backup Server service the copy to tape runs at ~90MB/s what is the vaule I've seen before upgrade to v8.