Veeam v9 and vSphere 6 - no VSS extension upload using VIX

A few days ago I was at a customer that uses Veeam to backup his vSphere environment. Nothing special in the configuration. We run this setup for a few years now and it was bulletproof.

Recently we upgraded to Veeam v9 and vSphere 6 to be on the latest major versions both vendors offer. Since then we have a strange problem that was overseen for a while.

The customers security requirements deny access for any system from the internal network to the DMZ, especially for the backup server. To have fully consistent backups of VMs running in the DMZ we use Veeam VAAIP agents uploaded to the VMs via VIX because RPC (admin share access via CIFS) is not allowed. This worked perfectly even with Veeam v9 and vSphere 6. Perfectly until the customer upgraded the VMware Tools on his DMZ VMs. The weeks before the problem arose the VMs ran on vSphere 6 but with VMware Tools from 5.5. The moment he upgraded to VMware Tools 10.x the VIX upload didn't work anymore. The problem only hits a few VMs that can not leverage RPC as alternative upload method.

Read More Comment (2) Hits: 13148

VeeamONE: failed to start VeeamRSS service

In one of my current projects I was faced to do an infrastructure assessment to get a clear overview of the load on a VMware virtualized infrastructure. Of course, one of the best tools to achieve this is VeeamONE. I already wrote about VeeamONE and what it can do for you but this is not the topic of todays post.

Back to the project, I decided to install a VeeamONE server on top of the virtualized infrastructure. Installation went fine until quite before the setup ends and where it tries to start the services. The VeeamONE services consist of two sub services. The monitoring service and the reporting services. The first service started without any problems but the second service (VeeamRSS) won't start.

VeeamONE timeout2

You can skip the startup of this service during setup to complete the installation but sooner or later you will need the reporting engine to see what the monitoring service really collected.

Read More Comment (0) Hits: 6845

Veeam Off-host Proxy, DataCore VSS and MS Hyper-V - lessons learned

Veeam is a perfect backup tool when it comes to making backups efficient and fast. DataCore is one of my favorite storage solutions and Hyper-V.... well, I still prefer VMware vSphere way over Hyper-V. They might have catched up with VMware in most features and probably even superceeded them in special functions but when it comes to a very, very important area they still are years behind: ease of use. Not the hypervisor itself or the management tools like SCVMM or even failover cluster manager. A long-year windows admin probably can handle this kind of construct, although I think VMware has smarter solutions but this is my personal opinion.

The problem begins when it comes to efficient backup and 3rd party integration. Hyper-V currently completely relies on VSS to make consistent snapshots of VMs. In the guest OS this approach is okay but using VSS on the Hyper-V host as well putting VSS on top of VSS is not really a good thing. MS has acknowledged this problem and will probably find another way to replace VSS at least at the hypervisor level. VMware does this for a very long time with their snapshots and that's what makes offloading backup processes to other systems that easy. Even with a Windows based backupserver it's much easier to offload the VMware backups to an external system (that isn't aware of a filesystem called VMFS) than it is for any Microsoft based hypervisor environment.

But back to the main point of this article. We have several customers running Hyper-V on FC or iSCSI based DataCore storage and want to use Veeam as a backup product. This setup isn't quite hard to install but when it comes to make the backup process as efficient as it can be, you will stumble across some problems sooner or later.

Read More Comment (2) Hits: 7568

Veeam and vVols

With Backup&Replication 8 Update 2 Veeam added support for backing up vVol-based VMs. There is nothing special to do to backup those VMs but simply create backup jobs and include them as you would do with any VMFS datastore based VM. The only important thing to remember is that vVol based VMs can currently only be backed up in NBD or hot-add mode. The reason for this limitation is because vVols are a VMware-related special construct that needs a special presentation schema from the storage system as well as special drivers for the HBA to support it. You can see the requirements for using vVols with a 3PAR system in this article.

One of the mentioned requirements is to set the host property within the 3PAR system to use the 11-VMware persona. With this persona there will be a meta device presented, the so called Protocol Endpoint meta device. This is the entry point for the vSphere host to create vVol 1:1 mappings between VM objects and storage LUNs. This PE device will only be presented to hosts with the VMware persona. As your physical Veeam backup server is a Windows system, you have to choose the Windows persona for that server and thus the PE meta device can't be presented to the host. Furthermore the special FC drivers you need to detect the PE device are not available for Windows.

So to cut a long story short: currently backups of vVols in SAN mode is NOT SUPPORTED.

Read More Comment (0) Hits: 5674

Updates for MS SQL Server break Veeam functionality

The last few posts about patches introducing problems with Veeam were mainly focused on VMware patches or better said, VMware bugs. This time, Microsoft released some patches for their SQL server products that will hit Veeam in doing it's job correctly.

To be more precise, their latest updates for SQL Server 2012 SP2 (CU7), SQL Server 2014 RTM (CU10) and SQL Server 2014 SP1 (CU3) introduced a major change for the SELECT @@VERSION query. Seems to be a pretty small change but 3rd party vendors like Veeam rely for some of their functionality on the output of the statement above and with this change, the application aware image processing of SQL databases is not possible anymore. Currently the whole impact is unknown but it seems that at least the transaction log handling is affected.

So if you rely on this feature and don't want to wait for Veeam v8 Update 3 or v9, skip the updates mentioned above or at least move them to a time after the Veeam product updates are available.


Comment (0) Hits: 2675

First look at Veeam Endpoint Backup BETA

As a blogger writing about Veeam software I was invited to participate on the new BETA program for Veeam Endpoint Backup a few days before public BETA will be available. So let's have a look at this new tool from Veeam.

Endpoint Backup (VEB) is a free to use tool for backing up endpoints aka clients. VEB is not written to be used in virtual environments as Backup&Replication is but rather to be used on physical systems like laptops, desktops and even servers. The hardware is negliable as VEB runs as a Windows process and thus only requires one of the supported operating systems. Currently all Windows versions with at least Windows 7 kernel (including server OS) are supported.

Beside the main product binaries the installer installs a localdb version of MS SQL Server 2012. This is a special, ressource-minimized version of SQL Server that is used to store all required information (e.g. configuration, schedules, backup files etc.) for VEB. Additionally there is some driver installation required for the virtual disk driver subsystem to open the backup files later. One thing to mention here, there was no reboot required after installing the product. It can be used instantly. That's because VEB only uses VSS to create snapshots of volumes and files so no additional driver layer has to be installed.

After first installation I took a look at the additional ressources needed by VEB. The SQL Server takes ~280MB of RAM, the VEB tray program that is used to configure VEB needs ~40MB and the Veeam service takes another 25MB. So in sum VEB needs ~320-350MB of RAM on my client system (Windows 8.1) in idle mode. During backup VEB spawns another process that takes additional 40MB. So plan for an amount of 400MB for VEB. Not a problem on modern clients but needs to be considered.

Read More Comment (0) Hits: 11463

Veeam real world scenario

I've done several Veeam projects until now but I was never faced with the requirements the last project had.

The given situation was a backup environment where Veeam was used in combination with HP DataProtector. Not unusual as I configured those combinations myself during the last years to get the best both products can offer. But here, the customer asked to remove DP completely as the maintenance was running out and a renewal would be quite expensive. DP was used to backup physical systems and databases, most of them SQL systems but also some Oracle systems already virtualized under vSphere.

DP was also used to copy backup-to-disk data from itself as well as the Veeam repository data to tape to get some kind of long term backups. As the legal requirements forces the customer to keep backups for a long period of time (up to 5 years), tape was the only solution to keep up with the high amount of data.

I thought about the problem and came to the point where I was unable to completely remove DP from the scene but in my opinion this isn't neccessary at all. Veeam won't be able to backup all kind of systems the customer has but if we get as close as possible to 100% virtualization, it would reduce the importance of DP extremely. This way DP can be used for all applications and systems Veeam won't offer a perfect solution for and the number of those systems will probably quite low. So perhaps we can't remove DP completely but we can reduce it's footprint and thus it's costs to an absolute minimum.

Read More Comment (2) Hits: 4230

Tape backup slow after upgrade to Veeam v8

A few days ago I upgraded one of our customers Veeam Backup&Replication v7 installation to v8. Normal backups and restores ran at nearly same speed. Problems started when I used tape to copy the backup files to a long-term storage medium.

Since the upgrade backup jobs to tape ran at exactly 18,4MB/s and didn't ever exceed or drop down from this speed. It seemed like any kind of bandwidth throtteling is in place. Restores also drop in speed from formerly 60-70MB/s (it's an LTO5 drive directly attached to the backup server via 6Gb SAS) to ~30-35MB/s. Clearly the hardware should do a better job and in v7 it did the same backups at 70-80MB/s in average with peaks up to 120MB/s.

Surely the load on the source disk system is a factor but even with no load on the storage the backups and restores keep this low speed.

I searched the Internet and especially the Veeam forum and found a post from someone having exactly the same issue but with the new tape role enabled thus transferring data from the backup storage to the tape server over LAN and then over SAS to tape. Surely this setup could be limited by the LAN performance but the reported 25MB/s should be no problem for a GBit LAN.

Checking for any bandwidth throtteling setup didn't show any misconfiguration as there was no throtteling enabled at all.

Veeam support checked the problem and found that the low speed was also related to a problem that v8 is handling the source backup files in a wrong way. This behavior is related to handling reverse incrementals in a wrong way thus copying more data to tape then needed. There is a KB article published in the meantime and a hotfix available. As you already may guess if you see the description of this KB article this hotfix won't be the solution to the performance problem. And you're right. Even after applying this hotfix, tape jobs remain at the same speed as before.

In my case, this will render tape integration more or less useless as I have to copy several TB to tape. No fun with <20MB/s.

I opened a case with Veeam but have to produce more logs to help Veeam support finding the problem. I expect (and hope) a solution to be available within the next week.

I post this article becasue I don't think this performance problem is related to the used hardware (IBM server with IBM tapelibrary and IBM storage) but will also hit users with other hardware. So it seems to be more general problem or a misconfiguration that is introduced during upgrade. This is possible as especially the tape support is changed through the implementation of the tape proxy role.

I will keep you informed about any news on that topic. In the meantime I would recommend to test your setup BEFORE the upgrade, get some logs that proves your tape installation to be correct and working fast and test your setup after upgrade. If you encounter the same performance hits as I did consider a rollback if there hasn't been a solution published in the meantime.

Update 24.11.2014

As soon as I uploaded the required logs to Veeam support I got nearly instantly (okay, after 30min but that's really fast) an update to my call. Veeam support checked the logs and found out that the limitation to 18MB/s was a result of enabled network encryption. Strange as copy to tape should be only local in my environment as I have backup to disk storage and tape drive attached to the same host. BUT in my environment as a result of reusing a server that has formerly other services installed, there was a second NIC configured using an official IP address. I don't know why this was done but it forced the Veeam installation to think it has to transfer data from an official IP to the private IP the server really used to do communication. In that case, Veeam encrypts network traffic by default. This setting is currenlty hard-coded and can't be disabled within the Veeam GUI. Veeam support send me a registry key that forces encryption to be disabled globally:

HKLM\Software\Veeam\Veeam Backup and Replication\DisablePublicIPTrafficEncryption = 1 where DisablePublicIPTrafficEncryption is a new DWORD to be manually set.

After restart of the Veeam backup Server service the copy to tape runs at ~90MB/s what is the vaule I've seen before upgrade to v8.

Comment (5) Hits: 19626


joomla templatesfree joomla templatestemplate joomla
2018   globbers joomla template