Browsed by
Month: February 2017

Veeam One Reporter Server installation failed

Veeam One Reporter Server installation failed

During the recent deployment of Veeam One as part of Veeam Availability Suite I met a mysterious problem during the installation.
The problems seemed to be related to Veeam Reporter Server.
Environment characteristic:
– Veeam 9.5 with Update One
– Virtual Machine in Windows AD domain
– Domain user with log on as service and local Administrator privileges
– External clustered MS-SQL Server 2012
– Database created by the installator or using the script provided by Veeam – both ways checked with the same result

After starting the installation (running as Administrator just to make sure) it was goint to Reporter Server installation and during that something was going wrong and the whole
installation was rolling back.

In the logs of Reporter Server I found such information:

MSI (s) (28:64) [14:13:16:973]: Product: Veeam ONE Reporter Server — Installation operation failed.
MSI (s) (28:64) [14:13:16:973]: Windows Installer installed the product. Product Name: Veeam ONE Reporter Server. Product Version: Product Language: 1033.
Manufacturer: Veeam Software. Installation success or error status: 1603.

Going deeper I realised that the installator was able to connect to DB, but after that, when it checked that the DB exist or not and tried to reconnect the connection failed.

MSI (s) (28!40) [14:13:14:801]: Doing action: VmaVeeamOneSQLSettings_Check
Action start 14:13:14: VmaVeeamOneSQLSettings_Check.
MSI (s) (28:5C) [14:13:14:816]: Invoking remote custom action. DLL: C:\Windows\Installer\MSID04F.tmp, Entrypoint: _VmaVeeamOneSQLSettings_Check@4
***Veeam*** Windows authentication selected, impersonating will be performed.
***Veeam*** Impersonating. User: ‘ENERGA\veeamone’…
***Veeam*** Impersonation completed successfully.
***Veeam*** Connecting to SQL server. Instance:, User:
***Veeam*** SQL Server connection established.
***Veeam*** Sql Server version: 11.0.6544.0.
***Veeam*** Database ‘VeeamOneDB’ exists.
MSI (s) (28!60) [14:13:15:441]: PROPERTY CHANGE: Adding VM_ONE_DB_EXISTS property. Its value is ‘1’.
***Veeam*** Reconnecting to SQL server…
***Veeam*** Checking HADR…
***Veeam*** GET HADR configuration…
***Veeam*** HADR is enabled.
***Veeam*** Get replica role…
***Veeam*** An error occurred while connecting to SQL. Error code: 0x80040E14.
***Veeam*** Description: ‘ADODB::Connection->Execute failed. ErrorMessage: IDispatch error #3092.
Description: The user does not have permission to perform this action.
State: 42000, Native: 297, Source: Microsoft OLE DB Provider for SQL Server
Failed to connect to SQL Server ‘’.
MSI (s) (28!60) [14:13:15:722]: Product: Veeam ONE Reporter Server — Failed to connect to SQL Server ‘’.
MSI (s) (28!60) [14:13:15:722]: PROPERTY CHANGE: Adding VMONE_SQL_CHECKING_RESULT property. Its value is ‘0’.
Action ended 14:13:15: VmaVeeamOneSQLSettings_Check. Return value 1.
CustomAction VmaRP_SQLSettings_Check returned actual error code 1603 (note this may not be 100% accurate if translation happened inside sandbox)
Action ended 14:13:15: VmaRP_SQLSettings_Check. Return value 3.
CustomAction VmACheckCustomDialogsParams returned actual error code 1603 (note this may not be 100% accurate if translation happened inside sandbox)
Action ended 14:13:15: VmACheckCustomDialogsParams. Return value 3.

I wasn’t able to find anything on the Web which would be similar to my issue.
Then I tried to use a different database instanace – let’s say it was test one, with small difference – it was a standalone SQL Server beside the cluster.
And this time it worked correctly. This was really strange for my expecially in case I used the same VM, domain user, etc.

After that, guided by a Veeam Support I tried one more time installing each component separately. I mean installing first Veeam One server with WebUI option deselected.
Then using the CLI with fallowing commands I was able to successfully install Reporter and Business View.
cd /d D:\VeeamONEReporterWeb_x64.msi
cd /d D:\BV_x64.msi

During the installation of these additional components there is a wizard where is poissible to change internal database to external one.

VMware vExperts 2017 awards

VMware vExperts 2017 awards

VMware vExpert is a non technical award granted by a special committee every year. It is a special award because there is no exam or anything like that. VMware judges give it for individuals who have demonstrated significant contributions to the community and share their expertise with others. This means different kinds of activities like blogging, sharing the knowledge by other channels or social media as well as  public speakers, book authors, CloudCred task writing, script writers, VMUG leaders and VMTN community moderators.

In this year we are very pleased to announce that both of us (Paweł Piotrowski and Daniel Okrasa) were awarded with this mysterious title for the first time. The whole list of vExpert 2017 can be found here.

We would like to thanks to all of our readers and your feedback. We promise to develop our blog with more interesting articles about our experiences and observations mostly in relation to VMware products 🙂

VMware vSphere tags limit – is it known ?

VMware vSphere tags limit – is it known ?

Recently I received quite interesting question – what is the supported maximum quantity  for tags in vCenter 6.0U2 ?

Malignant author of the question is a good friend of mine and VMware administrator in one person. He ssked about tags limit because he want to use them to provide more information about each of its production VM’s – roughly speaking need to create about 20000 tags.

I thought ok., give me couple seconds to verify this,  and looked fast in vmware configuration maxims …. couple minuntes later it was clear that this is not a easy question 😉

Furthermore after some additional research (no clear statement in official documentation)  we decide to perform tests in lab environment !

We used simple powercli script to create 20000 tags in test vcenter appliance (6.0U2) , below our script:

$i -le 20000
New-Tag -Name $i -Description $i -Category test

Script worked like a charm without any issue – so far so good :), but when we tried to assign one tag to first vm we encounter web client error 1009  – very strange!

We decided to perform additional test and find out that limit is below 10000.  At this stage we decide to clear this issue with Vmware support and after some time received wery interesting feedback:

  1. NGC has upper bound of retrieve 10000 objects max.
  2. If the tags are less than 10000 then data service timeouts after 120 seconds(default dataservice timeout is 120 seconds).
  3. Decreasing the count to 9994 tags and increasing dataservice timeout, shows up all the tags(Assign) now.

As a temporary workaround for now.
1. Have total created tags less than 10000.
2. Increase data service timeout to 600 seconds(10 min).

VMware GSS stands that engineering working now to remove tag limit boundary in next releases vSphere 6.x.

vSphere 6.5 – Update Manager changes

vSphere 6.5 – Update Manager changes

Going through our list of articles about new features in vSphere 6.5 the last one is vSphere Update Manager for vCenter Server Appliance. Since vSphere 6.5 it’s fully embedded and integrated with vCenter Server Appliance with no Windows dependencies. It means that vCenter Server Appliance delivers now Update Manager as an optional service similar to Auto Deploy, etc.

Since vSphere 6.5 there is no longer possible to connect Update Manager instance that is installed on a Windows Server machine with vCenter Appliance.

That’s mean that you have two ways to use Update Manager component:
• You can install the Update Manager server component either on the same Windows server where the vCenter Server is installed or on a separate machine. To install Update Manager, you must have Windows administrator credentials for the computer on which you install Update Manager.
• You can deploy vSphere Update Manager in a secured network without Internet access. In such a case, you can use the vSphere Update Manager download service to download update metadata and update binaries.

In the a facelifted Web Client Update Manager Web Client appears as an Update Manager tab under the Configure tab in vSphere Web Client.

The whole management processes are rather the same so there isn’t anything special and worth to notice here since the product is pretty simple, easy on of course it’s doing the job.

VMware Postgres Installer failed with error code ‘3010’

VMware Postgres Installer failed with error code ‘3010’

Couple days ago I hit strange issue with vcenter 6.x appliance update process. Attempting to upgrade vCenter server 6.0 U1 to 6U2, encounter following error: Installation of component VMware Postgres Installer failed with error code ‘3010’. The installation rolled back after clicking at OK button. I was able to find following KB 2144389 with a similar issue but the workaround didn’t work. After rolling back to a snapshot before the update, we also tried to update vCenter to 6U 1b with the same result.

Additional test has been conducted :

  1. Update attempt:
    I Stop the Syslog collector service.
    II Disable the Syslog collector service.
    III Attempt the upgrade.

2.Update attempt:

I Stop all services service.
II Attempt the upgrade.

3.update attempt:
I Snapshot the vCenter VM.
II Stop and disable the Syslog Collector service.
III Restart the VM.
IV Open task manager and kill the nxlog.exe process.
V  Attempt the upgrade.

4. Update attempt:

Try to determine if this an issue with installing the Postgres service only or if this is an issue with another component holding a java process.
I Stop ALL Vmware services and set them to manual.
II Restart the vCenter and ensure no services are restarted.
III  Open the installation and navigate to the postgres installtion msi and attempt to just install this component.

Each update attempt failed with error:
PitCA_MessageBox: Displaying message: “Installation of component VMware Postgres Installer failed with error code ‘3010’

After this series of failures I digged deeper in update process logs and in msi.log:

MSI (s) (A8:34) [14:49:29:836]: Executing op: FileCopy(SourceName=pgevent.dll,SourceCabKey=fil0D75DEE8C36C2980055560AE16671228,DestName=pgevent.dll,Attributes=512,FileSize=34520,PerTick=65536,,VerifyMedia=1,,,,,CheckCRC=0,Version=,Language=0,InstallMode=126091264,,,,,,,)
MSI (s) (A8:34) [14:49:29:836]: File: C:\Program Files\VMware\vCenter Server\vPostgres\lib\pgevent.dll; Overwrite; Won’t patch; Existing file is a lower version
MSI (s) (A8:34) [14:49:29:836]: Source for file ‘fil0D75DEE8C36C2980055560AE16671228’ is compressed
MSI (s) (A8:34) [14:49:29:836]: Re-applying security from existing file.
MSI (s) (A8:34) [14:49:29:836]: Note: 1: 2205 2: 3: Error
MSI (s) (A8:34) [14:49:29:836]: Note: 1: 2228 2: 3: Error 4: SELECT `Message` FROM `Error` WHERE `Error` = 1603
MSI (s) (A8:34) [14:49:30:195]: Verifying accessibility of file: pgevent.dll
MSI (s) (A8:34) [14:49:30:195]: Note: 1: 2318 2:
MSI (s) (A8:34) [14:49:30:195]: Note: 1: 2318 2:
Info 1603.The file C:\Program Files\VMware\vCenter Server\vPostgres\lib\pgevent.dll is being held in use. Close that application and retry.

Finally find root cause – looks like some application is using this DLL, that is why we see this 3010 error.

How find which process is holding this ?

1) Open Process Explorer.
2) Select Find -> Find Handle or DLL
3) Type C:\Program Files\VMware\vCenter Server\vPostgres\lib\pgevent.dll in the search box.
4) Hit search
5) Wait…
6) You should see the Process ID that is holding the lock on that file.

(“Please make sure the customer has a snapshot of VM before trying these steps”)

1. Please take a backup of pgevent.dll
2. Delete the file pgevent.dll
3. Now start the update to 6.0 U2.