Getting the VMware VSA running in a nested ESXi environment

In the previous VSA article we took at a look at the storage architecture of the appliance, as well as some of the caveats and considerations when deploying it. In this article, we’ll take a look at how to get it up and running in a nested ESXi environment as well as some of the functions the VSA provides.

First, in order to create a nested ESXi 5.0 environment, have a look at this great article.

When creating your environment, my recommendation is to create 4 individual vDS port groups or individual standard 4 vSwitches for the environment. You will assign each to a vNIC of the vESXi host to simulate connecting each pNIC to a physical switch in a real deployment.

Be sure to configure the vSwitches (or vDS port groups) with promiscuous mode enabled and create 2 vESXi VMs with 4 NICs minimum and a SINGLE VMFS volume (this is important or else the VSA will not install). I recommend a thin provisioned volume of about 200GB for testing.

You should end up with something like this:

Same applies if you are using standard vSwitches in your environment.

Now you need a Windows based vCenter 5.0 instance to manage this environment. Install the VSA manager software onto that vCenter which will then expose the VSA manager plug-in/tab on the vCenter client once you click on a vSphere data center:

In normal installations, you would then click on the VSA manager tab and follow the instructions to install. The problem is that since we are installing in a nested vESXi environment, “EVC” does not work with nested vESXi and is a requirement the installer checks for, thus you will not be able to proceed:

Thus far I have not been able to find a workaround for this for the GUI based install. However, after lots of lab time I found there is a way around this problem: in order to install the VSA in nested ESXi and bypass the EVC requirement, we need to tweak a configuration file and then do the installation via command line. Download the full zipfile which includes the command line installer if you haven’t already and unzip that onto your system.

Here is the minimum syntax to get it going:

install.exe -u root -p <password_to_ESX_hosts> -si <start_address_for_VSA_front_end_IPs> -nh

Recall that the VSA has a front-end network and a back-end network. The “-si” switch tells it what public IPs to use for the front-end. You can specify a “-bs” start range for the back-end IPs, but it will default to 192.168.0.1 as the start range if you do not specify anything. You can also specify netmasks and VLANs. See the manual for details.

The “-nh” tells it not to join the hosts into a high availability cluster and this will be important to help bypass the EVC check. If we execute this command this will be the result:

As you can see the automated command line installer runs an audit stage and it fails for the same EVC reason!

Well, after much lab time, I figured a way around this problem. We need to change a parameter in C:\Program Files\VMware\Infrastructure\tomcat\webapps\VSAManager\WEB-INF\classes\dev.properties. Search for this line:

evc.config=true

and change it to

evc.config=false

This will effectively bypass the audit check for EVC. Cool huh?

Now re-run the install.exe command, and it should complete:

And you end up with this in your nested ESXi environment:

The result is 2x 100GB data stores, which correlates with each VSA having 200GB of RAW storage, for a total of 400GB RAW or 200GB usable after RAID10 internal to the VSA.

 

Here is a peak at the networking the VSA installer sets up:

There are front-end and back-end port groups that live on separate vSwitches and pNICs. You are now free to customize the networking however you see fit, but it HAS to have a default configuration starting out or else the install WILL fail.

Now that the VSA is installed, you can continue to manage it through the VSA plug-in in vCenter. We only needed to do the hack and command line to get it up and running. Again, its important to note this would not be required in a real installation, it was required due to the limitations of nested ESXi.

End result:

In the next article, we’ll take a look at some administrative tasks, and testing out some of the failure scenarios and how the VSA handles it from a downtime/uptime/reliability perspective.



Categories: storage, storage virtualization, vmware

7 replies

  1. Hi There,

    Tremendous post, thank you!

    I am trying to set up the VSA in a nested ESXi VM, only difference is that its inside VMware Fusion 4.0.1. I have followed the steps you have mentioned, but run into a problem with the following error : Scripted installation failed com.vmware.vim25.DuplicateName

    I would truly appreciate some assistance. Thanks in advance!

    Microsoft Windows [Version 6.1.7600]
    Copyright (c) 2009 Microsoft Corporation. All rights reserved.

    C:\Users\Administrator>d:

    D:\>install.exe -u root -p password -si 192.168.20.161 -ei 192.168.20.151 192.16
    8.20.153 -nh
    Current user: Administrator
    Current IP: 192.168.20.200
    Virtual Center is already installed
    The VMware VirtualCenter Management Webservices service is stopping.
    The VMware VirtualCenter Management Webservices service was stopped successfully
    .

    Installing VSA Manager, please wait …
    The requested service has already been started.

    More help is available by typing NET HELPMSG 2182.

    2011-09-22 16:27:04,910
    2011-09-22 16:27:04,956 Start to scan ESX hosts, please wait …
    2011-09-22 16:27:05,003 localhost netmask: /24
    2011-09-22 16:27:05,010 localhost broadcast: /192.168.20.255
    2011-09-22 16:27:09,693 Total IP in subnet: 255
    2011-09-22 16:27:09,694 start IP: 192.168.20.0
    2011-09-22 16:27:09,695 end IP: 192.168.20.255
    2011-09-22 16:27:25,439 Create Datacenter : vsaDC
    2011-09-22 16:27:25,464 Add host to SVA Datacenter, please wait …
    2011-09-22 16:27:26,702 Scripted installation failed.
    com.vmware.vim25.DuplicateName
    at com.vmware.vim25.DuplicateName$Factory.parse(DuplicateName.java:571)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
    java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
    sorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.vmware.vim25.ExtensionMapper.getTypeObject(ExtensionMapper.java:2
    8)
    at com.vmware.vim25.MethodFault$Factory.parse(MethodFault.java:738)
    at com.vmware.vim25.LocalizedMethodFault$Factory.parse(LocalizedMethodFa
    ult.java:652)
    at com.vmware.vim25.TaskInfo$Factory.parse(TaskInfo.java:2367)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
    java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
    sorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.vmware.vim25.ExtensionMapper.getTypeObject(ExtensionMapper.java:2
    8)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
    java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
    sorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.vmware.vim25.ConverterUtil.getAnyTypeObject(ConverterUtil.java:15
    89)
    at com.vmware.vim25.PropertyChange$Factory.parse(PropertyChange.java:762
    )
    at com.vmware.vim25.ObjectUpdate$Factory.parse(ObjectUpdate.java:891)
    at com.vmware.vim25.PropertyFilterUpdate$Factory.parse(PropertyFilterUpd
    ate.java:830)
    at com.vmware.vim25.UpdateSet$Factory.parse(UpdateSet.java:818)
    at com.vmware.vim25.WaitForUpdatesResponse$Factory.parse(WaitForUpdatesR
    esponse.java:419)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
    java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
    sorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.vmware.vim25.VimServiceStub.fromOM(VimServiceStub.java)
    at com.vmware.vim25.VimServiceStub.waitForUpdates(VimServiceStub.java:45
    606)
    at com.vmware.vim25.util.PropertyCollectorHelper.waitForUpdates(Property
    CollectorHelper.java:338)
    at com.vmware.vim25.util.PropertyCollectorHelper.access$000(PropertyColl
    ectorHelper.java:34)
    at com.vmware.vim25.util.PropertyCollectorHelper$1.run(PropertyCollector
    Helper.java:51)
    at java.lang.Thread.run(Thread.java:662)
    2011-09-22 16:27:26,709 Start to clean up, please wait …
    2011-09-22 16:27:26,710 Remove VSA cluster …
    2011-09-22 16:27:27,514 Remove VSA datacenter …
    2011-09-22 16:27:27,555 Clean up done.
    System error 109 has occurred.

    The pipe has been ended.

    The VMware VirtualCenter Management Webservices service is starting.
    The VMware VirtualCenter Management Webservices service was started successfully
    .

    vSphere Client is already installed

    AFTER THIS STEP, I MODIFY THE C:\Program Files\VMware\Infrastructure\tomcat\webapps\VSAManager\WEB-INF\classes\dev.properties file to set evc.config=false

    D:\>install.exe -u root -p password -si 192.168.20.161 -ei 192.168.20.151 192.16
    8.20.153 -nh
    Current user: Administrator
    Current IP: 192.168.20.200
    Virtual Center is already installed
    VSA Manager is already installed
    2011-09-22 16:29:41,207
    2011-09-22 16:29:41,212 Start to scan ESX hosts, please wait …
    2011-09-22 16:29:41,213 localhost netmask: /24
    2011-09-22 16:29:41,213 localhost broadcast: /192.168.20.255
    2011-09-22 16:29:41,215 Total IP in subnet: 255
    2011-09-22 16:29:41,215 start IP: 192.168.20.0
    2011-09-22 16:29:41,216 end IP: 192.168.20.255
    2011-09-22 16:29:43,927 Create Datacenter : vsaDC
    2011-09-22 16:29:43,953 Add host to SVA Datacenter, please wait …
    2011-09-22 16:29:44,232 Scripted installation failed.
    com.vmware.vim25.DuplicateName
    at com.vmware.vim25.DuplicateName$Factory.parse(DuplicateName.java:571)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
    java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
    sorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.vmware.vim25.ExtensionMapper.getTypeObject(ExtensionMapper.java:2
    8)
    at com.vmware.vim25.MethodFault$Factory.parse(MethodFault.java:738)
    at com.vmware.vim25.LocalizedMethodFault$Factory.parse(LocalizedMethodFa
    ult.java:652)
    at com.vmware.vim25.TaskInfo$Factory.parse(TaskInfo.java:2367)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
    java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
    sorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.vmware.vim25.ExtensionMapper.getTypeObject(ExtensionMapper.java:2
    8)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
    java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
    sorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.vmware.vim25.ConverterUtil.getAnyTypeObject(ConverterUtil.java:15
    89)
    at com.vmware.vim25.PropertyChange$Factory.parse(PropertyChange.java:762
    )
    at com.vmware.vim25.ObjectUpdate$Factory.parse(ObjectUpdate.java:891)
    at com.vmware.vim25.PropertyFilterUpdate$Factory.parse(PropertyFilterUpd
    ate.java:830)
    at com.vmware.vim25.UpdateSet$Factory.parse(UpdateSet.java:818)
    at com.vmware.vim25.WaitForUpdatesResponse$Factory.parse(WaitForUpdatesR
    esponse.java:419)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
    java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
    sorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.vmware.vim25.VimServiceStub.fromOM(VimServiceStub.java)
    at com.vmware.vim25.VimServiceStub.waitForUpdates(VimServiceStub.java:45
    606)
    at com.vmware.vim25.util.PropertyCollectorHelper.waitForUpdates(Property
    CollectorHelper.java:338)
    at com.vmware.vim25.util.PropertyCollectorHelper.access$000(PropertyColl
    ectorHelper.java:34)
    at com.vmware.vim25.util.PropertyCollectorHelper$1.run(PropertyCollector
    Helper.java:51)
    at java.lang.Thread.run(Thread.java:662)
    2011-09-22 16:29:44,241 Start to clean up, please wait …
    2011-09-22 16:29:44,241 Remove VSA cluster …
    2011-09-22 16:29:44,478 Remove VSA datacenter …
    2011-09-22 16:29:44,500 Clean up done.
    The VMware VirtualCenter Management Webservices service is stopping.
    The VMware VirtualCenter Management Webservices service was stopped successfully
    .

    The VMware VirtualCenter Management Webservices service is starting.
    The VMware VirtualCenter Management Webservices service was started successfully
    .

    vSphere Client is already installed

  2. Hi there, follow up. I have removed the hosts, and the datacenter, and allowed the VSA installer to create the DC. It looks ok, however, now I have the error : Audit failed – Not a greenfield host, invalid number of vSwitches.

    I have installed 4 standard vSwitches, each with one uplink, with promiscuous mode enabled, first for management, and the rest named Trunk 1,2,3. Any ideas would be welcome!

  3. I continue to get stuck at a certain place during the install and can’t get passed it.

    “Cannot create VSA cluster: java.lang.Exception: Failed to ping VSA service on VM 172.16.100.32”

    My vESXi hosts both have 4 NICs, each one is connected to a separate distributed port group, and all have promiscuous mode set to Accept. I’m only using two physical NICs on the dvSwitch.

    During the install i’m able to ping the VSA Cluster Service IP address and both vmkernel interfaces that are setup for vMotion.

    Here is an excerpt from the log:

    2011-10-02 23:00:21,305 193 [Utils] [Thread-178] INFO – Retrieve dev.property: network.config=true
    2011-10-02 23:01:03,285 88 [SVAManager] [http-127.0.0.1-8080-8] INFO – VSA Manager dummyPing invoked.
    2011-10-02 23:01:16,842 193 [Utils] [Thread-179] INFO – Retrieve dev.property: svaservice.detect.time=3
    2011-10-02 23:01:16,842 315 [SVAVMConfigurator] [Thread-179] INFO – Detecting VSA service waiting time is 3 min(s).
    2011-10-02 23:01:18,870 193 [Utils] [Thread-178] INFO – Retrieve dev.property: svaservice.detect.time=3
    2011-10-02 23:01:18,870 315 [SVAVMConfigurator] [Thread-178] INFO – Detecting VSA service waiting time is 3 min(s).
    2011-10-02 23:04:55,508 334 [SVAVMConfigurator] [Thread-178] ERROR – Cannot access VSA VM 172.16.100.34
    2011-10-02 23:04:55,508 391 [SVAVMConfigurator] [Thread-178] ERROR – Failed to ping VSA service on VM 172.16.100.34
    2011-10-02 23:04:55,508 242 [SVAVMConfigurator] [Thread-178] ERROR – VSA service is not running.
    2011-10-02 23:04:55,508 447 [SVAVMConfigurator] [Thread-178] ERROR – Customize VM on host host-326 failed
    java.lang.Exception: Failed to ping VSA service on VM 172.16.100.34
    at com.vmware.sva.manager.config.SVAVMConfigurator.connectToSVAService(SVAVMConfigurator.java:392)
    at com.vmware.sva.manager.config.SVAVMConfigurator.attemptSVAConnect(SVAVMConfigurator.java:240)
    at com.vmware.sva.manager.config.SVAVMConfigurator.initializeVM(SVAVMConfigurator.java:175)
    at com.vmware.sva.manager.config.SVAVMConfigurator.customizeSvaVM(SVAVMConfigurator.java:509)
    at com.vmware.sva.manager.config.SVAVMConfigurator.call(SVAVMConfigurator.java:445)
    at com.vmware.sva.manager.config.SVAVMConfigurator.call(SVAVMConfigurator.java:61)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)
    2011-10-02 23:04:55,508 334 [SVAVMConfigurator] [Thread-179] ERROR – Cannot access VSA VM 172.16.100.32
    2011-10-02 23:04:55,508 391 [SVAVMConfigurator] [Thread-179] ERROR – Failed to ping VSA service on VM 172.16.100.32
    2011-10-02 23:04:55,508 242 [SVAVMConfigurator] [Thread-179] ERROR – VSA service is not running.
    2011-10-02 23:04:55,508 447 [SVAVMConfigurator] [Thread-179] ERROR – Customize VM on host host-323 failed
    java.lang.Exception: Failed to ping VSA service on VM 172.16.100.32
    at com.vmware.sva.manager.config.SVAVMConfigurator.connectToSVAService(SVAVMConfigurator.java:392)
    at com.vmware.sva.manager.config.SVAVMConfigurator.attemptSVAConnect(SVAVMConfigurator.java:240)
    at com.vmware.sva.manager.config.SVAVMConfigurator.initializeVM(SVAVMConfigurator.java:175)
    at com.vmware.sva.manager.config.SVAVMConfigurator.customizeSvaVM(SVAVMConfigurator.java:509)
    at com.vmware.sva.manager.config.SVAVMConfigurator.call(SVAVMConfigurator.java:445)
    at com.vmware.sva.manager.config.SVAVMConfigurator.call(SVAVMConfigurator.java:61)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)
    2011-10-02 23:04:55,539 271 [VCUserSession] [Thread-560] ERROR – Customize VM on host host-326failed.
    java.util.concurrent.ExecutionException: java.lang.Exception: Failed to ping VSA service on VM 172.16.100.32
    at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
    at java.util.concurrent.FutureTask.get(FutureTask.java:83)
    at com.vmware.sva.manager.dao.VCUserSession.installSvaVms(VCUserSession.java:264)
    at com.vmware.sva.manager.service.CreateClusterThread.run(CreateClusterThread.java:156)
    Caused by: java.lang.Exception: Failed to ping VSA service on VM 172.16.100.32
    at com.vmware.sva.manager.config.SVAVMConfigurator.connectToSVAService(SVAVMConfigurator.java:392)
    at com.vmware.sva.manager.config.SVAVMConfigurator.attemptSVAConnect(SVAVMConfigurator.java:240)
    at com.vmware.sva.manager.config.SVAVMConfigurator.initializeVM(SVAVMConfigurator.java:175)
    at com.vmware.sva.manager.config.SVAVMConfigurator.customizeSvaVM(SVAVMConfigurator.java:509)
    at com.vmware.sva.manager.config.SVAVMConfigurator.call(SVAVMConfigurator.java:445)
    at com.vmware.sva.manager.config.SVAVMConfigurator.call(SVAVMConfigurator.java:61)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)
    2011-10-02 23:04:55,539 273 [VCUserSession] [Thread-560] ERROR – Cancelling the running customization tasks
    2011-10-02 23:04:55,555 193 [Utils] [Thread-560] INFO – Retrieve dev.property: vm.rollback=true
    2011-10-02 23:04:55,555 291 [VCUserSession] [Thread-560] INFO – Deleting all the installed VM
    2011-10-02 23:05:12,606 188 [CreateClusterThread] [Thread-560] ERROR – Create cluster failed:
    java.util.concurrent.ExecutionException: java.lang.Exception: Failed to ping VSA service on VM 172.16.100.32
    at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
    at java.util.concurrent.FutureTask.get(FutureTask.java:83)

    Any help that anyone can provide is much appreciated

    • In the dev.properties file located in c:\program files\VMware\infrastructure\tomcat\webapps\VSAManager\web-inf\classes

      change the value of this line from the default of 3 to 30 — the problem may be that your VSAs are not coming on line fast enough to meet the default timeout value (because your nested of ESX hosts are slower than what VMware thought would be the time necessary to get the VSAs up and running).

      svaservice.detect.time=30

      Datto

  4. thanks for the great article, works nicely on workstation 8 too

  5. I am trying to install VSA in a nested environment. The installation is proceeding to creation of clusters and then fails with error: datacenter – 2 timed out. From the error log i see that the SAS connection is getting timed out.

    I am pasting the logs just before the rollback below.

    2013-06-28 11:47:53,742 158 [BaseEventListener] [ActiveMQ Transport: ssl:///10.249.15.54:4334] INFO – TransportListener: onCommand: MessageDispatch {commandId = 0, responseRequired = false, consumerId = ID:WIN-BQT4D35O0VN-62004-1372441629589-0:1:1:1, destination = topic://cluster.topic, message = ActiveMQObjectMessage {commandId = 35, responseRequired = true, messageId = ID:localhost.localdom-55043-1372441765704-2:0:7:1:1, originalDestination = null, originalTransactionId = null, producerId = ID:localhost.localdom-55043-1372441765704-2:0:7:1, destination = topic://cluster.topic, transactionId = null, expiration = 0, timestamp = 1372441828750, arrival = 0, brokerInTime = 1372441828750, brokerOutTime = 1372441828750, correlationId = null, replyTo = null, persistent = true, type = null, priority = 4, groupID = null, groupSequence = 0, targetConsumerId = null, compressed = false, userID = null, content = org.apache.activemq.util.ByteSequence@657b8cd3, marshalledProperties = null, dataStructure = null, redeliveryCounter = 0, size = 0, properties = null, readOnlyProperties = true, readOnlyBody = true, droppable = false}, redeliveryCounter = 0}
    2013-06-28 11:47:53,742 158 [BaseEventListener] [ActiveMQ Transport: ssl:///10.249.15.54:4334] INFO – TransportListener: onCommand: MessageDispatch {commandId = 0, responseRequired = false, consumerId = ID:WIN-BQT4D35O0VN-62004-1372441629589-0:1:1:1, destination = topic://cluster.topic, message = ActiveMQObjectMessage {commandId = 40, responseRequired = true, messageId = ID:localhost.localdom-55043-1372441765704-2:0:8:1:1, originalDestination = null, originalTransactionId = null, producerId = ID:localhost.localdom-55043-1372441765704-2:0:8:1, destination = topic://cluster.topic, transactionId = null, expiration = 0, timestamp = 1372441828780, arrival = 0, brokerInTime = 1372441828780, brokerOutTime = 1372441828780, correlationId = null, replyTo = null, persistent = true, type = null, priority = 4, groupID = null, groupSequence = 0, targetConsumerId = null, compressed = false, userID = null, content = org.apache.activemq.util.ByteSequence@10dc65c0, marshalledProperties = null, dataStructure = null, redeliveryCounter = 0, size = 0, properties = null, readOnlyProperties = true, readOnlyBody = true, droppable = false}, redeliveryCounter = 0}
    2013-06-28 11:47:53,742 158 [BaseEventListener] [ActiveMQ Transport: ssl:///10.249.15.54:4334] INFO – TransportListener: onCommand: MessageDispatch {commandId = 0, responseRequired = false, consumerId = ID:WIN-BQT4D35O0VN-62004-1372441629589-0:1:1:1, destination = topic://cluster.topic, message = ActiveMQObjectMessage {commandId = 45, responseRequired = true, messageId = ID:localhost.localdom-55043-1372441765704-2:0:9:1:1, originalDestination = null, originalTransactionId = null, producerId = ID:localhost.localdom-55043-1372441765704-2:0:9:1, destination = topic://cluster.topic, transactionId = null, expiration = 0, timestamp = 1372441828784, arrival = 0, brokerInTime = 1372441828784, brokerOutTime = 1372441828784, correlationId = null, replyTo = null, persistent = true, type = null, priority = 4, groupID = null, groupSequence = 0, targetConsumerId = null, compressed = false, userID = null, content = org.apache.activemq.util.ByteSequence@759c28e5, marshalledProperties = null, dataStructure = null, redeliveryCounter = 0, size = 0, properties = null, readOnlyProperties = true, readOnlyBody = true, droppable = false}, redeliveryCounter = 0}
    2013-06-28 11:47:53,742 158 [BaseEventListener] [ActiveMQ Transport: ssl:///10.249.15.54:4334] INFO – TransportListener: onCommand: MessageDispatch {commandId = 0, responseRequired = false, consumerId = ID:WIN-BQT4D35O0VN-62004-1372441629589-0:1:1:1, destination = topic://cluster.topic, message = ActiveMQObjectMessage {commandId = 50, responseRequired = true, messageId = ID:localhost.localdom-55043-1372441765704-2:0:10:1:1, originalDestination = null, originalTransactionId = null, producerId = ID:localhost.localdom-55043-1372441765704-2:0:10:1, destination = topic://cluster.topic, transactionId = null, expiration = 0, timestamp = 1372441828800, arrival = 0, brokerInTime = 1372441828801, brokerOutTime = 1372441828801, correlationId = null, replyTo = null, persistent = true, type = null, priority = 4, groupID = null, groupSequence = 0, targetConsumerId = null, compressed = false, userID = null, content = org.apache.activemq.util.ByteSequence@643dc2ee, marshalledProperties = null, dataStructure = null, redeliveryCounter = 0, size = 0, properties = null, readOnlyProperties = true, readOnlyBody = true, droppable = false}, redeliveryCounter = 0}
    2013-06-28 11:47:53,805 74 [EventService] [ActiveMQ Session Task] INFO – Event StorageClusterOnlineEvent posted successfully
    2013-06-28 11:47:53,805 111 [ClusterEventHandlerImpl] [ActiveMQ Session Task] INFO – Received Cluster message of type StorageClusterOnlineEvent
    2013-06-28 11:47:53,820 232 [ClusterService] [ActiveMQ Session Task] INFO – Connect to SAS: 10.249.15.53
    2013-06-28 11:47:53,820 235 [Utils] [ActiveMQ Session Task] INFO – Retrieve dev.property: com.vmware.sva.manager.security.ssl.factory=null
    2013-06-28 11:47:54,101 126 [SVAManager] [tomcat-http–16] INFO – VSA Manager dummyPing invoked.
    2013-06-28 11:48:08,190 126 [SVAManager] [tomcat-http–5] INFO – VSA Manager dummyPing invoked.
    2013-06-28 11:48:24,102 126 [SVAManager] [tomcat-http–10] INFO – VSA Manager dummyPing invoked.
    2013-06-28 11:48:29,203 206 [FailoverTransport] [ActiveMQ Transport: ssl:///10.249.15.54:4334] WARN – Transport failed to ssl://10.249.15.54:4334?trace=true&daemon=true&keepAlive=true&soTimeout=60000 , attempting to automatically reconnect due to: java.net.SocketException: Connection reset
    2013-06-28 11:48:29,203 206 [FailoverTransport] [ActiveMQ Transport: ssl:///10.249.15.54:4334] WARN – Transport failed to ssl://10.249.15.54:4334?trace=true&daemon=true&keepAlive=true&soTimeout=60000 , attempting to automatically reconnect due to: java.net.SocketException: Connection reset
    2013-06-28 11:48:29,203 147 [BaseEventListener] [ActiveMQ Transport: ssl:///10.249.15.54:4334] INFO – TransportListener: transportInterupted
    2013-06-28 11:48:38,204 142 [BaseEventListener] [ActiveMQ Task] INFO – TransportListener: transportResumed
    2013-06-28 11:48:38,220 158 [BaseEventListener] [ActiveMQ Transport: ssl:///10.249.15.54:4334] INFO – TransportListener: onCommand: WireFormatInfo { version=3, properties={CacheSize=1024, CacheEnabled=true, SizePrefixDisabled=false, MaxInactivityDurationInitalDelay=10000, TcpNoDelayEnabled=true, MaxInactivityDuration=30000, TightEncodingEnabled=true, StackTraceEnabled=true}, magic=[A,c,t,i,v,e,M,Q]}
    2013-06-28 11:48:38,220 158 [BaseEventListener] [ActiveMQ Transport: ssl:///10.249.15.54:4334] INFO – TransportListener: onCommand: BrokerInfo {commandId = 0, responseRequired = false, brokerId = ID:localhost.localdom-51898-1372441868126-0:0, brokerURL = ssl://localhost.localdom:4334, slaveBroker = false, masterBroker = false, faultTolerantConfiguration = false, networkConnection = false, duplexConnection = false, peerBrokerInfos = [], brokerName = sva1, connectionId = 0, brokerUploadUrl = null, networkProperties = null}
    2013-06-28 11:48:38,235 126 [SVAManager] [tomcat-http–5] INFO – VSA Manager dummyPing invoked.
    2013-06-28 11:48:54,089 126 [SVAManager] [tomcat-http–10] INFO – VSA Manager dummyPing invoked.
    2013-06-28 11:49:08,285 126 [SVAManager] [tomcat-http–5] INFO – VSA Manager dummyPing invoked.
    2013-06-28 11:49:17,988 338 [ClusterEventHandlerImpl] [ActiveMQ Session Task] ERROR – Failed to initialize SAS Connection
    com.vmware.sva.manager.exception.LoginFailedException: Unable to login
    at com.vmware.sva.manager.service.ClusterService.loginSAS(ClusterService.java:124)
    at com.vmware.sva.manager.service.ClusterService.initializeSAS(ClusterService.java:236)
    at com.vmware.sva.manager.jms.ClusterEventHandlerImpl.handleStorageClusterOnlineEvent(ClusterEventHandlerImpl.java:335)
    at com.vmware.sva.manager.jms.ClusterEventHandlerImpl.clusterMessage(ClusterEventHandlerImpl.java:114)
    at com.vmware.sva.manager.jms.ClusterEventListener.dispatchClusterEvent(ClusterEventListener.java:67)
    at com.vmware.sva.manager.jms.ClusterEventListener.onMessage(ClusterEventListener.java:79)
    at org.apache.activemq.ActiveMQMessageConsumer.dispatch(ActiveMQMessageConsumer.java:1021)
    at org.apache.activemq.ActiveMQSessionExecutor.dispatch(ActiveMQSessionExecutor.java:122)
    at org.apache.activemq.ActiveMQSessionExecutor.iterate(ActiveMQSessionExecutor.java:192)
    at org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122)
    at org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)
    Caused by: org.apache.axis2.AxisFault: Connection timed out: connect
    at org.apache.axis2.AxisFault.makeFault(AxisFault.java:430)
    at org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:203)
    at org.apache.axis2.transport.http.HTTPSender.send(HTTPSender.java:76)
    at org.apache.axis2.transport.http.CommonsHTTPTransportSender.writeMessageWithCommons(CommonsHTTPTransportSender.java:400)
    at org.apache.axis2.transport.http.CommonsHTTPTransportSender.invoke(CommonsHTTPTransportSender.java:225)
    at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:435)
    at org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:402)
    at org.apache.axis2.description.OutInAxisOperationClient.executeImpl(OutInAxisOperation.java:229)
    at org.apache.axis2.client.OperationClient.execute(OperationClient.java:165)
    at com.vmware.sva.messaging.ws.sas.SasserviceStub.login(SasserviceStub.java:855)
    at com.vmware.sva.manager.service.SasMessagingService.login(SasMessagingService.java:62)
    at com.vmware.sva.manager.service.ClusterService.loginSAS(ClusterService.java:122)
    … 13 more
    Caused by: java.net.ConnectException: Connection timed out: connect
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.PlainSocketImpl.doConnect(Unknown Source)
    at java.net.PlainSocketImpl.connectToAddress(Unknown Source)
    at java.net.PlainSocketImpl.connect(Unknown Source)
    at java.net.SocksSocketImpl.connect(Unknown Source)
    at java.net.Socket.connect(Unknown Source)
    at com.sun.net.ssl.internal.ssl.SSLSocketImpl.connect(Unknown Source)
    at com.sun.net.ssl.internal.ssl.SSLSocketImpl.(Unknown Source)
    at com.sun.net.ssl.internal.ssl.SSLSocketFactoryImpl.createSocket(Unknown Source)
    at com.vmware.sva.manager.security.SSLSocketFactory.createSocket(SSLSocketFactory.java:71)
    at org.apache.commons.httpclient.protocol.ControllerThreadSocketFactory$1.doit(ControllerThreadSocketFactory.java:91)
    at org.apache.commons.httpclient.protocol.ControllerThreadSocketFactory$SocketTask.run(ControllerThreadSocketFactory.java:158)
    … 1 more
    2013-06-28 11:49:17,988 409 [UserSession] [ActiveMQ Session Task] INFO – Sending event to UI: com.vmware.sva.messaging.messages.BaseMessage@564307ac
    2013-06-28 11:49:24,088 126 [SVAManager] [tomcat-http–10] INFO – VSA Manager dummyPing invoked.
    2013-06-28 11:49:38,315 126 [SVAManager] [tomcat-http–5] INFO – VSA Manager dummyPing invoked.
    2013-06-28 11:49:54,118 126 [SVAManager] [tomcat-http–10] INFO – VSA Manager dummyPing invoked.
    2013-06-28 11:50:05,506 1479 [ClusterService] [SanityService-3] INFO – No Hosts have been added/removed since: 13-06-28 11:30:04
    2013-06-28 11:50:08,314 126 [SVAManager] [tomcat-http–5] INFO – VSA Manager dummyPing invoked.
    2013-06-28 11:50:24,116 126 [SVAManager] [tomcat-http–16] INFO – VSA Manager dummyPing invoked.
    2013-06-28 11:50:38,390 126 [SVAManager] [tomcat-http–5] INFO – VSA Manager dummyPing invoked.
    2013-06-28 11:50:54,115 126 [SVAManager] [tomcat-http–10] INFO – VSA Manager dummyPing invoked.
    2013-06-28 11:51:08,405 126 [SVAManager] [tomcat-http–5] INFO – VSA Manager dummyPing invoked.
    2013-06-28 11:51:24,114 126 [SVAManager] [tomcat-http–10] INFO – VSA Manager dummyPing invoked.
    2013-06-28 11:51:38,435 126 [SVAManager] [tomcat-http–23] INFO – VSA Manager dummyPing invoked.
    2013-06-28 11:51:54,113 126 [SVAManager] [tomcat-http–10] INFO – VSA Manager dummyPing invoked.
    2013-06-28 11:52:08,403 126 [SVAManager] [tomcat-http–23] INFO – VSA Manager dummyPing invoked.
    2013-06-28 11:52:24,112 126 [SVAManager] [tomcat-http–10] INFO – VSA Manager dummyPing invoked.
    2013-06-28 11:52:38,401 126 [SVAManager] [tomcat-http–5] INFO – VSA Manager dummyPing invoked.
    2013-06-28 11:52:50,164 235 [Utils] [pool-6-thread-25] INFO – Retrieve dev.property: com.vmware.sva.manager.security.ssl.factory=null
    2013-06-28 11:52:54,111 126 [SVAManager] [tomcat-http–10] INFO – VSA Manager dummyPing invoked.
    2013-06-28 11:53:08,416 126 [SVAManager] [tomcat-http–23] INFO – VSA Manager dummyPing invoked.
    2013-06-28 11:53:24,125 126 [SVAManager] [tomcat-http–7] INFO – VSA Manager dummyPing invoked.
    2013-06-28 11:53:38,415 126 [SVAManager] [tomcat-http–23] INFO – VSA Manager dummyPing invoked.
    2013-06-28 11:53:54,155 126 [SVAManager] [tomcat-http–7] INFO – VSA Manager dummyPing invoked.
    2013-06-28 11:54:08,492 126 [SVAManager] [tomcat-http–11] INFO – VSA Manager dummyPing invoked.
    2013-06-28 11:54:24,185 126 [SVAManager] [tomcat-http–7] INFO – VSA Manager dummyPing invoked.
    2013-06-28 11:54:38,475 126 [SVAManager] [tomcat-http–11] INFO – VSA Manager dummyPing invoked.
    2013-06-28 11:54:54,168 126 [SVAManager] [tomcat-http–7] INFO – VSA Manager dummyPing invoked.
    2013-06-28 11:55:05,588 1479 [ClusterService] [SanityService-3] INFO – No Hosts have been added/removed since: 13-06-28 11:30:04
    2013-06-28 11:55:08,474 126 [SVAManager] [tomcat-http–11] INFO – VSA Manager dummyPing invoked.
    2013-06-28 11:55:24,167 126 [SVAManager] [tomcat-http–10] INFO – VSA Manager dummyPing invoked.
    2013-06-28 11:55:38,473 126 [SVAManager] [tomcat-http–11] INFO – VSA Manager dummyPing invoked.
    2013-06-28 11:55:54,182 126 [SVAManager] [tomcat-http–7] INFO – VSA Manager dummyPing invoked.
    2013-06-28 11:56:08,471 126 [SVAManager] [tomcat-http–11] INFO – VSA Manager dummyPing invoked.
    2013-06-28 11:56:24,243 126 [SVAManager] [tomcat-http–36] INFO – VSA Manager dummyPing invoked.
    2013-06-28 11:56:38,470 126 [SVAManager] [tomcat-http–11] INFO – VSA Manager dummyPing invoked.
    2013-06-28 11:56:54,258 126 [SVAManager] [tomcat-http–36] INFO – VSA Manager dummyPing invoked.
    2013-06-28 11:57:08,485 126 [SVAManager] [tomcat-http–11] INFO – VSA Manager dummyPing invoked.
    2013-06-28 11:57:24,303 126 [SVAManager] [tomcat-http–36] INFO – VSA Manager dummyPing invoked.
    2013-06-28 11:57:26,394 106 [VSAEventForwarder] [pool-83-thread-1] INFO – Removed listener CreateStorageCluster-datacenter-2 for class com.vmware.sva.messaging.ws.model.mgmtevents.CreateStorageClusterTaskCompletedEvent
    2013-06-28 11:57:26,394 106 [VSAEventForwarder] [pool-83-thread-1] INFO – Removed listener CreateStorageCluster-datacenter-2 for class com.vmware.sva.messaging.ws.model.mgmtevents.CreateStorageClusterTaskFailedEvent
    2013-06-28 11:57:26,394 89 [DefaultTaskRunner] [pool-83-thread-1] INFO – Notifying listeners of task exception. TaskID:CreateCluster
    2013-06-28 11:57:26,394 80 [DefaultJobRunner] [pool-83-thread-1] ERROR – Exception caught. Starting rollback for job if rollback support . JobName:ClusterCreation-datacenter-2
    com.vmware.sva.manager.jobservices.JobErrorException: Creating cluster for datacenter: datacenter-2 timed out.
    at com.vmware.sva.manager.jobservices.tasks.TaskCreateCluster.execute(TaskCreateCluster.java:115)
    at com.vmware.sva.manager.jobservices.impl.DefaultTaskRunner.handleTaskStart(DefaultTaskRunner.java:70)
    at com.vmware.sva.manager.jobservices.impl.DefaultTaskRunner.call(DefaultTaskRunner.java:33)
    at com.vmware.sva.manager.jobservices.impl.SyncTaskGroup.execute(SyncTaskGroup.java:46)
    at com.vmware.sva.manager.jobservices.impl.DefaultJobRunner.run(DefaultJobRunner.java:60)
    at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
    at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
    at java.util.concurrent.FutureTask.run(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)
    2013-06-28 11:57:26,394 109 [DefaultJobRunner] [pool-83-thread-1] INFO – Rollback started for job. JobName:ClusterCreation-datacenter-2
    2013-06-28 11:57:26,394 116 [DefaultTaskRunner] [pool-83-thread-1] INFO – Notifying listeners of task rollback. TaskID:CreateCluster
    2013-06-28 11:57:26,394 136 [DefaultTaskRunner] [pool-83-thread-1] INFO – Starting rollback of task. TaskID:CreateCluster
    2013-06-28 11:57:26,409 324 [LogCollectionService] [pool-83-thread-1] INFO – Skip collecting upgrade logs: C:\Program Files\VMware\Infrastructure\tomcat\webapps\VSAManager\WEB-INF\logs
    2013-06-28 11:57:26,409 420 [LogCollectionService] [pool-83-thread-1] INFO – Number of nodes to collect logs from : 2
    2013-06-28 11:57:29,467 72 [MessagingService] [pool-83-thread-1] INFO – Using server endpoint for PSVA: https://10.249.15.43:4336/services/pseudosvaservice
    2013-06-28 11:57:29,763 239 [LogCollectionService] [pool-83-thread-1] INFO – Executing: java -jar WSCLI.jar 10.249.15.53 -pass ${password} getStorageCluster
    2013-06-28 11:57:38,484 126 [SVAManager] [tomcat-http–11] INFO – VSA Manager dummyPing invoked.
    2013-06-28 11:57:54,318 126 [SVAManager] [tomcat-http–36] INFO – VSA Manager dummyPing invoked.
    2013-06-28 11:58:08,482 126 [SVAManager] [tomcat-http–11] INFO – VSA Manager dummyPing invoked.
    2013-06-28 11:58:24,316 126 [SVAManager] [tomcat-http–36] INFO – VSA Manager dummyPing invoked.
    2013-06-28 11:58:38,481 126 [SVAManager] [tomcat-http–11] INFO – VSA Manager dummyPing invoked.
    2013-06-28 11:58:54,315 126 [SVAManager] [tomcat-http–36] INFO – VSA Manager dummyPing invoked.
    2013-06-28 11:58:55,548 258 [LogCollectionService] [pool-83-thread-1] INFO – Command finished with exitValue:-4. Command: java -jar WSCLI.jar 10.249.15.53 -pass ${password} getStorageCluster
    2013-06-28 11:58:55,548 265 [LogCollectionService] [pool-83-thread-1] INFO – File WSCLI/getStorageCluster_10.249.15.53_2013-06-28-11-57-29-763.txt written out
    2013-06-28 11:58:55,548 239 [LogCollectionService] [pool-83-thread-1] INFO – Executing: java -jar WSCLI.jar 10.249.15.53 -pass ${password} sasGetAllParameters
    2013-06-28 11:59:08,511 126 [SVAManager] [tomcat-http–23] INFO – VSA Manager dummyPing invoked.
    2013-06-28 11:59:24,314 126 [SVAManager] [tomcat-http–36] INFO – VSA Manager dummyPing invoked.
    2013-06-28 11:59:38,541 126 [SVAManager] [tomcat-http–42] INFO – VSA Manager dummyPing invoked.
    2013-06-28 11:59:54,313 126 [SVAManager] [tomcat-http–36] INFO – VSA Manager dummyPing invoked.
    2013-06-28 12:00:05,779 1479 [ClusterService] [SanityService-3] INFO – No Hosts have been added/removed since: 13-06-28 11:30:04
    2013-06-28 12:00:08,525 126 [SVAManager] [tomcat-http–42] INFO – VSA Manager dummyPing invoked.
    2013-06-28 12:00:21,348 258 [LogCollectionService] [pool-83-thread-1] INFO – Command finished with exitValue:-4. Command: java -jar WSCLI.jar 10.249.15.53 -pass ${password} sasGetAllParameters
    2013-06-28 12:00:21,348 265 [LogCollectionService] [pool-83-thread-1] INFO – File WSCLI/sasGetAllParameters_10.249.15.53_2013-06-28-11-58-55-548.txt written out
    2013-06-28 12:00:21,348 239 [LogCollectionService] [pool-83-thread-1] INFO – Executing: java -jar WSCLI.jar 10.249.15.53 -pass ${password} getTaskList
    2013-06-28 12:00:24,312 126 [SVAManager] [tomcat-http–36] INFO – VSA Manager dummyPing invoked.
    2013-06-28 12:00:38,523 126 [SVAManager] [tomcat-http–42] INFO – VSA Manager dummyPing invoked.
    2013-06-28 12:00:54,311 126 [SVAManager] [tomcat-http–36] INFO – VSA Manager dummyPing invoked.
    2013-06-28 12:01:08,538 126 [SVAManager] [tomcat-http–42] INFO – VSA Manager dummyPing invoked.
    2013-06-28 12:01:24,356 126 [SVAManager] [tomcat-http–27] INFO – VSA Manager dummyPing invoked.
    2013-06-28 12:01:38,537 126 [SVAManager] [tomcat-http–42] INFO – VSA Manager dummyPing invoked.
    2013-06-28 12:01:47,273 258 [LogCollectionService] [pool-83-thread-1] INFO – Command finished with exitValue:-4. Command: java -jar WSCLI.jar 10.249.15.53 -pass ${password} getTaskList
    2013-06-28 12:01:47,273 265 [LogCollectionService] [pool-83-thread-1] INFO – File WSCLI/getTaskList_10.249.15.53_2013-06-28-12-00-21-348.txt written out
    2013-06-28 12:01:47,273 239 [LogCollectionService] [pool-83-thread-1] INFO – Executing: java -jar WSCLI.jar 10.249.15.56 -pass ${password} getSvaServerInfo
    2013-06-28 12:01:49,566 258 [LogCollectionService] [pool-83-thread-1] INFO – Command finished with exitValue:0. Command: java -jar WSCLI.jar 10.249.15.56 -pass ${password} getSvaServerInfo
    2013-06-28 12:01:49,566 265 [LogCollectionService] [pool-83-thread-1] INFO – File WSCLI/getSvaServerInfo_10.249.15.56_2013-06-28-12-01-47-273.txt written out
    2013-06-28 12:01:49,566 239 [LogCollectionService] [pool-83-thread-1] INFO – Executing: java -jar WSCLI.jar 10.249.15.54 -pass ${password} getSvaServerInfo
    2013-06-28 12:01:51,781 258 [LogCollectionService] [pool-83-thread-1] INFO – Command finished with exitValue:0. Command: java -jar WSCLI.jar 10.249.15.54 -pass ${password} getSvaServerInfo
    2013-06-28 12:01:51,781 265 [LogCollectionService] [pool-83-thread-1] INFO – File WSCLI/getSvaServerInfo_10.249.15.54_2013-06-28-12-01-49-566.txt written out
    2013-06-28 12:01:51,828 193 [BaseEventListener] [pool-83-thread-1] INFO – Closing the JMS listener connection on 10.249.15.56
    2013-06-28 12:01:51,859 193 [BaseEventListener] [pool-83-thread-1] INFO – Closing the JMS listener connection on 10.249.15.56
    2013-06-28 12:01:51,859 193 [BaseEventListener] [pool-83-thread-1] INFO – Closing the JMS listener connection on 10.249.15.54
    2013-06-28 12:01:51,859 193 [BaseEventListener] [pool-83-thread-1] INFO – Closing the JMS listener connection on 10.249.15.54
    2013-06-28 12:01:51,859 196 [MappingService] [pool-83-thread-1] INFO – Removing context for datacenter-2
    2013-06-28 12:01:51,859 989 [VCUtils] [pool-83-thread-1] INFO – Removing VSA Info from Datacenter datacenter-2
    2013-06-28 12:01:51,937 138 [DefaultTaskRunner] [pool-83-thread-1] INFO – Task rollback finished with result:true. TaskID:CreateCluster
    2013-06-28 12:01:51,937 140 [DefaultTaskRunner] [pool-83-thread-1] INFO – Notifying listeners of task rollback end. TaskID:CreateCluster
    2013-06-28 12:01:51,937 116 [DefaultTaskRunner] [pool-83-thread-1] INFO – Notifying listeners of task rollback. TaskID:Task_DisableHA_DRS
    2013-06-28 12:01:51,937 136 [DefaultTaskRunner] [pool-83-thread-1] INFO – Starting rollback of task. TaskID:Task_DisableHA_DRS
    2013-06-28 12:01:52,109 138 [DefaultTaskRunner] [pool-83-thread-1] INFO – Task rollback finished with result:true. TaskID:Task_DisableHA_DRS
    2013-06-28 12:01:52,109 140 [DefaultTaskRunner] [pool-83-thread-1] INFO – Notifying listeners of task rollback end. TaskID:Task_DisableHA_DRS
    2013-06-28 12:01:52,109 116 [DefaultTaskRunner] [pool-83-thread-1] INFO – Notifying listeners of task rollback. TaskID:InstallsVSAVMs
    2013-06-28 12:01:52,109 136 [DefaultTaskRunner] [pool-83-thread-1] INFO – Starting rollback of task. TaskID:InstallsVSAVMs
    2013-06-28 12:01:52,109 235 [Utils] [pool-83-thread-1] INFO – Retrieve dev.property: vm.rollback=true
    2013-06-28 12:01:52,249 304 [CleanupUtils] [pool-83-thread-1] INFO – Powering off and deleting all SVA VMs
    2013-06-28 12:01:52,249 310 [CleanupUtils] [pool-83-thread-1] INFO – Received 2 SVA VMs
    2013-06-28 12:01:54,308 126 [SVAManager] [tomcat-http–27] INFO – VSA Manager dummyPing invoked.
    2013-06-28 12:02:08,551 126 [SVAManager] [tomcat-http–5] INFO – VSA Manager dummyPing invoked.
    2013-06-28 12:02:08,801 138 [DefaultTaskRunner] [pool-83-thread-1] INFO – Task rollback finished with result:true. TaskID:InstallsVSAVMs
    2013-06-28 12:02:08,801 140 [DefaultTaskRunner] [pool-83-thread-1] INFO – Notifying listeners of task rollback end. TaskID:InstallsVSAVMs
    2013-06-28 12:02:08,801 116 [DefaultTaskRunner] [pool-83-thread-1] INFO – Notifying listeners of task rollback. TaskID:ConfigureHA
    2013-06-28 12:02:08,801 136 [DefaultTaskRunner] [pool-83-thread-1] INFO – Starting rollback of task. TaskID:ConfigureHA
    2013-06-28 12:02:08,801 235 [Utils] [pool-83-thread-1] INFO – Retrieve dev.property: ha.config=true
    2013-06-28 12:02:24,307 126 [SVAManager] [tomcat-http–36] INFO – VSA Manager dummyPing invoked.
    2013-06-28 12:02:38,535 126 [SVAManager] [tomcat-http–5] INFO – VSA Manager dummyPing invoked.
    2013-06-28 12:02:54,369 126 [SVAManager] [tomcat-http–15] INFO – VSA Manager dummyPing invoked.
    2013-06-28 12:03:08,533 126 [SVAManager] [tomcat-http–5] INFO – VSA Manager dummyPing invoked.
    2013-06-28 12:03:24,352 126 [SVAManager] [tomcat-http–15] INFO – VSA Manager dummyPing invoked.
    2013-06-28 12:03:38,095 138 [DefaultTaskRunner] [pool-83-thread-1] INFO – Task rollback finished with result:true. TaskID:ConfigureHA
    2013-06-28 12:03:38,095 140 [DefaultTaskRunner] [pool-83-thread-1] INFO – Notifying listeners of task rollback end. TaskID:ConfigureHA
    2013-06-28 12:03:38,095 116 [DefaultTaskRunner] [pool-83-thread-1] INFO – Notifying listeners of task rollback. TaskID:NetworkConfig
    2013-06-28 12:03:38,095 136 [DefaultTaskRunner] [pool-83-thread-1] INFO – Starting rollback of task. TaskID:NetworkConfig
    2013-06-28 12:03:38,095 235 [Utils] [pool-83-thread-1] INFO – Retrieve dev.property: network.config=true
    2013-06-28 12:03:38,095 57 [TaskNetworkConfig] [pool-83-thread-1] INFO – Networking was not configured. Thus, don’t revert to greenfield networking.
    2013-06-28 12:03:38,095 138 [DefaultTaskRunner] [pool-83-thread-1] INFO – Task rollback finished with result:false. TaskID:NetworkConfig
    2013-06-28 12:03:38,095 140 [DefaultTaskRunner] [pool-83-thread-1] INFO – Notifying listeners of task rollback end. TaskID:NetworkConfig
    2013-06-28 12:03:38,095 116 [DefaultTaskRunner] [pool-83-thread-1] INFO – Notifying listeners of task rollback. TaskID:AddTagsToHosts
    2013-06-28 12:03:38,095 136 [DefaultTaskRunner] [pool-83-thread-1] INFO – Starting rollback of task. TaskID:AddTagsToHosts
    2013-06-28 12:03:38,298 138 [DefaultTaskRunner] [pool-83-thread-1] INFO – Task rollback finished with result:true. TaskID:AddTagsToHosts
    2013-06-28 12:03:38,298 140 [DefaultTaskRunner] [pool-83-thread-1] INFO – Notifying listeners of task rollback end. TaskID:AddTagsToHosts
    2013-06-28 12:03:38,298 116 [DefaultTaskRunner] [pool-83-thread-1] INFO – Notifying listeners of task rollback. TaskID:Run2NLV
    2013-06-28 12:03:38,298 136 [DefaultTaskRunner] [pool-83-thread-1] INFO – Starting rollback of task. TaskID:Run2NLV
    2013-06-28 12:03:38,298 72 [MessagingService] [pool-83-thread-1] INFO – Using server endpoint for PSVA: https://10.249.15.43:4336/services/pseudosvaservice
    2013-06-28 12:03:38,532 126 [SVAManager] [tomcat-http–5] INFO – VSA Manager dummyPing invoked.
    2013-06-28 12:03:41,871 46 [TaskRun2NLV] [pool-83-thread-1] INFO – Destroyed VCS with best effort
    2013-06-28 12:03:41,871 138 [DefaultTaskRunner] [pool-83-thread-1] INFO – Task rollback finished with result:true. TaskID:Run2NLV
    2013-06-28 12:03:41,871 140 [DefaultTaskRunner] [pool-83-thread-1] INFO – Notifying listeners of task rollback end. TaskID:Run2NLV
    2013-06-28 12:03:41,871 122 [DefaultJobRunner] [pool-83-thread-1] INFO – TaskGroup rolled back successfully.TaskGroupName:ClusterCreation-PostHostReboot
    2013-06-28 12:03:41,871 83 [AsyncTaskGroup] [pool-83-thread-1] INFO – Cancelling running tasks for taskGroup. TaskGroupName:Set Hosts VMFS Heap Size
    2013-06-28 12:03:41,871 116 [DefaultTaskRunner] [pool-84-thread-3] INFO – Notifying listeners of task rollback. TaskID:SetVMFSHeapSizehost-140
    2013-06-28 12:03:41,871 136 [DefaultTaskRunner] [pool-84-thread-3] INFO – Starting rollback of task. TaskID:SetVMFSHeapSizehost-140
    2013-06-28 12:03:41,871 138 [DefaultTaskRunner] [pool-84-thread-3] INFO – Task rollback finished with result:true. TaskID:SetVMFSHeapSizehost-140
    2013-06-28 12:03:41,871 140 [DefaultTaskRunner] [pool-84-thread-3] INFO – Notifying listeners of task rollback end. TaskID:SetVMFSHeapSizehost-140
    2013-06-28 12:03:41,871 116 [DefaultTaskRunner] [pool-84-thread-4] INFO – Notifying listeners of task rollback. TaskID:SetVMFSHeapSizehost-130
    2013-06-28 12:03:41,871 136 [DefaultTaskRunner] [pool-84-thread-4] INFO – Starting rollback of task. TaskID:SetVMFSHeapSizehost-130
    2013-06-28 12:03:41,871 138 [DefaultTaskRunner] [pool-84-thread-4] INFO – Task rollback finished with result:true. TaskID:SetVMFSHeapSizehost-130
    2013-06-28 12:03:41,871 140 [DefaultTaskRunner] [pool-84-thread-4] INFO – Notifying listeners of task rollback end. TaskID:SetVMFSHeapSizehost-130
    2013-06-28 12:03:41,871 122 [DefaultJobRunner] [pool-83-thread-1] INFO – TaskGroup rolled back successfully.TaskGroupName:Set Hosts VMFS Heap Size
    2013-06-28 12:03:41,871 116 [DefaultTaskRunner] [pool-83-thread-1] INFO – Notifying listeners of task rollback. TaskID:Audit2N-LV
    2013-06-28 12:03:41,871 136 [DefaultTaskRunner] [pool-83-thread-1] INFO – Starting rollback of task. TaskID:Audit2N-LV
    2013-06-28 12:03:41,871 138 [DefaultTaskRunner] [pool-83-thread-1] INFO – Task rollback finished with result:true. TaskID:Audit2N-LV
    2013-06-28 12:03:41,871 140 [DefaultTaskRunner] [pool-83-thread-1] INFO – Notifying listeners of task rollback end. TaskID:Audit2N-LV
    2013-06-28 12:03:41,871 116 [DefaultTaskRunner] [pool-83-thread-1] INFO – Notifying listeners of task rollback. TaskID:NetworkConfigValidation
    2013-06-28 12:03:41,871 136 [DefaultTaskRunner] [pool-83-thread-1] INFO – Starting rollback of task. TaskID:NetworkConfigValidation
    2013-06-28 12:03:41,871 138 [DefaultTaskRunner] [pool-83-thread-1] INFO – Task rollback finished with result:true. TaskID:NetworkConfigValidation
    2013-06-28 12:03:41,871 140 [DefaultTaskRunner] [pool-83-thread-1] INFO – Notifying listeners of task rollback end. TaskID:NetworkConfigValidation
    2013-06-28 12:03:41,871 116 [DefaultTaskRunner] [pool-83-thread-1] INFO – Notifying listeners of task rollback. TaskID:AuditHostGroup
    2013-06-28 12:03:41,886 136 [DefaultTaskRunner] [pool-83-thread-1] INFO – Starting rollback of task. TaskID:AuditHostGroup
    2013-06-28 12:03:41,886 138 [DefaultTaskRunner] [pool-83-thread-1] INFO – Task rollback finished with result:true. TaskID:AuditHostGroup
    2013-06-28 12:03:41,886 140 [DefaultTaskRunner] [pool-83-thread-1] INFO – Notifying listeners of task rollback end. TaskID:AuditHostGroup
    2013-06-28 12:03:41,886 116 [DefaultTaskRunner] [pool-83-thread-1] INFO – Notifying listeners of task rollback. TaskID:ClusterExists
    2013-06-28 12:03:41,886 136 [DefaultTaskRunner] [pool-83-thread-1] INFO – Starting rollback of task. TaskID:ClusterExists
    2013-06-28 12:03:41,886 138 [DefaultTaskRunner] [pool-83-thread-1] INFO – Task rollback finished with result:true. TaskID:ClusterExists
    2013-06-28 12:03:41,886 140 [DefaultTaskRunner] [pool-83-thread-1] INFO – Notifying listeners of task rollback end. TaskID:ClusterExists
    2013-06-28 12:03:41,886 116 [DefaultTaskRunner] [pool-83-thread-1] INFO – Notifying listeners of task rollback. TaskID:LogParams
    2013-06-28 12:03:41,886 136 [DefaultTaskRunner] [pool-83-thread-1] INFO – Starting rollback of task. TaskID:LogParams
    2013-06-28 12:03:41,886 138 [DefaultTaskRunner] [pool-83-thread-1] INFO – Task rollback finished with result:true. TaskID:LogParams
    2013-06-28 12:03:41,886 140 [DefaultTaskRunner] [pool-83-thread-1] INFO – Notifying listeners of task rollback end. TaskID:LogParams
    2013-06-28 12:03:41,886 116 [DefaultTaskRunner] [pool-83-thread-1] INFO – Notifying listeners of task rollback. TaskID:TaskInitCreateCluster
    2013-06-28 12:03:41,886 136 [DefaultTaskRunner] [pool-83-thread-1] INFO – Starting rollback of task. TaskID:TaskInitCreateCluster
    2013-06-28 12:03:41,886 72 [TaskInitCreateCluster] [pool-83-thread-1] ERROR – Failed to clean up persisted data for cluster creation task. Task has already been cleaned up.
    2013-06-28 12:03:41,886 196 [MappingService] [pool-83-thread-1] INFO – Removing context for datacenter-2
    2013-06-28 12:03:42,042 171 [SvaSolutionLicenseManager] [pool-83-thread-1] INFO – ReleaseLicense – Updating license count to : 0
    2013-06-28 12:03:42,073 83 [TaskInitCreateCluster] [pool-83-thread-1] INFO – Released license for datacenter: datacenter-2
    2013-06-28 12:03:42,073 138 [DefaultTaskRunner] [pool-83-thread-1] INFO – Task rollback finished with result:true. TaskID:TaskInitCreateCluster
    2013-06-28 12:03:42,073 140 [DefaultTaskRunner] [pool-83-thread-1] INFO – Notifying listeners of task rollback end. TaskID:TaskInitCreateCluster
    2013-06-28 12:03:42,073 122 [DefaultJobRunner] [pool-83-thread-1] INFO – TaskGroup rolled back successfully.TaskGroupName:ClusterCreation-PreHostReboot
    2013-06-28 12:03:42,073 84 [DefaultJobRunner] [pool-83-thread-1] ERROR – Rollback finished successfully for job. JobName:ClusterCreation-datacenter-2
    2013-06-28 12:03:42,073 92 [DefaultJobRunner] [pool-83-thread-1] INFO – Notifying listeners of job exception for job: ClusterCreation-datacenter-2
    2013-06-28 12:03:54,787 126 [SVAManager] [tomcat-http–13] INFO – VSA Manager dummyPing invoked.
    2013-06-28 12:04:08,578 126 [SVAManager] [tomcat-http–5] INFO – VSA Manager dummyPing invoked.
    2013-06-28 12:04:24,365 126 [SVAManager] [tomcat-http–13] INFO – VSA Manager dummyPing invoked.
    2013-06-28 12:04:38,546 126 [SVAManager] [tomcat-http–5] INFO – VSA Manager dummyPing invoked.
    2013-06-28 12:04:54,411 126 [SVAManager] [tomcat-http–13] INFO – VSA Manager dummyPing invoked.
    2013-06-28 12:05:05,892 1505 [ClusterService] [SanityService-3] INFO – Host added/removed from/to DC: datacenter-2, which does not contain a VSA Cluster.
    2013-06-28 12:05:08,544 126 [SVAManager] [tomcat-http–5] INFO – VSA Manager dummyPing invoked.
    2013-06-28 12:05:24,410 126 [SVAManager] [tomcat-http–13] INFO – VSA Manager dummyPing invoked.
    2013-06-28 12:05:38,543 126 [SVAManager] [tomcat-http–5] INFO – VSA Manager dummyPing invoked.
    2013-06-28 12:05:54,408 126 [SVAManager] [tomcat-http–15] INFO – VSA Manager dummyPing invoked.
    2013-06-28 12:06:08,542 126 [SVAManager] [tomcat-http–5] INFO – VSA Manager dummyPing invoked.
    2013-06-28 12:06:24,407 126 [SVAManager] [tomcat-http–13] INFO – VSA Manager dummyPing invoked.
    2013-06-28 12:06:38,557 126 [SVAManager] [tomcat-http–5] INFO – VSA Manager dummyPing invoked.
    2013-06-28 12:06:54,437 126 [SVAManager] [tomcat-http–13] INFO – VSA Manager dummyPing invoked.
    2013-06-28 12:07:08,587 126 [SVAManager] [tomcat-http–38] INFO – VSA Manager dummyPing invoked.
    2013-06-28 12:07:24,436 126 [SVAManager] [tomcat-http–13] INFO – VSA Manager dummyPing invoked.
    2013-06-28 12:07:38,554 126 [SVAManager] [tomcat-http–38] INFO – VSA Manager dummyPing invoked.
    2013-06-28 12:07:54,435 126 [SVAManager] [tomcat-http–13] INFO – VSA Manager dummyPing invoked.
    2013-06-28 12:08:08,553 126 [SVAManager] [tomcat-http–5] INFO – VSA Manager dummyPing invoked.
    2013-06-28 12:08:24,434 126 [SVAManager] [tomcat-http–13] INFO – VSA Manager dummyPing invoked.
    2013-06-28 12:08:38,568 126 [SVAManager] [tomcat-http–38] INFO – VSA Manager dummyPing invoked.
    2013-06-28 12:08:54,433 126 [SVAManager] [tomcat-http–13] INFO – VSA Manager dummyPing invoked.
    2013-06-28 12:09:08,566 126 [SVAManager] [tomcat-http–38] INFO – VSA Manager dummyPing invoked.
    2013-06-28 12:09:24,432 126 [SVAManager] [tomcat-http–13] INFO – VSA Manager dummyPing invoked.
    2013-06-28 12:09:38,581 126 [SVAManager] [tomcat-http–9] INFO – VSA Manager dummyPing invoked.
    2013-06-28 12:09:54,508 126 [SVAManager] [tomcat-http–10] INFO – VSA Manager dummyPing invoked.
    2013-06-28 12:10:05,974 1479 [ClusterService] [SanityService-3] INFO – No Hosts have been added/removed since: 13-06-28 12:05:05
    2013-06-28 12:10:08,564 126 [SVAManager] [tomcat-http–9] INFO – VSA Manager dummyPing invoked.
    2013-06-28 12:10:24,476 126 [SVAManager] [tomcat-http–10] INFO – VSA Manager dummyPing invoked.
    2013-06-28 12:10:38,563 126 [SVAManager] [tomcat-http–9] INFO – VSA Manager dummyPing invoked.
    2013-06-28 12:10:54,475 126 [SVAManager] [tomcat-http–13] INFO – VSA Manager dummyPing invoked.
    2013-06-28 12:11:08,577 126 [SVAManager] [tomcat-http–9] INFO – VSA Manager dummyPing invoked.
    2013-06-28 12:11:24,474 126 [SVAManager] [tomcat-http–10] INFO – VSA Manager dummyPing invoked.
    2013-06-28 12:11:38,576 126 [SVAManager] [tomcat-http–9] INFO – VSA Manager dummyPing invoked.
    2013-06-28 12:11:54,519 126 [SVAManager] [tomcat-http–5] INFO – VSA Manager dummyPing invoked.
    2013-06-28 12:12:08,575 126 [SVAManager] [tomcat-http–9] INFO – VSA Manager dummyPing invoked.
    2013-06-28 12:12:24,503 126 [SVAManager] [tomcat-http–5] INFO – VSA Manager dummyPing invoked.
    2013-06-28 12:12:38,574 126 [SVAManager] [tomcat-http–9] INFO – VSA Manager dummyPing invoked.
    2013-06-28 12:12:54,502 126 [SVAManager] [tomcat-http–5] INFO – VSA Manager dummyPing invoked.
    2013-06-28 12:13:08,573 126 [SVAManager] [tomcat-http–9] INFO – VSA Manager dummyPing invoked.
    2013-06-28 12:13:24,500 126 [SVAManager] [tomcat-http–5] INFO – VSA Manager dummyPing invoked.
    2013-06-28 12:13:38,587 126 [SVAManager] [tomcat-http–38] INFO – VSA Manager dummyPing invoked.
    2013-06-28 12:13:54,546 126 [SVAManager] [tomcat-http–5] INFO – VSA Manager dummyPing invoked.
    2013-06-28 12:14:08,586 126 [SVAManager] [tomcat-http–9] INFO – VSA Manager dummyPing invoked.
    2013-06-28 12:14:24,545 126 [SVAManager] [tomcat-http–5] INFO – VSA Manager dummyPing invoked.
    2013-06-28 12:14:38,632 126 [SVAManager] [tomcat-http–9] INFO – VSA Manager dummyPing invoked.
    2013-06-28 12:14:54,559 126 [SVAManager] [tomcat-http–5] INFO – VSA Manager dummyPing invoked.
    2013-06-28 12:15:06,057 1479 [ClusterService] [SanityService-3] INFO – No Hosts have been added/removed since: 13-06-28 12:05:05
    2013-06-28 12:15:08,662 126 [SVAManager] [tomcat-http–35] INFO – VSA Manager dummyPing invoked.
    2013-06-28 12:15:24,605 126 [SVAManager] [tomcat-http–5] INFO – VSA Manager dummyPing invoked.
    2013-06-28 12:15:38,645 126 [SVAManager] [tomcat-http–35] INFO – VSA Manager dummyPing invoked.
    2013-06-28 12:15:54,604 126 [SVAManager] [tomcat-http–5] INFO – VSA Manager dummyPing invoked.
    2013-06-28 12:16:08,644 126 [SVAManager] [tomcat-http–35] INFO – VSA Manager dummyPing invoked.
    2013-06-28 12:16:24,603 126 [SVAManager] [tomcat-http–10] INFO – VSA Manager dummyPing invoked.
    2013-06-28 12:16:38,643 126 [SVAManager] [tomcat-http–35] INFO – VSA Manager dummyPing invoked.
    2013-06-28 12:16:54,602 126 [SVAManager] [tomcat-http–5] INFO – VSA Manager dummyPing invoked.
    2013-06-28 12:17:08,642 126 [SVAManager] [tomcat-http–35] INFO – VSA Manager dummyPing invoked.
    2013-06-28 12:17:24,600 126 [SVAManager] [tomcat-http–5] INFO – VSA Manager dummyPing invoked.
    2013-06-28 12:17:38,672 126 [SVAManager] [tomcat-http–26] INFO – VSA Manager dummyPing invoked.
    2013-06-28 12:17:54,615 126 [SVAManager] [tomcat-http–7] INFO – VSA Manager dummyPing invoked.
    2013-06-28 12:18:01,510 196 [UserSession] [tomcat-http–5] INFO – **** extenMoref **** Datacenter:datacenter-2
    2013-06-28 12:18:01,526 324 [LogCollectionService] [tomcat-http–5] INFO – Skip collecting upgrade logs: C:\Program Files\VMware\Infrastructure\tomcat\webapps\VSAManager\WEB-INF\logs
    2013-06-28 12:18:01,526 420 [LogCollectionService] [tomcat-http–5] INFO – Number of nodes to collect logs from : 2
    2013-06-28 12:18:08,686 126 [SVAManager] [tomcat-http–26] INFO – VSA Manager dummyPing invoked.
    2013-06-28 12:18:22,898 432 [LogCollectionService] [tomcat-http–5] ERROR – Failed to collect logs from VSA 10.249.15.56
    org.apache.axis2.AxisFault: Read timed out

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: