Monthly Archives: June 2016

VPLEX GUI Not Showing Performance Counters

Solution: VPLEX: How to resolve GUI default monitors not reporting events

Follow the steps below to resolve this issue:

  1. Log in to the affected Cluster’s VPLEXCLI. To get into engineering mode, enter the command:

debug engineering-mode on

Example:

VPlexcli:/> debug engineering-mode on

The CLI is now in engineering mode.

  1. Change directory (cd) to /monitoring/directors/<affected director>/monitors.

Example:

VPlexcli:/> cd /monitoring/directors/director-1-1-A/monitors

  1. Type ll to see the list of monitors running, and look for the one listed where vXX will equal a value like v17

director-1-1-A_PERPETUAL_vplex_sys_perf_mon_vX

Example:

VPlexcli:/monitoring/directors/director-1-1-A/monitors> ll

Name                                             Ownership  Collecting  Period  Average  Idle     Bucket  Bucket  Bucket  Bucket
———————————————–  ———  Data        ——  Period   For      Min     Max     Width   Count
———————————————–  ———  ———-  ——  ——-  ——-  ——  ——  ——  ——
DEFAULT_director-1-1-A_director-FE-stats         false      false       5s      –        103days  –       –       –       64
DEFAULT_director-1-1-A_testmon                   false      false       5s      –        103days  –       –       –       64
director-1-1-A_Justintest                        true       false       5s      –        103days  –       –       –       64
director-1-1-A_PERPETUAL_vplex_sys_perf_mon_v17  true       false       5s      5s       0s       –       –       –       64
director-1-1-A_SETH_VIRTUAL                      true       true        5min    5min     4.5min   –       –       –       64
director-1-1-A_directorbusy                      true       false       5s      –        103days  –       –       –       64
director-1-1-A_diskReportMonitor                 true       true        0s      –        103days  –       –       –       64
director-1-1-A_portReportMonitor                 true       true        0s      –        103days  –       –       –       64
director-1-1-A_test                              true       true        5s      5s       1s       –       –       –       64
director-1-1-A_testtest                          true       false       0s      –        103days  –       –       –       64
director-1-1-A_volumeReportMonitor               true       true        0s      –        103days  –       –       –       64
director-1-1-A_vvol                              true       true        5s      5s       1s       –       –       –       64

  1. Destroy the default monitor by issuing the command:

monitor destroy DEFAULT_director-1-1-A_PERPETUAL_vplex_sys_perf_mon_vXX

This will destroy the monitor for director-1-1-A.

  1. Repeat the above step for each directors monitors of the affected cluster.

PERPETUAL_vplex_sys_perf_mon_vXX

  1. Exit VPLEXCLI. This will take you back to the management server prompt.

VPlexcli:/> exit

  1. From the management server prompt, restart the VPlexManagementConsole by issuing the command:

sudo /etc/init.d/VPlexManagementConsole restart

example:

service@ManagementServer:~> sudo /etc/init.d/VPlexManagementConsole restart

This will recreate the default monitor files.

  1. Log back in to the VPlexcli and repeat Step 2 above to check the column labeled ownership for each PERPETUAL_vplex_sys_perf_mon_vXX monitors of each director that was affected. The file will now show as true.

 

  1. Open a browser window and log in to the VPLEX GUI to verify that the GUI stats are now being displayed.
  2. From the VPlexcli prompt, disable the debug engineering mode by typing the command:

 

debug engineering-mode off

Example:

VPlexcli:/> debug engineering-mode off

The CLI is now not in engineering mode.

  1. Once completed with the above steps, exit the VPlexcli and the management server and close the VPLEX GUI browser session.

OR

In case after management station restart the counters dis appear follow below

Solution: VPLEX: GeoSynchrony 5.4 SP1 – Perpetual monitors do not start automatically, workaround required

Option 1
This command will append “ll /monitoring/directors/*/monitors/” to the VPlexcli-init file, if it’s not already present.

From the management station, enter the following command (all on one line):

  • service@VPlex01:~> grep -cq ‘ll /monitoring’ /var/log/VPlex/cli/VPlexcli-init|| echo ‘ll /monitoring/directors/*/monitors/’ >> /var/log/VPlex/cli/VPlexcli-init

Option 2
The manual editing method:

  1. Edit the CLI start-up command file /var/log/VPlex/cli/VPlexcli-init

service@VPlex01:~> vi /var/log/VPlex/cli/VPlexcli-init

2. Add the line below to the end of the file (MUST be at the end of the file):
       ll /monitoring/directors/*/monitors/

3. To add this line using VI, open the file as shown in step 1 above, then move the cursor to the current last line in the file using the down arrow key or the ‘ j ‘ key or by pressing shift-G then return. Either of these three actions will get you to the end of the current file.

4. Once at the end of the file, if the cursor is not on a line of text press ‘ a ‘ to append the new line, you will see ‘ — INSERT –‘ appear at the bottom of the screen. Copy and paste the the command line from step 2. If the line shows indented, move the cursor to the front of the line and onto the first character using the left arrow key or the ‘ h ‘ key.
Once at the front of the line, and on the first character use the backspace key to move
the whole line to the left hand edge of the screen.

service@ManagementServer:~> cat /var/log/VPlex/cli/VPlexcli-init
#——————————————————————————-
#- (C) 2007-2010 EMC Corporation. All rights reserved.
#-
#- This CLI initialization script is executed if it’s located in any of the
#- following locations:
#- (CLI terminates the search on first success.)
#- if the –init-file option is specified on the command line then use that file
#- else search for the file “VPlexcli-init” in the following order:
#-   a.  CLI directory (specified with the –cli-directory option)
#-   b.  current dir (of the shell that started CLI)
#-   c.  user.dir (usually equivalent to the current dir)
#-   d.  user.home
#-   e.  classpath
#- This script is processed as if it had been sourced using the ‘source’ command
#——————————————————————————-
#——————————————————————————-
#- start the webserver
#——————————————————————————-
webserver start
#——————————————————————————-
#- connect to any directors listed in the ‘connections’ file in this CLI directory
#——————————————————————————-
connect –no-prompt -c
#——————————————————————————-
#- connect to the directors in your system
#- WARNING: do not use both the ‘connections’ file method and this method
#  at the same time.  Connecting to the same director twice can have
#  unwanted side-effects.
#——————————————————————————-
#connect xxx.xxx.xxx.xx –secondary-host xxx.xxx.xxx.xx –name director-1-1-A
#connect xxx.xxx.xxx.xx –secondary-host xxx.xxx.xxx.xx –name director-1-1-B
#connect xxx.xxx.xxx.xx –secondary-host xxx.xxx.xxx.xx –name director-2-1-A
#connect xxx.xxx.xxx.xx –secondary-host xxx.xxx.xxx.xx –name director-2-1-B

ll /monitoring/directors/*/monitors/
— INSERT —
5. Once you have the line from step 2 inserted and lined up with the left hand edge of the screen press the ‘ Esc ‘ key to end the append session. The ‘ — INSERT — ‘ will
disappear from the bottom of the screen.

6. Save the changes and exit vi.

7. To save the change(s) and exit vi press the shift-key and then the colon ( : ) then release the shift-key and press wq!

8. Cat the file to ensure the new entered line is there.

9. Next time the CLI restarts, the perpetual monitors should start like they should instead of stall.

 

Deprecated VMFS volume warning reported by ESXi hosts

This issue occurs because the version of the filesystem is not known during the initial detection. Therefore, comparing it against the list of valid filesystems does not return a match.

Resolution

This is a known issue affecting vCenter Server 6.0.

This issue is resolved in vCenter Server 6.0 Update 2, available at VMware Downloads.

To work around this issue, restart the management agents on the impacted hosts to clear the warning.

From the Local Console or SSH:

  1. Log in to SSH or Local console as root.
  2. Run these commands:

    /etc/init.d/hostd restart
    /etc/init.d/vpxa restart

Stopping, starting, or restarting vCenter Server Appliance services

Checking the status vCenter Server Appliance services

To check the status of vCenter Server Appliance services:

  1. Connect to the vCenter Server Appliance through SSH. For more information, see Enable or Disable SSH Administrator Login on the VMware vCenter Server Appliance section in the vCenter Server and Host Management Guide.
  2. Log in as root.
  3. Run this command in the SSH client:

    service –-status-all

  4. Scroll through the output to check the status of all services.

To check the status of a specific service:

  1. Run this command to review the list of all services:

    ls /etc/init.d

  2. Scroll through the output and find the name of the service you are interested in. For a list of core vCenter Server services, see theAdditional Information section of this article.
  3. Run this command in the SSH client:

    service ServiceName status

    Where ServiceName corresponds to one of the services in /etc/init.d or the core list in the Additional Information section of this article. For example:

    service vmware-vpxd status

    If the vpxd service is running, you see output similar to:

    vmware-vpxd is running
    tomcat is running

Note: The tomcat service is within the vmware-vpxd service. You can check the status of tomcat by checking the status of vmware-vpxd. This service can be stopped and started outside of the vmware-vpxd service by running these commands:

vmware-vpxd tomcat-stop

vmware-vpxd tomcat-start

Stopping vCenter Server Appliance services

To stop a vCenter Server Appliance service if it has started:

  1. Connect to the vCenter Server Appliance via SSH. For more information, see Enable or Disable SSH Administrator Login on the VMware vCenter Server Appliance section in the vCenter Server and Host Management Guide.
  2. Log in as root.
  3. Run this command to review the list of all services:

    ls /etc/init.d

  4. Scroll through the output and find the name of the service you are interested in. For a list of core vCenter Server services, see theAdditional Information section of this article.
  5. Run this command in the SSH client:

    service ServiceName stop

    Where ServiceName corresponds to one of the services in /etc/init.d.

Note: When you stop the VMware VirtualCenter Server service (vmware-vpxd), all connected users are disconnected from their session.

Starting vCenter Server Appliance services

To start a vCenter Server Appliance service if it has stopped:

  1. Connect to the vCenter Server Appliance through SSH. For more information, see Enable or Disable SSH Administrator Login on the VMware vCenter Server Appliance section in the vCenter Server and Host Management Guide.
  2. Log in as root.
  3. Run this command to review the list of all services:

    ls /etc/init.d

  4. Scroll through the output and find the name of the service you are interested in. For a list of core vCenter Server services, see theAdditional Information section of this article.
  5. Run this command in the SSH client:

    service ServiceName start

    Where ServiceName corresponds to one of the services in /etc/init.d.

Note: If the VMware VirtualCenter Server service does not start, verify that it is able to get exclusive access to the vCenter Server repository. For more information, see Unable to get an exclusive access to the vCenter Server repository (1006369).

Restarting vCenter Server Appliance services

To restart a vCenter Server Appliance service:

  1. Connect to the vCenter Server Appliance through SSH. For more information, see Enable or Disable SSH Administrator Login on the VMware vCenter Server Appliance section in the vCenter Server and Host Management Guide.
  2. Log in as root.
  3. Run this command to review the list of all services:

    ls /etc/init.d

  4. Scroll through the output and find the name of the service you are interested in. For a list of core vCenter Server services, see theAdditional Information section of this article.
  5. Run this command in the SSH client:

    service ServiceName restart

    Where ServiceName corresponds to one of the services in /etc/init.d.

Note: When you restart the VMware VirtualCenter Server service, all connected users are disconnected from their vSphere/Virtual Infrastructure Client sessions.

vCenter Server Appliance Service Description and Service Name correlation list
Service Description Service Name
VMware VirtualCenter Server Service vmware-vpxd
Embedded vPostgres Database service vmware-vpostgres
VMware Tools Service vmware-tools-services
VMware vSphere Profile-Driven Storage Service vmware-sps
VMware vCenter Inventory Service vmware-inventoryservice
VMware vSphere Network Dump Collector service vmware-netdumper
VMware vSphere Log Browser vmware-logbrowser
VMware vSphere Auto Deploy service vmware-rbd-watchdog
VMware vSphere Web Client vsphere-client
[For vSphere 5.1 Only] vCenter Single Sign-On 5.1 vmware-sso
[For vSphere 5.5 Only] vCenter Single Sign-On 5.5 –
VMware Directory Service		 vmdird
[For vSphere 5.5 Only] vCenter Single Sign-On 5.5 –
VMware Certificate Service		 vmcad
[For vSphere 5.5 Only] vCenter Single Sign-On 5.5 –
VMware Secure Token Service		 vmware-stsd
[For vSphere 5.5 Only] vCenter Single Sign-On 5.5 –
VMware Kdc Service		 vmkdcd
[For vSphere 5.5 Only] vCenter Single Sign-On 5.5 –
VMware Identity Management Service		 vmware-sts-idmd

Resolution :-Host cannot communicate with all other nodes in virtual SAN enabled cluster

This message normally appears on all hosts in a Virtual SAN cluster while one or more nodes are down for maintenance and therefore are not communicating with the rest of cluster, or when there is a legitimate problem with VSAN cluster communications. Under normal circumstances, this message is automatically cleared once the hosts have resumed communication. If this message appears in the Summary tab while all other indicators report that the VSAN network is healthy, this issue may be cosmetic.

VMware is aware of this cosmetic issue. It is due to a bug identified by VMware in vSphere 6.0 Update 1b that results in this message not being cleared automatically following maintenance or service restoration.

Resolution

This cosmetic issue is resolved in VMware ESXi 6.0 Update 2, available at VMware Downloads.

You can work around this cosmetic issue without upgrading, by clearing the message, using one of these methods:
  • Restart the VPXA management agent on VSAN nodes

    Restarting the VPXA (vCenter Server) management agent on the VSAN cluster ESXi host nodes refreshes the information in vCenter Server and clears the message.

  • Force an update to cluster membership

    Removing and then re-adding a host to the VSAN cluster forces all hosts to update the information about the cluster membership, which clears the message.

Note: These workarounds successfully clear the message, but the effect does not persist. You may have to apply the workaround again if the message reappears. This message will reappear in the event of a genuine network problem, host failure/reboot or host maintenance.
Restarting the VPXA management agent on VSAN nodes
To restart the VPXA (vCenter Server) management agent on the VSAN cluster ESXi host nodes:
  1. Enable SSH or the ESXi Shell on each host in the VSAN cluster. For information about enabling SSH or the ESXi Shell, see Using ESXi Shell in ESXi 5.x and 6.0 (2004746).
  2. Log in to each host in the VSAN cluster using SSH or the ESXi Shell.
  3. Restart the VPXA management agent using this command:

    # /etc/init.d/vpxa restart

    Note: Wait approximately one minute before proceeding to the next host. This allow information from each host to update in vCenter Server one-at-a-time.

Unisphere for VMAX 8.x showing blank page

I have faced issue with regards to Unisphere i.e after entering the URL the browser shows blank page . Below were the various resolution steps for me :-

  • Allow the Unisphere URL in the browser as trusted site.
  • Make sure correct Java and Flash version is installed.
  • Make sure that ports of any other software like DCNM are not conflicting with Unisphere .
  • There was weird situation wherein if Unisphere was installed in D it didnt worked whereas after installation in C it worked .

 

ESXI host 5.5 takes long time to boot with RDM’s

ESXI host 5.5 takes long time to boot and stays at VMW_Vaaip_Cx or vmw_vaaip_symm loaded successfully

Issue :  ESXI 5.5 boot take long time and stays at ( Storage API like)  loaded successfully , this may take an hour or so to bring the ESXI 5.5 completely.

Cause : This will be happening on the environment which have Microsoft cluster vm and RDM disk is presented to that Microsoft cluster vm . Especially the RDM lun is not showing as Perennially reserved =true. You can check whether the lun is perennially reserved or not by running the below command

esxcli storage core device list 

Solution: Will explain how to fix this on ESXi 5.5 version , for rest of the version please check the vmware kb article number 1016106.

1.We need to know the LUN identifier ( naa.id ) for all the rdm disk and  this identifier varies for different storage vendor.
2. There are multiple ways to get the identifier , simple way is to click edit settings on the vm and click the rdm lun and on the right hand side click manage path , note down the lun number and identifier.
3. We can also get the identifier from Configuration—storage adapter and click one of the hba .
4.ssh into the esxi server
5.Run the command for each RDM lun

esxcli storage core device setconfig -d naa.id –perennially-reserved=true

Vcenter 5.5U1 to 6.01 Certificate Mismatch Errors

Today I was upgrading Vcenter 5.5 to 6.01 and face the below error during pre upgrade check .

The system name in the vCenter Server 5.5 SSL certificate and the vCenter Single Sign-On 5.5 SSL certificates are not compatible. Please replace either the vCenter Server SSL certificates or the vCenter Single Sign-On SSL certificates so both vCenter Server and vCenter Single Sign-On SSL certificates use the same system name.

The resolution that I followed was as below

Re-installing the vCenter Server stack

  1. Completely uninstall in this order:vCenter Server
    vCenter Inventory Service
    vSphere Web Client
    vCenter Single Sign-On
  2. Rename the following directories underC:\ProgramData\VMware to ensure new certificates are generated using the correct hostname.
    1. CISto old
    2. Infrastructureto old
    3. SSLto old
    4. VMware VirtualCenterto VMware VirtualCenter.old
    5. vSphere Web Clientto vSphere Web Client.old
  3. Re-install vCenter Single Sign-On, vSphere Web Client, vCenter Inventory Service, and vCenter Service, and  ensure to reconnect to existing database.
  4. Reconnect all of the hosts to the vCenter Server.
  • After completing, attempt to upgrade the vCenter Server 5.x to 6.0.