How To Get vCenter Database Records from VMware Support Bundle

If you want to find out the information in the vCenter database, the VMware support bundle comes handy. For example, if you want to analyze the event history, task history, you can dig out these information from the support bundle. By default, the vCenter support bundle is collected as part of the VMware support bundle on standalone client, but not on the vSphere Web Client. So make sure to mark the check box if you want the vCenter info.

The vCenter support bundle can be generated by running the vc-support.sh command on the vCenter appliance. I am sure it should be able to do so on Windows version of vCenter.

Bothered by SLOW Web UI to manage vSphere? Want to manage ALL your VMware vCenters, AWS, Azure, Openstack, container behind a SINGLE pane of glass? Want to search, analyze, report, visualize VMs, hosts, networks, datastores, events as easily as Google the Web? Find out more about vSearch 3.0: the search engine for all your private and public clouds.

doublecloud-vc:~ # find / -name vc-support*
/usr/lib/vmware-vpx/vc-support.sh
/usr/sbin/vc-support.sh

Once the command is run, a support bundle with .zip format is generated at /var/log/vmware/vpx/ directory with timestamp in its file name, i.e. /var/log/vmware/vpx/vcsupport-2014-10-01.12705.zip. You can use scp to copy it to anywhere you want.

doublecloud-vc:~ # /usr/lib/vmware-vpx/vc-support.sh
...
File: /var/log/vmware/vpx/vcsupport-2014-10-01.12705.zip
NOTE: vcsupport-2014-10-01.12705.zip is greater than 10 MB.
Please do not attach this file when submitting an incident report.
Please contact VMware support for an ftp site.
To file a support incident, go to http://www.vmware.com/support/sr/sr_login.jsp
 
doublecloud-vc:/var/log/vmware/vpx # ll /var/log/vmware/vpx/vcsupport-2014-10-01.12705.zip
-rw------- 1 root root 381131541 Oct  1 23:23 /var/log/vmware/vpx/vcsupport-2014-10-01.12705.zip

A Close Look at the vc-support Script

Within the vc-support.sh script, there is a fuction for dumping data from database as follows. As you can see, it uses Java JDBC API to retrieve data. Because of this, there could be issue that I will discuss in the end.

349 dbdump(){
350 
351 #Adding common jars
352 export VPXD_LIB=/usr/lib/vmware-vpx
353 COMMON_JARS=
354 for JAR in $VPXD_LIB/common-jars/*; do
355    if [ -f "$JAR" ]; then
356       COMMON_JARS="$COMMON_JARS${COMMON_JARS:+:}$JAR"
357    fi
358 done
359 
360 export CLASSPATH=$CLASSPATH${CLASSPATH:+:}${JARDIR}/common.jar:${JARDIR}/ojdbc5.jar:${JARDIR}/commons-logging-1.1.1.jar:${JARDIR}/vimtool.jar:${JARDIR}/commons-codec-1.3.jar    :${JARDIR}/commons-cli-1.1.jar:${COMMON_JARS:+:}$COMMON_JARS
361 
362 
363 rm -rf "${DBDUMPDIR}"
364 mkdir -p "${DBDUMPDIR}"
365 
366 ${JAVA} com/vmware/vim/vimtool/VimTool vcsupport -d "${DBDUMPDIR}"  >>"${DBDUMPDIR}vcsupport.log" 2>&1
367 if [[ $? -ne 0 ]]; then
368    banner "Error collecting database diagnostics..."
369 fi
370 # check if the DB dump filled up the partition (at least 100MB free)
371 if ! df -P -B1M "${DBDUMPDIR}" | tail -n 1 | awk '$4 < 100 {exit 1}'; then
372    banner "Low on disk space after DB dump: ${DBDUMPDIR}. Deleting incomplete bundle '$ZIPFILE' and DB dump."
373    rm -rf "${DBDUMPDIR}"
374    rm -f "${ZIPFILE}"
375    abort
376 fi
377 }

There are a few constants in the script that are defined above or below.

444 if ! mkdir "$BUNDLENAME"; then
445    banner "Could not create temporary directory '$working_dir/$BUNDLENAME'."
446    exit
447 fi
448 
449 #Set this after bundle location has been created
450 DBDUMPDIR="${BUNDLENAME}/var/log/vmware/dbdump/"
451 mkdir -p ${DBDUMPDIR}
 
 
27 LOGDIR=/var/log/vmware/vpx
 28 VPXDLOGDIR=$LOGDIR
 29 BUNDLENAME=vcsupport-$(date -I).$$
 30 ZIPFILE="$BUNDLENAME.zip"
 31 VER=0.03
 32 JARDIR=/usr/lib/vmware-vpx/scripts
 33 JAVA=/usr/java/jre-vmware/bin/java
 34 SLAPCAT=/usr/sbin/slapcat
 35 XMLCFG="/usr/lib/vmware-vpx/py/xmlcfg.py"
 36 VPXD_CONFIG_FILE="/etc/vmware-vpx/vpxd.cfg"
 37 MAXLOGFILES_XPATH="/config/log/maxFileNum"
 38 LOGCREATEFILTER=0
 39 FILE_TIME_FILTER=""
 40 BUG_NUMBER=

What are Dumped from vCenter Database

After unzipping the support bundle, a new directory will be created. Under that directory, you will find var directory for copied content from original vCenter server. There are many logs, but since we are only interested in the database dump, we can jump right to the dbdump directory as follows:

root@steve-dev:/var/tmp/vc-sb/vcsupport-2014-10-01.12705/var/log/vmware/dbdump# ls
DB_PRODUCT_INFO.xml  vcsupport.log      VPX_COMPUTE_RESOURCE.xml  VPX_DISABLED_METHODS.xml  VPX_EXT_SERVER.xml  VPX_HIST_STAT2.xml  VPX_HOST.xml           VPX_STAT_DEF.xml  VPX_USAGE_STAT.xml
DB_STAT_INFO.xml     VMwareVCMSDS.ldif  VPX_DATACENTER.xml        VPX_ENTITY.xml            VPX_EXT.xml         VPX_HIST_STAT3.xml  VPX_RESOURCE_POOL.xml  VPX_TASK.xml      VPX_VM.xml
DB_TABLE_SIZE.xml    VPX_ACCESS.xml     VPX_DEVICE.xml            VPX_EVENT.xml             VPX_HIST_STAT1.xml  VPX_HIST_STAT4.xml  VPX_STAT_COUNTER.xml   VPX_UPTIME.xml

Now, let’s take a look at the event history in the VPX_EVENT.xml file:

root@sjin-dev:/var/tmp/vc-sb/vcsupport-2014-10-01.12705/var/log/vmware/dbdump# vi VPX_EVENT.xml
<VPX_EVENT.xml xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" ><Row>
<event_id>6925</event_id><chain_id>6925</chain_id><event_type>vim.event.BadUsernameSessionEvent</event_type><extended_class>0</extended_class><create_time>2014-10-01 14:28:16.27</create_time><username>storvisor</username><category>error</category></vm_id></vm_name><host_id>39</host_id><host_name>sysmgmt004-esx-1a.eng.storvisor.com</host_name><computeresource_id>36</computeresource_id><computeresource_type>3</computeresource_type><computeresource_name>mnode-cluster</computeresource_name><datacenter_id>31</datacenter_id><datacenter_name>mnode-dc</datacenter_name></datastore_id></datastore_name></network_id></network_name></network_type></dvs_id></dvs_name></storagepod_id></storagepod_name></change_tag_id>
</Row>
...
</Row>
<Row>
<event_id>1</event_id><chain_id>1</chain_id><event_type>vim.event.ServerStartedSessionEvent</event_type><extended_class>0</extended_class><create_time>2014-07-29 10:24:26.035</create_time></username><category>info</category></vm_id></vm_name></host_id></host_name></computeresource_id></computeresource_type></computeresource_name></datacenter_id></datacenter_name></datastore_id></datastore_name></network_id></network_name></network_type></dvs_id></dvs_name></storagepod_id></storagepod_name></change_tag_id>
</Row>
</VPX_EVENT.xml >

Troubleshooting

Software can fail anywhere not just the product itself – the vc-support script for troubleshooting can fail too. To figure out what is happening, just get the log for the support bundle generation. The following is a successful sample which does show what are dumped and how much time it took for each file.

root@steve-dev:/var/tmp/vc-sb/vcsupport-2014-10-01.12705/var/log/vmware/dbdump$ vi vcsupport.log
Picked up JAVA_TOOL_OPTIONS: -Xms16M -Xmx128M
[2014-10-01 23:19:42 INFO] Invoking vcsupport "-d" "vcsupport-2014-10-01.12705/var/log/vmware/dbdump/"
[2014-10-01 23:19:42 INFO] Starting Virtual Center Database Support
[2014-10-01 23:19:42 INFO] Config name=vcdb
[2014-10-01 23:19:42 INFO] Property file=/etc/vmware-vpx/vcdb.properties
[2014-10-01 23:19:42 INFO] Loaded url from props=jdbc:postgresql://127.0.0.1:5432/VCDB
[2014-10-01 23:19:42 INFO] Overrides=
[2014-10-01 23:19:42 INFO] Dump of VPX_ACCESS took :204 ms
[2014-10-01 23:19:43 INFO] Dump of VPX_HOST took :276 ms
[2014-10-01 23:19:43 INFO] Dump of VPX_VM took :11 ms
[2014-10-01 23:19:43 INFO] Dump of VPX_RESOURCE_POOL took :9 ms
[2014-10-01 23:19:43 INFO] Dump of VPX_COMPUTE_RESOURCE took :19 ms
[2014-10-01 23:19:43 INFO] Dump of VPX_ENTITY took :14 ms
[2014-10-01 23:19:43 INFO] Dump of VPX_DISABLED_METHODS took :31 ms
[2014-10-01 23:19:43 INFO] Dump of VPX_EXT took :35 ms
[2014-10-01 23:19:43 INFO] Dump of VPX_EXT_SERVER took :8 ms
[2014-10-01 23:19:43 INFO] Dump of VPX_DATACENTER took :6 ms
[2014-10-01 23:19:44 INFO] Dump of VPX_EVENT took :847 ms
[2014-10-01 23:19:44 INFO] Dump of VPX_TASK took :524 ms
[2014-10-01 23:19:44 INFO] Dump of VPX_STAT_DEF took :14 ms
[2014-10-01 23:19:44 INFO] Dump of VPX_STAT_COUNTER took :7 ms
[2014-10-01 23:19:44 INFO] Dump of VPX_DEVICE took :6 ms
[2014-10-01 23:19:44 INFO] Dump of VPX_HIST_STAT1 took :382 ms
[2014-10-01 23:19:45 INFO] Dump of VPX_HIST_STAT2 took :174 ms
[2014-10-01 23:19:45 INFO] Dump of VPX_HIST_STAT3 took :119 ms
[2014-10-01 23:19:45 INFO] Dump of VPX_HIST_STAT4 took :136 ms
[2014-10-01 23:19:45 INFO] Dump of VPX_USAGE_STAT took :8 ms
[2014-10-01 23:19:45 INFO] Dump of VPX_UPTIME took :30 ms
[2014-10-01 23:19:45 INFO] Dump of DB_PRODUCT_INFO took :6 ms
[2014-10-01 23:19:45 INFO] Dump of DB_TABLE_SIZE took :20 ms
[2014-10-01 23:19:45 INFO] Dump of DB_STAT_INFO took :90 ms

In one case, I saw the VPX_EVENT.xml is zero byte, and the VPX_TASK.xml is totally missing. The log revealed that there was OutOfMemoryError from the Java code. It could be that the vCenter itself ran out of memory, or maybe the -Xmx128M is too low for working with a product vCenter server. It shouldn’t be the second case if the Java code is well designed. But if that is true, the JAVA_TOOL_OPTIONS should be updated before running the script.

doublecloud-vc:~ # echo $JAVA_TOOL_OPTIONS
-Xms16M -Xmx128M
doublecloud-vc:~ # export JAVA_TOOL_OPTIONS="-Xms16M -Xmx512M"
doublecloud-vc:~ # echo $JAVA_TOOL_OPTIONS
-Xms16M -Xmx512M
This entry was posted in Virtualization and tagged , , , . Bookmark the permalink. Post a comment or leave a trackback: Trackback URL.

Post a Comment

Your email is never published nor shared. Required fields are marked *

*
*

You may use these HTML tags and attributes <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

  • NEED HELP?


    My company has created products like vSearch ("Super vCenter"), vijavaNG APIs, EAM APIs, ICE tool. We also help clients with virtualization and cloud computing on customized development, training. Should you, or someone you know, need these products and services, please feel free to contact me: steve __AT__ doublecloud.org.

    Me: Steve Jin, VMware vExpert who authored the VMware VI and vSphere SDK by Prentice Hall, and created the de factor open source vSphere Java API while working at VMware engineering. Companies like Cisco, EMC, NetApp, HP, Dell, VMware, are among the users of the API and other tools I developed for their products, internal IT orchestration, and test automation.