Home > Virtualization > vSphere Support Bundle: How to Collect Anything You Want From ESXi, Not Just Logs

vSphere Support Bundle: How to Collect Anything You Want From ESXi, Not Just Logs

November 25th, 2013 Leave a comment Go to comments

vSphere Client and vSphere Web Client allow administrators to download system logs from different ESXi hosts with choices of predefined groups of information like System, Storage, Network, UserWorld, etc. Under each group, there could be multiple types. For example, under the UserWorld, there are HostAgent and ProcessInformation.

While most of the predefined types are good enough, you can define your own groups and types with the existing mechanism. In other words, you can have your own groups that can show up in the “Export System Logs” dialog box. This is especially useful if you have own software running on ESXi. Say, if you have a new driver installed via VIB, you should leverage it to collect your logs and other information. Here is also the link to VMware document “Customizing vm-support in vSphere 5.0”.

Lost VMs or Containers? Too Many Consoles? Too Slow GUI? Time to learn how to "Google" and manage your VMware and clouds in a fast and secure HTML5 App.

The following are something more than what is not quite covered out there, including some advanced techniques.

Support Manifests

To have your category and type, it’s pretty simple. Just design your own manifest which is a text file that go to the /etc/vmware/vm-support directory. ESXi would include your manifest when inquired by vCenter. If you are interested in knowing what groups and types of VMware support bundles in an ESXi server, you can type in the following URL in a browser. You would need valid username and password to the ESXi server. Don’t forget the ?listmanifests=1 or you would download a full support bundle as vm-support.tgz which could be huge in size.

The response is an XML as shown below – enough information to construct the dialog in vSphere GUI (both vSphere Client and vSphere Web Client).

<manifest name="Base" id="FaultTolerance:Base" group="FaultTolerance" defaultSelected="true" vmOnly="false"/>
<manifest name="Base" id="System:Base" group="System" defaultSelected="true" vmOnly="false"/>
<manifest name="Coredump_VM" id="HungVM:Coredump_VM" group="HungVM" defaultSelected="false" vmOnly="true"/>
<manifest name="EsxImage" id="System:EsxImage" group="System" defaultSelected="true" vmOnly="false"/>
<manifest name="FCoE" id="Storage:FCoE" group="Storage" defaultSelected="true" vmOnly="false"/>
<manifest name="GPUResource" id="System:GPUResource" group="System" defaultSelected="true" vmOnly="false"/>
<manifest name="HostAgent" id="Userworld:HostAgent" group="Userworld" defaultSelected="true" vmOnly="false"/>
<manifest name="Modules" id="System:Modules" group="System" defaultSelected="true" vmOnly="false"/>
<manifest name="Multipathing" id="Storage:Multipathing" group="Storage" defaultSelected="true" vmOnly="false"/>
<manifest name="NAS" id="Storage:NAS" group="Storage" defaultSelected="true" vmOnly="false"/>
<manifest name="ProcessInformation" id="Userworld:ProcessInformation" group="Userworld" defaultSelected="true" vmOnly="false"/>
<manifest name="ResourceGroups" id="System:ResourceGroups" group="System" defaultSelected="true" vmOnly="false"/>
<manifest name="Send_NMI_To_Guest" id="HungVM:Send_NMI_To_Guest" group="HungVM" defaultSelected="false" vmOnly="true"/>
<manifest name="SnapshotsAndMemoryImages" id="VirtualMachines:SnapshotsAndMemoryImages" group="VirtualMachines" defaultSelected="false" vmOnly="false"/>
<manifest name="Suspend_VM" id="HungVM:Suspend_VM" group="HungVM" defaultSelected="false" vmOnly="true"/>
<manifest name="System" id="Configuration:System" group="Configuration" defaultSelected="true" vmOnly="false"/>
<manifest name="System" id="Logs:System" group="Logs" defaultSelected="true" vmOnly="false"/>
<manifest name="SystemImageCacheHostProfile" id="hostProfiles:SystemImageCacheHostProfile" group="hostProfiles" defaultSelected="true" vmOnly="false"/>
<manifest name="VMKernel" id="System:VMKernel" group="System" defaultSelected="true" vmOnly="false"/>
<manifest name="VMKernel_Cores" id="System:VMKernel_Cores" group="System" defaultSelected="true" vmOnly="false"/>
<manifest name="VMKernel_Old_Cores" id="System:VMKernel_Old_Cores" group="System" defaultSelected="false" vmOnly="false"/>
<manifest name="VirtualMachineStats" id="VirtualMachines:VirtualMachineStats" group="VirtualMachines" defaultSelected="true" vmOnly="false"/>
<manifest name="base" id="ActiveDirectory:base" group="ActiveDirectory" defaultSelected="true" vmOnly="false"/>
<manifest name="base" id="CIM:base" group="CIM" defaultSelected="true" vmOnly="false"/>
<manifest name="base" id="FileSystem:base" group="FileSystem" defaultSelected="true" vmOnly="false"/>
<manifest name="base" id="Hardware:base" group="Hardware" defaultSelected="true" vmOnly="false"/>
<manifest name="base" id="Installer:base" group="Installer" defaultSelected="true" vmOnly="false"/>
<manifest name="base" id="Network:base" group="Network" defaultSelected="true" vmOnly="false"/>
<manifest name="base" id="Storage:base" group="Storage" defaultSelected="true" vmOnly="false"/>
<manifest name="base" id="VirtualMachines:base" group="VirtualMachines" defaultSelected="true" vmOnly="false"/>
<manifest name="corefiles" id="VirtualMachines:corefiles" group="VirtualMachines" defaultSelected="true" vmOnly="false"/>
<manifest name="diskinfo" id="VirtualMachines:diskinfo" group="VirtualMachines" defaultSelected="true" vmOnly="false"/>
<manifest name="dvs" id="Network:dvs" group="Network" defaultSelected="true" vmOnly="false"/>
<manifest name="hostProfiles" id="hostProfiles:hostProfiles" group="hostProfiles" defaultSelected="true" vmOnly="false"/>
<manifest name="iodm" id="Storage:iodm" group="Storage" defaultSelected="true" vmOnly="false"/>
<manifest name="iscsi" id="Storage:iscsi" group="Storage" defaultSelected="true" vmOnly="false"/>
<manifest name="logs" id="VirtualMachines:logs" group="VirtualMachines" defaultSelected="true" vmOnly="false"/>
<manifest name="md5sums" id="IntegrityChecks:md5sums" group="IntegrityChecks" defaultSelected="true" vmOnly="false"/>
<manifest name="tcpip" id="Network:tcpip" group="Network" defaultSelected="true" vmOnly="false"/>
<manifest name="test-esx-logs" id="Testing:test-esx-logs" group="Testing" defaultSelected="false" vmOnly="false"/>
<manifest name="usb" id="Hardware:usb" group="Hardware" defaultSelected="true" vmOnly="false"/>
<manifest name="userworld" id="CrashDumps:userworld" group="CrashDumps" defaultSelected="true" vmOnly="false"/>
<manifest name="vmfsheader" id="FileSystem:vmfsheader" group="FileSystem" defaultSelected="false" vmOnly="false"/>
<manifest name="vsi" id="PerformanceSnapshot:vsi" group="PerformanceSnapshot" defaultSelected="false" vmOnly="false"/>
<manifest name="vxlan" id="Network:vxlan" group="Network" defaultSelected="true" vmOnly="false"/>

How To Write Your Support Manifest

Coming with VMware ESXi at the /etc/vmware/vm-support directory is a README file, which is a very good and concise tutorial. To save my time on advance topics in the last part, I just include it as it is in the following. Remember that you can not only copy log files but also all sorts of information, even the output from running shell commands which opens up a great flexibility for potentially anything.

/etc/vmware/vm-support # vi README
README for vm-support plugins
New manifests should be added via a .vib.  Manifests manually added via cp
will not persist across reboots.
Manifest file format:
   # Manifest name: <foo>
   # Manifest group: <bar>
   # Manifest default: Enabled/Disabled
   # action Options file/command
   copy   -               /etc/vmware/*
   run    -               vmware -v
   run    TIMEOUT=10      /sbin/esxcfg-info -a
Manifest name:
   Friendly, user visible name for this manifest.  Displayed in ViClient and
   via the cmdline vm-support. Must be a-zA-Z only, spaces and punctuation
   are not supported.
Manifest group:
- README [Readonly] 1/95 1%
      Adds the specified files into the bundle. Supports only the '*' wildcard
      (not full regular expressions).  It will not recursively descend
      directories unless RECURSE option is specified.
      like copy, but internally uses 'cat <filename>'. This prevents tar from
      erroring out on files (like proc nodes) that incorrectly report length
      info.  Timestamps will not be preserved on these files.
      Runs the specified command and puts the output into the bundle. Output
      must be to stdout. Both stdout and stderr are redirected to the same file.
      Adds the specified file from the VM directory of all VMs into the bundle.
      Supports the same wildcards as the 'copy' command.  (ie *.log would add
      all vmware.log files from a vm to the bundle.)
      Runs the specified command against all registered vms, substituting any
      instances of '{VMDIR}' and '{VMCFG}' for the paths to the vm's directory
      and config file.
      Runs the specified command against all volumes under /vmfs/volumes. Any
      instances of '{VMFS}' will be replaced with the full path to the vmfs
      Ensures the specified file doesn't end up in the final vm-support.tgz.
      The prune command is global across all enabled manifests.  Wildcards
      are not supported.
Manifest Command Options
   TIMEOUT=<timeout in seconds>
      This command will be terminated if there is not at least some output
      generated at least every TIMEOUT seconds.
      Do not print an error if command exits with non-zero error code.
      Do not print an error if file missing.
   MAXSIZE=<size in bytes>
      Gather at most <size in bytes> of the specified file/command.  The file/cmd output
      will be truncated in vm-support.
      recursively copy directories.
   RELOCATE=<new path in tar file>
      Put the output of this command at the specified location in the tar
      causes the command to be automatically run a 2nd time with the
      '--formatter=json" parameter added.  Output will be directed to th 'json'
      subdirectory of the tarball.  In the future, plain text output for
      these commands may be removed.  This command will work with all
      'localcli' commands.

Advanced Topics

If you want to download all the support bundles directly from an ESX server, you can use the following URL: (just use as an example, replace it with your ESXi IP address or host name.)

Note that it could be very big. Depending on what you system, it could easily be hundreds of MBs. If you just one type of support bundle, you can issue a URL as follows. Check out the id in the above the XML file. Be careful while typing – it’s case sensitive

What if you want more than one type of support bundles, you can add them with spaces in between as follows:

By using the URL as these, you can avoid the high IO on the vCenter. You can automate it with wget command. The only problem is that you have to provide credentials of each individual ESXi hosts.

Another advanced topic is the streaming capability of support bundles. The above URL download is streamed without saving to a file, which is the case with the vm-support command in ESXi shell. You can issue # vm-support –help for details.

The streaming capability can be extremely useful if you have big external data to be imported and included in your support bundle because the space of ESXi itself is quite limited. Depending on the case, the implementation can be very complicated. Please feel free to contact me if you need help on this.

Categories: Virtualization Tags: , , ,