Just fixed an issue related to configuration of Logback recently. You may be wondering why the Logback is used given that it’s much less known than Log4j and Java Logging. Very good question. This page from Logback may provide you some insights. I haven’t tested the performance, but it’s said to be 10 times faster than others. There is also an independent version of comparison on StackOverflow. After browsing it, I didn’t have an impression that I have to use one over the other.
In one of my recent projects, I got into a “big data” issue. One of the open source components emits so many logs that it quickly fills a hard disk. After isolating problem, I found huge number of log entries by the “find” command in a single log file whose size exceeds 50G – too big data for most system to handle.
The following is an example log entry in the log file:
If you want to find out the information in the vCenter database, the VMware support bundle comes handy. For example, if you want to analyze the event history, task history, you can dig out these information from the support bundle. By default, the vCenter support bundle is collected as part of the VMware support bundle on standalone client, but not on the vSphere Web Client. So make sure to mark the check box if you want the vCenter info.
In my last article, I talked about how to use logrotate to manage logs. As everything else, there are some tricks that are only learned when using it. Here are a few tricks and tips I learned recently. Hope it can save you some time.
Stickiness of Logrotate Rules
Logging is an important for software development and operation. Over the time, the log files can grow fast to fill up the disk space. To avoid the problem, log files are rotated, compressed, and deprecated based on certain rules, for example, periodically, over certain size limit, and retention limit.
Most mordern logging frameworks can do log rotation and compression, but different applications may use different frameworks thus configure them differently. If you want to have a solution across different applications for consistent policies, the logrote (https://fedorahosted.org/logrotate/) is a good choice.
During software development, we often add lots of logs that help debug and trace the code. When the log files grow bigger, it gets harder to locate the right information of interest. Even we restart the application, the old log remains and new info appends the end of the file unless we delete log file. It’s OK to delete a log file but it’s better to keep it in case for information of previous runs. Here is a trick that I use to make it easier for me to find right log of interest, and it may help you as well.
vSphere Client and vSphere Web Client allow administrators to download system logs from different ESXi hosts with choices of predefined groups of information like System, Storage, Network, UserWorld, etc. Under each group, there could be multiple types. For example, under the UserWorld, there are HostAgent and ProcessInformation.
If you have a log file that you want to monitor the incremental changes, you can use the following simple code. Whatever new log entries written to the log will be quickly picked up and printed out to console. It does not interfere with the application that writes the log file. To test the code, you can use any text editor to append more entries to the end of log file (don’t forget to save it).
Logging is an important tool for system monitoring and troubleshooting. vCenter has comprehensive logs for itself and related solutions. We’ll introduce how to change the settings for these logs in vCenter appliance. One obvious use case is to increase the log levels for troubleshooting.
As usual, the vCenter configuration file resides in a subfolder in the /etc folder.
If you manage a vSphere infrastructure, you may want to collect logs for troubleshooting, debugging, etc. You can get these logs from vSphere Client manually. You can also use vSphere API to collect them automatically.
The related managed object type in vSphere API is the DiagnosticManager. It helps to access logs from either a vCenter server or ESX server. It has no property but three methods:
1. queryDescriptions() provides a list of diagnostic files for a given system. It takes in an optional parameter host for specifying the HostSystem to extract information from. When you connect to the ESX server directly, the parameter isn’t needed. In vSphere Java API, you just pass in a null. When you connect to the vCenter server and the parameter isn’t specified, the method assumes you’re looking for vCenter logs. The return of this method is an array of DiagnosticManagerLogDescriptor data objects. The data object includes six properties: creator, fileName, format, info, key, and mimeType.
Examining logs is an important way for debugging and troubleshooting a system. There are about ten log files in the ESX server for the hostd agent, which listens API calls, with the same naming pattern as hostd-?.log under the /var/log/vmware directory. The hostd-index file has the number of currently used log files.
The log entry has a similar format to that of VC server logs. Following is a quick sample:
[2008-06-21 07:24:40.769 ‘SOAP’ 64834480 trivia] Received soap request from : checkForUpdates
The log level can be configured in the /etc/vmware/vpxa.cfg file. Just look for a section like the following. The possible levels are the same as those of VC logs: none, error, warning,info, verbose, or trivia, in an order from less to more detailed messages.