Harmful Big Data

In one of my recent projects, I got into a “big data” issue. One of the open source components emits so many logs that it quickly fills a hard disk. After isolating problem, I found huge number of log entries by the “find” command in a single log file whose size exceeds 50G – too big data for most system to handle.

The following is an example log entry in the log file:

fbbd6d5d0d75a6f28224363fbcca4d11000

Posted in Applications & Tools | Tagged , | Leave a comment

How To Get vCenter Database Records from VMware Support Bundle

If you want to find out the information in the vCenter database, the VMware support bundle comes handy. For example, if you want to analyze the event history, task history, you can dig out these information from the support bundle. By default, the vCenter support bundle is collected as part of the VMware support bundle on standalone client, but not on the vSphere Web Client. So make sure to mark the check box if you want the vCenter info.

Posted in Virtualization | Tagged , , , | Leave a comment

vSphere Web Client URL Patterns

Although I don’t think vSphere Web Client is a good choice as main GUI for vSphere administration, it has done one thing right and well – it has a unique URL for almost everything you can browse the vSphere Web Client. The benefit is that you can bookmark a URL and open to the page without many clicks.

Posted in Virtualization | Tagged , , | Leave a comment

Key Takeawys from VMworld 2014

Last week was a pretty busy week with VMworld 2014 in San Franciso, followed by a long weekend during which some of us may be even busier. If you haven’t got chance to the conference, you can check out the recorded VMworld keynotes, and maybe later recorded break-out sessions.

While there were many announcements/news in the conference from VMware and other vendors. I think the following three are the most important ones.

VMware EVO

Posted in News & Events | Tagged | Leave a comment

VMworld 2014

Today is the first day of VMworld 2014 San Francisco. For the last 7 consecutive years, I have attended all the VMworld conferences in US, either as a speaker, booth duty staff, or normal attendee. This year is no exception. As always, I find the solution exchange is the best part of the conference, so I spent several hours in the reception party this afternoon. While enjoying the good food, I talked to quite a few vendors. When the party was over, I found I only finished two rows. So there are a lot more work tomorrow.

Posted in News & Events | Tagged , | Leave a comment

Tips and Tricks in Using Logrotate

In my last article, I talked about how to use logrotate to manage logs. As everything else, there are some tricks that are only learned when using it. Here are a few tricks and tips I learned recently. Hope it can save you some time.

Stickiness of Logrotate Rules

Posted in Applications & Tools | Tagged , , | Leave a comment

How to Use Logrotate for Managing Log Files

Logging is an important for software development and operation. Over the time, the log files can grow fast to fill up the disk space. To avoid the problem, log files are rotated, compressed, and deprecated based on certain rules, for example, periodically, over certain size limit, and retention limit.

Most mordern logging frameworks can do log rotation and compression, but different applications may use different frameworks thus configure them differently. If you want to have a solution across different applications for consistent policies, the logrote (https://fedorahosted.org/logrotate/) is a good choice.

Posted in Applications & Tools, Software Development | Tagged , , | 1 Response

How to Create New User in ESXi With Shell Script

It’s pretty easy to create a new user using vSphere APIs in Java. If you want one or two lines of scripts, you can write a few line Python script using PyVimomi wrapped by a command line. Even easier is a solution discussed at PureVirtual.

The trick is really about how to access the adduser command, which is available but not accessible from console as it is. To use the command, you have to type the following command from ESXi console: (I assume there are a few more commands that can be used in the same way)

Posted in Virtualization | Tagged , , | Leave a comment

How to Hack vCenter Database in vCSA Appliance

As @lamw documented in his blog, you can access the database in the vCenter software appliance (vCSA). In the first appliance of version 5.0, VMware included IBM DB2 Embedded and then switched to vPostgres right after. That’s because vPostgres is VMware’s own product based on the open source Postgres.

By default, the access to the vPostgress database is limited to local applications. In other words, if you want to access the database remotely, it will not work. After researching a little, I figured out how to configure the databse for remote access.

Posted in Virtualization | Tagged , , , , , | Leave a comment

How to Expire ESXi License immediately

VMware has evaluation license for ESXi servers. After 60 days, it expires and you have to apply a paid or free license to continue. Technically, there is a trick to reset the evaluation key by deleting two files (/etc/vmware/vmware.lic and /etc/vmware/license.cfg) and rebooting the server. It’s of course not complying with VMwrae license terms. Under some circumstances like training lab, it may be OK. Make sure to consult VMware on this if you are not working for VMware. But wait – if you are working for VMware, do you need evaluation license? In his reply to my tweet, Duncan mentioned he never saw license expiration.

Posted in Virtualization | Tagged , , , | Leave a comment

How to Manually Change ESXi Time

Although VMware ESXi supports common Linux commands, its implementation is based on busybox. Some of the commands are not supported, or functionalities are reduced. Setting time is one of the cases.

To find out the current time, you issue the following command:

~ # date
Sun Jul 13 10:50:59 UTC 2014

Now if you want to use the same date command to change the time, it’s a different story. In fact, the help of the date command works as normal, but when you really type in full command, you’ll see it won’t work.

Posted in Virtualization | Tagged , , , | Leave a comment

Parsing, Modifying, and Generating Java Source Code With Javaparser

It’s not common to do something with Java source code programmatically, but I find the javaparser project is very handy if such a use case comes up. It’s a Java 1.5 parser with AST generation and visitor support. With the API, you can access various elements in the source code like class, methods, statements, javadoc, comments, etc.

Posted in Software Development | Tagged , , , | 1 Response

How To Monitor Log File in Real Time and Save New Log To Another File

During software development, we often add lots of logs that help debug and trace the code. When the log files grow bigger, it gets harder to locate the right information of interest. Even we restart the application, the old log remains and new info appends the end of the file unless we delete log file. It’s OK to delete a log file but it’s better to keep it in case for information of previous runs. Here is a trick that I use to make it easier for me to find right log of interest, and it may help you as well.

Posted in Applications & Tools | Tagged , , | Leave a comment

Which Process Holds My File? Tools on Linux and Windows

Once in a while, I got into issues that my build failed because the IDE could not clean certain folder or file. The root cause was that they were opened by another application/process. If the application is known, just closed it and the build worked well. Some times I had no clue which application held the file. To find out quickly without guessing, right tools are needed.

On Linux, it’s quite easy with a command called lsof as follows:

Posted in Applications & Tools | Tagged , | Leave a comment

A Hidden Gem VMware Should Invest More For Its Long Term

For a long time, VMware wanted to grow upward the stack but did not work out well. Instead, it took the horizontal strategy as I discussed in my blog article two years ago. So far, the new strategy has worked well.

Posted in Uncategorized | 4 Responses

Cisco UCS Director: Key Concepts Illustrated In Big Pictures

During the last few months, I worked a lot with Cisco UCS Director on daily basis. As I wrote before, UCS Director is a powerful platform for you to manage and orchestrate infrastructures from VMware, Hyper-V, KVM, to the public clouds like Amazon, Azure.

Just like any other management platform, it abstracts the underlying infrastructure and operations using its own concepts and workflows. By exploring its Flex GUI, one can gradually get familiar with these concepts. It takes time to master a product, and no exception for UCS Director. Understanding key concepts and their relationships can help speed up the process significantly.

Posted in Cloud Computing | Tagged , , , | 2 Responses

Java 8 New Features: JavaFX

The JavaFX is not really a new feature, but it’s the first time for it to make into a major Java release. JavaFX has a pretty long history as the next big thing for building cross platform GUI applications. Initially it has its own scripting language and hasn’t made its way in the Java community. The reason is simple: although the JavaFX has richer features on graphics, it does not seem worth learning a new language for it. For a long time, I was wonder why Sun didn’t use the investment to improve its Swing library which has much bigger audience than JavaFX.

Posted in Software Development | Tagged , , | Leave a comment

Java 8 New Features: Nashorn JavaScript Engine

When JavaScript was created, it had not much to do with Java. It’s named as such maybe due to the popularity of Java language at that time. But over the years, JavaScript has gained its own popularity and dominated the client side of Web applications. There are even some work for JavaScript to extend its reach to the server side, for example, Node.js.

Posted in Software Development | Tagged , , , | Leave a comment

Accessing Database in Java with JDO

I haven’t written Java code to access relational database for a while. Over the years there have been lots of progresses, especially with various Object Relational Mapping (ORM) frameworks. With these frameworks, accessing relational databases becomes pretty easy. In some cases, you don’t have to know SQL at all. Of course understanding SQL is always an advantage.

Posted in Software Development | Tagged , , , , | Leave a comment

Java 8 New Features: Map Reduce Made Easy With Stream APIs

In my article, I introduced the new Stream API. With the new stream APIs, you can apply many different operations on the stream, including the map-reduce functions.

One of the most famous framework to support map-reduce for large scale data processing, a.k.a. BigData, is Hadoop as I introduced almost two years ago here. Data processing wise, the Java 8 stream API can do pretty much the same. Here is a quick sample that shows how it count number of words in string. There are significant differences in how they are implemented and the cases in which they should be used. Let’s discuss them after the sample.

Posted in Big Data, Software Development | Tagged , , , | Leave a comment
  • NEED HELP?


    My consulting helps clients with virtualization and cloud computing, including VMware infrastructure automation and orchestration, vSphere management APIs, and deep product integration with hypervisors. Current training offerings include vSphere APIs training, vCenter Orchestrator training, and etc. Should you, or someone you know, need these consulting services or training, please feel free to contact me: steve __AT__ doublecloud.org.

    Me: Steve Jin, VMware vExpert who authored the VMware VI and vSphere SDK by Prentice Hall, and created the de factor open source vSphere Java API while working at VMware engineering. Companies like Cisco, EMC, NetApp, HP, Dell, VMware, are among the users of the API and other tools I developed for their products, internal IT orchestration, and test automation.