How You Can Use vSphere APIs to Collect vCenter and ESX Logs

October 20th, 2010 3 comments

If you manage a vSphere infrastructure, you may want to collect logs for troubleshooting, debugging, etc. You can get these logs from vSphere Client manually. You can also use vSphere API to collect them automatically.

The related managed object type in vSphere API is the DiagnosticManager. It helps to access logs from either a vCenter server or ESX server. It has no property but three methods:

1. queryDescriptions() provides a list of diagnostic files for a given system. It takes in an optional parameter host for specifying the HostSystem to extract information from. When you connect to the ESX server directly, the parameter isn’t needed. In vSphere Java API, you just pass in a null. When you connect to the vCenter server and the parameter isn’t specified, the method assumes you’re looking for vCenter logs. The return of this method is an array of DiagnosticManagerLogDescriptor data objects. The data object includes six properties: creator, fileName, format, info, key, and mimeType.

Categories: vSphere API Tags: , , , ,

Cloud Architecture Patterns: Overview

October 18th, 2010 No comments

Design patterns have been very popular among software developers since the book by Gang of Four (Enrich Gamma, Richard Helm, Ralph Johnson, John Vlissides) in 1995. If you go to an interview for a software engineering position today, the chances are you most likely get one or more questions on design patterns. 

Like many concepts in software that from other disciplines, the “pattern” idea was “borrowed” from “A Pattern Language,” a book by Christopher Alexander on architectural patterns published 20 years before we started to talk about software design pattern. His book turns out to be a great way to summarize and document reusable design elements for different engineering works.

As Christopher points out, “Each pattern describes a problem which occurs over and over again in our environment, and then described the core of the solution to that problem, in such a way you can use the solution a million times over, without ever doing it the same way twice.”

Today I “borrow” the same idea and apply the concept to cloud computing on architecture designs. The main focus is on virtual machine-based architecture patterns so that you can best leverage virtual machines for your cloud computing.

What is a Cloud Architecture Pattern?

An architectural pattern extracts the common and re-usable design concepts and components. If we put it in the big picture, it should be somewhere below the overall system architecture and above software design as the middle layer.

Really Simple Tricks to Speed up Your CLIs 10 Times Using vSphere Java API

October 15th, 2010 4 comments

I recently had a short discussion with my colleague on implementing CLIs with vSphere Java API. One problem is that if you have multiple commands to run, each of them connects to the server and authenticate over and over. You’d better remember to logout the connection each time after you are done, or leave many un-used connections on the server that could significantly slow down your ESX or vCenter server (read this blog for details).

You can have two solutions to this problem. The first one is to have your own “interpreter” command. After you type the command, it shows you prompt for more sub-commands. It’s very much like the “ftp” command in that sense. You can have subcommands like “login” or “open” or “connect” for connecting to a server, and other commands. The “interpreter” command can then hold the ServiceInstance object until it’s closed in the end.

You can save about 0.3 to 0.5 second on creating new HTTPS connection and login for each command after the first one. It’s not a big deal given that vSphere Java API has hugely reduced that time from 3 to 4 seconds with Apache AXIS. So if you switch to vSphere Java API, you get instant 10 time performance gain. Still, if you have many commands to run, it could be a decent saving.

With this solution, you can also implement batch mode in which you can save all your commands into a file and then execute them all with one command. You can find many examples like PowerShell which support interactive mode and batch mode.

Another solution is just having normal commands. The problem becomes how to avoid the authentication for each command after the first. Luckily we have something for you in the API.

Categories: vSphere API Tags: , ,

The Future of the Operating System in Cloud Computing

October 11th, 2010 No comments

An operating system (OS) is a piece of software. It manages the computer hardware and provides common services for various applications. With the rise of cloud computing, people may wonder whether the OS is still relevant and what role it will play in the future cloud.

Key Components of OS

There are different flavors of operating systems: from real-time OS, desktop OS, all the way to a mainframe OS. The most recent OS is the Cloud OS.

In general, every OS has these common components:

  • The kernel, which manages memory, processes, etc.
  • Device drivers, which drive different hardware from different vendors.
  • User interfaces, including command line shell and Window system.
  • File system, which provides a hierarchical way to persist data.
  • Security, which authenticates users and protects information.

Depending on the type of OS, you may miss something here or have something extra. For example, an embedded OS may not have a user interface and everything is controlled remotely. For the desktop OS, you may have extra commonly used applications such as a calculator, a calendar, a browser, and so on.

Squeezed Sandwich

vSphere Java API Exceeds 10,000 Downloads

October 7th, 2010 No comments

October 6, 2010, is a historical moment for VI Java API project – the total downloads exceeded 10,000. It’s two days earlier than I had expected. After yesterday’s blog on the NetApp and Brocade’s testimonials, the daily downloads suddenly doubled. When I found the stats approaching 10,000, I tweeted “vSphere Java API 9,999 downloads now. Who want to be No. 10,000?” I wish I could have been able to track who made the No. 10,000. :-)

Strictly speaking, the total had exceeded 10,000 a while back. Besides typical downloads, you can also directly sync up with the subversion. As I checked the number there, it had passed 1,000 reads early this year.

Thanks to you all, the vSphere Java API community!

10,000 downloads is not a big deal for an application especially when it’s for end users. It’s a big deal for an API, and even bigger for an enterprise API which requires vSphere environment which not every developer has access to.

Besides the download number, I would like to brag these numbers:

NetApp and Brocade Opt for Open Source vSphere Java API

October 6th, 2010 No comments

I am very pleased to welcome NetApp and Brocade to the vSphere(VI) Java API poweredby page. Many thanks to Patric Chang and Katie Colbert from Brocade, and George Costea and Eric Forgette from NetApp for making this happen.

NetApp and Brocade have been using open source vSphere(VI) Java API for quite some time and each has several products shipped with this open source API. As you may recall from my previous blog on VMworld 2010, I did not talk about NetApp and Brocade because I hadn’t got written permission even though they had great shows out there. Please feel free to check them out at VMworld in Copenhagen next week.

I think the key takeaway from this is that vSphere Java API has been stable enough to be used by companies like NetApp and Brocade that demand highest quality of products. For one thing, you can prabably afford not connecting to networks for a little while, but for sure cannot afford messing up your data storage. NetApp and Brocade’s confidence in this API is the best testimonial on the quality and readiness of the API. There are many other even bigger companies are using the API as well. I will talk more about them later.

A Workaround to Identify NFS based Datastore in vSphere

October 4th, 2010 2 comments

We just had the longest discussion in vSphere Java API forum regarding the “UUID of an NFS datastore.” The question is basically how to find the “UUID” via the vSphere API.

You can create datastores based on either VMFS or NFS. The VMFS can be backed up by local SCSI, or SAN (FC, iSCSI). It’s very easy to find UUID of a VMFS based datastore by calling getUuid() method from the corresponding data object VmfsDatstoreInfo.

For NFS based datastore, it’s a lot complicated. I am glad we digged to the bottom of the issue. Instead of going through the long discussion, I summarize the key takeways from the discussion.

Before jumping into details, let’s clarify one thing: NFS datastore does not have a UUID. (If you want to know more about UUID in vSphere, you should read this blog article.) You can check out the NasDatastoreInfo which does not have uuid property. It does, however, have an identifier like 73ca9790-6dbf88b0, which is not a UUID per se. We will call it simply an ID.

Motivation

You may be wondering why you should care about the ID. It is pretty important in that it’s used in performance stats like the following:

Two Important Tips Reading vSphere API Reference

September 29th, 2010 4 comments

Last week I answered a question in VMware Web Services SDK forum about asterisks in vSphere API reference. Underneath these asterisks comes a note saying “May not be present.” What does it really mean?

The asterisks normally show up after properties or sub-properties defined with a managed object. As it says, it’s possible that there is NO value to the property.

How can it be like this?

There are two major causes. First, it reflects the different implementations of ESX, ESXi and vCenter. As a quick example, you can find many of the properties in the “content” (type: ServiceContent) come with asterisks.

On a vCenter server, you will find values to almost all the properties, but not quite so for ESX/ESXi. But we have one API reference document, so it’s natural to mark whatever possible no value as “may not present.”

Secondly, it may be as such depending on the state of a managed object. For instance, a virtual machine can be a bare machine without an OS installed. Therefore, the “guest” property of the virtual machine could have no value at all.

What does it mean to you?

How You Can Beat Amazon EC2 by Differentiating Your Cloud Service

September 27th, 2010 No comments

With the gradual adoption of cloud computing, more companies are coming to the cloud service business. With so many new players in the game, how can you stand out in the competition as a service provider? In a marketplace already dominated by companies like Amazon, do you even have a chance?

Yes, you do! Differentiate.

I recently read a book, “Differentiate or Die: Survival in Our Era of Killer Competition” by Jack Trout and Steve Rivkin. Although it mainly talks about differentiation ideas in traditional businesses, it does offer good insights that can be applied in the cloud service business as well.

Conventional wisdom

Before we get to cloud services, let us take a look at what Jack and Steve say about differentiation. They first discuss some commonly held misconceptions about differentiating ideas:

DoubleCloud Named Top 50 Blogs on Cloud Computing

September 22nd, 2010 4 comments

I am glad to learn that I have been recently named as top 50 bloggers on cloud computing by Cloud Computing Journal. Check out here for all my cloud related articles.

Thanks all for making this happen! The momentum will continue…

“Last Inch” of Cloud Computing

September 20th, 2010 2 comments

The “last mile” or “last kilometer” is a term in the networking industry describing “the close to end connectivity from a communication service provider to a customer”. Although your infrastructure like backbones is very powerful, your end user experience could suffer if your last mile is not there yet.

For cloud computing, we’ve talked a lot about the data centers, backend servers. What about the end users? Your cloud data center could be very powerful too, but does it mean your users will fully leverage that power? Not necessarily. It depends on how you deliver the service to them.

Because cloud service is delivered with traditional network, the traditional network “last mile” issue is there as well. You surely need a good, if not better, connection to the network.

Beyond the connectivity, you will need good interfaces for your users to interact with the cloud. Let me go over the “last inch” options here.

Google made Instantiations free

September 18th, 2010 No comments

Following my prediction in a previous blog, Google just made the tools from its recently acquired Instantiations free to all developers/testers. Besides goodwills to the developer community, this move makes GWT a serious competitor to Adobe Flex, Microsoft Silverlight for Rich Internet Applications.

You can give these tools a try while developing and testing Web applications, or Java GUI applications:

  • GWT Designer
    “a powerful set of Eclipse-based development tools that enables Java developers to quickly create Ajax web applications using the Google Web Toolkit (GWT).”

Object Model of VMware vCloud API: The Big Picture in 2 Minutes

September 16th, 2010 No comments

VMware released the long-awaited vCloud API at VMworld 2010. The API is based on REST with 75 URLs defined in the user related part as you would find in the vCloud API Specification and vCloud API Programming Guide.

I am an OO guy (I am sure many of you are as well), and find it difficult to go through the 75 URLs and numerous XML tags as either input or return. These URLs are like trees in a forest. But where is the forest?

So I decided to create and show you a UML diagram (shown below) so that you can easily capture the key concepts of the vCloud API. In fact, there was a similar diagram in the programming guide of version 0.8.

Why No DatacenterRemovedEvent? A Deep Dive into vSphere Event Model

September 14th, 2010 13 comments

Last week I discussed how to get event type with vSphere API, followed by a comment asking why there is no DatacenterRemovedEvent? Very good question and almost got me there.

vSphere API has a comprehensive object model for event. In version 4.1, we have 474 different types of events representing different things happened in vSphere. When a managed entity is removed, there are normally events associated, for example, VmDeployedEvent, HostRemovedEvent, ClusterDestroyedEvent, ResourcePoolDestroyedEvent, DatastoreDestroyedEvent, DatastoreRemovedOnHostEvent, etc. It’s natural to expect DatacenterRemovedEvent. But you don’t find one.

It’s not only Datacenter which does have DatacenterCreatedEvent and DatacenterRenamedEvent. For Folder type, you find nothing other than VmBeingClonedNoFolderEvent which is not related to lifecycle of Folder at all.

Why are these events missing?

Categories: vSphere API Tags: ,

How to Avoid Virtual Machine Sprawl in Cloud Age?

September 13th, 2010 2 comments

Technology can be a lot like fashion, with quickly shifting trends. Once we embraced big iron but after the mainframe age the industry went into the client/server age where we soon found too many servers to manage. So we consolidated them, not back to mainframe age, but onto hypervisors. With one physical server, you can run multiple virtual machines.

Server consolidation solved a big problem and resulted in big cost savings. From management’s point of view, however, it does not actually reduce the number of servers to manage in your enterprise. To some extent, it worsens the problem!  In some circumstances it’s so easy and inexpensive to create a new virtual machine that you end up with many more servers than you really want or can effectively manage. This problem not only exists in private clouds, but also in the public cloud.

According to VMware CEO Paul Maritz in his keynote at VMworld 2010, the number of virtual machines exceeded physical machines in 2009, and will reach 10 million by the end of this year. This is definitely great news for the virtualization software industry but also a challenge moving forward.

So how should you try to solve the problem of virtual machine sprawl or even better, prevent it from happening? I discuss some solutions one by one here.

Creating Your Own Task and Event in vSphere

September 9th, 2010 33 comments

vSphere has a powerful extension mechanism that allows you to add new features as integral part of the platform. Many vendors have already leveraged this by providing plug-ins so that users can manage their components seamlessly within same vSphere Client.

You can actually do more than that with the extension. The following sample shows how to create your own task and event with vSphere API. The code should be self explanatory therefore I don’t elaborate much here. Note that you must run the sample with a vCenter server as extensibility is implemented only in vCenter.

When running the code, you can see a new task created and progresses with 10% every second in the “Recent Tasks” pane of vSphere Client. When the task is done, you will also see a new event posted in the “Tasks & Events” tab of the host you associate the task with.

What can you do with this capability? Here are two typical use cases:

Really Easy Ways to Capture VM Screenshot

September 8th, 2010 7 comments

Since vSphere 2.5, there is a feature allowing you to capture screen shot of a running virtual machine. It’s not well publicized but you can find a short description with screenshotSupported (boolean) in the HostCapability data object. Thanks to Nikita for bringing this up in the comment of the vSphere Java API 2.1 GA post.

Indicates whether the screenshot retrival over https is supported for this host’s virtual machines. If true, a screenshot can be retrieved at the HTTPS relative path /screen?id=<managed object ID of virtual machine or snapshot>. If any of the optional parameters ‘top’, ‘left’, ‘bottom’, and ‘right’ is specified, the returned image will be cropped from the rectangle with upper left corner (left, top) and bottom right corner (right – 1, bottom – 1). These values default to the top, left, bottom and right edges of the image. The client must use an authenticated session with privilege VirtualMachine.Interact.ConsoleInteract on the requested virtual machine or, in the case of a snapshot, the virtual machine associated with that snapshot.

The managed object ID of virtual machine is the value of ManagedObjectReference, which can be easily found using MOB.

Once you have it, you can issue a URL as follows in any browser and get the screen shot in PNG format.

How to Get Metadata of VMware vSphere API?

September 7th, 2010 No comments

VMware vSphere API is defined by WSDL. As discussed in my previous blog REST or SOAP, Web Services is by nature procedural, and it does not support OO (object oriented). This contributes to the learning curve of vSphere Web Service API which is modeled with OO.

What if you want to find out what properties are supported by a particular managed object type in vSphere API? There was a specific question recently in blog comment: how to get valid/supported property paths like summary.hardware.numNics with HostSystem type.

Currently there is no systematically way to get this metadata which is not defined in WSDL. You have to manually read through vSphere API Reference.

Since vSphere Java API 1.0 (a.k.a. VI Java API by then), I have manually added a getter method for every property in the Java API. So the metadata is built in vSphere Java API from the beginning. Whenever there is a manual process, it could be error-prone. As much carefully as I liked, I made mistakes with properties ignored in vSphere Java API occasionally. These mistakes have been immediately patched up upon bug reports or self reviewing.

To get exactly what you want programmatically, you have to do something extra with Java reflection API. Let’s pick HostSystem as an example here.

vSphere Java API at VMworld 2010

September 6th, 2010 No comments

Last week was a super busy week for all the people involved in VMworld 2010 in San Francisco. Because I spent two hours driving to Moscone Center and back home, I didn’t write any blog after getting back totally exhausted. Now it’s time to get back to it.

I believe there are many blogs/news on VMworld in general. Let me get down to a much narrow part: VMware Sponsored open source vSphere Java API at VMworld 2010.

Thanks to the community, my presentation on vSphere API Best Practice went very well. It’s based on the top 10 best practices blog (part 1, part 2) I wrote early this year, with real world experiences shared with partner engagements. Two copies of my book were given away at the end of the presentation. Thanks to Pablo Roesch for getting the books!

After the presentation, I was invited to check out new products built on top of vSphere Java API. I cannot disclose all of them here because some are not yet on the poweredby page. Here are several companies I can publicly talk about:

Ruby to Manage and Automate VMware vSphere?

August 30th, 2010 No comments

As I mentioned in a previous blog, vSphere(VI) Java API can be used in any JVM languages/frameworks. We have samples in Jython, Groovy, Grail. This weekend I got a sample in JRuby shared by our community member Martin Jackson in the API forum. Thanks Martin!

I think it would be fun to share it with you. If you can write Ruby code, you can take advantage of VI Java API for managing and automating vSphere as well. If you have samples leveraging the API to share, I am happy to hear about it.

Now, let us take a look at Martin’s sample code ported from a VI Java API sample.