Innovation, Openness, Community: Celebrating 2nd Anniversary of vSphere Java API
It’s been two years since the first release of VI Java API on May 22, 2008. As of today, this VMware sponsored open source API has been downloaded more than 8,200 times including 1,000+ direct access to the source code from SVN. More importantly, the API has been widely adopted by VMware Community including partners and customers in their products and operation environment. I was amazed to hear success stories from companies which use this API to manage 10,000 servers.
Besides Java, people also started to use JVM languages such as Jython and Groovy to leverage the API. I think that is the direction for the API to be used by system administrators who look for an alternative to vSphere PowerCLI which is a great product with no choice but on Windows only.
Lost VMs or Containers? Too Many Consoles? Too Slow GUI? Time to learn how to "Google" and manage your VMware and clouds in a fast and secure HTML5 App.
How Was It Started?
The API was started because of my personal experience while helping VMware strategic partners. I found the Web Services SDK hard to use, and related samples hard to follow because of the procedural nature of Web Services. So I started to think how to solve the problem. During March and April 2008, I wrote code every night on my old Compaq notebook after helping my kid to sleep at night.
At the same time, I heard the company’s policy regarding open source, and filed a request for internal review process. When the request was approved by our legal team and CTO Steve Herrod, I submitted a request on sourceforge.net for a new project which took only two days to finish. After everything was ready, I pushed the code to sourceforge.net and announced the first alpha release of the API.
At the time, I was also doing marketing for people’s awareness of the API. Thanks to my colleagues Mike DiPetrillo, Duncan Epping, and other external bloggers like Alessandro Perilli and Eric Sloof to spread the words. Websites like infoQ.com also released a news on the availability of the API at the same time. InfoQ.com also sent folks to record my presentation on the API at SpringOne last year.
Internally, the API was selected at VMware Engineering Innovation Offsite which is a yearly event for new ideas from engineers. This was indeed encouraging.
Two Major Challenges
There were many challenges along the way for the open source project. Technically, there were two major challenges: simplicity and performance.
The first release tried to solve the problem with a well designed object model which is partially inferred from the API reference. It also creatively hides the ManagedObjectReference and PropertyCollector which are two major hurdles for people to use the SDK. Normally simplicity is taken for granted and mostly under-valued. But it takes effort to simplify things. VI Java API 1.0 nicely simplified the SDK.
The next biggest challenge was to get rid of Apache AXIS. The AXIS is a generic Web Services tool, but is too slow and creates huge memory leak over the time. For instance, a QA team found their memory consumption went up to 1,000+M after a 3-day run of their tests. Later on, they tried vijava 2.0 and got only 8M consistently for the same tests, not to mention the huge performance improvement. This was of course after 2.0 version was released.
By the end of 2008, I started to think about a solution. With Apache brand and code base there, it took courage to take on it. To be honest, I tried to understand how AXIS works but never really got it. Reading the code and inspecting the stack trace just drove me crazy.
I was then thinking, “why not write a new one?”, “Because we have Just Enough Operating System, how about Just Enough Web Services?” Following this thinking, I started to study the SOAP messages and WSDL, and write test code to serialize and de-serialize the data objects.
By the time of Christmas of 2008, I had the basic pilot code working. During the year end, I had more time to work on the stuff. By that time, I was done with the draft of my book as well. So I could focus on writing a new Web Services engine.
After many weekends and nights, the first beta of 2.0 was released on Feb 5, 2009. Right after the release, I got emails from James Burke and David Kelly from NetApp teams telling me their happiness with performance boost. It’s always encouraging to get such feedbacks from the community.
There were bugs as always. It took two months to stabilize. Along the way, I did a big code refactoring for better organization. Many folks in the community filed bugs. I still remember an interesting bug filed by Brian McFeely from Tripwire. I said it was interesting because it surfaced after about 200 de-serialization and choked the JVM with the dimension of array it can possible handle. No wonder I didn’t find it in my test. Thanks to Brian who also offered an analysis and a solution!
After 2.0 was released, gone was the Apache AXIS for Java developers. The adoption was accelerated because of its technical excellence. For one thing you don’t need to handle 20 or so jar files, which could be a big hassle for anyone. Also, people reported up to 15 times performance gain over AXIS.
The Success Factors
There are many factors contributing to the success of this API as it has helped the community. Besides the innovations involved in the two major releases, I think open source model really worked very well. Everyone got a chance to participate: reads my code, contributes code, files bugs, suggests features, and ask/answer questions. We also got contributions from other colleagues Tom Elloit and Lu Yu at VMware and Eric Forgette from NetApp and Elsa Bigonoli from Altor Networks.
Being open is not enough. We’ve seen many open source project is no longer actively maintained. Community building is critical. Actively maintaining the project and answering the question help build confidence in the project. Critical bugs have been fixed overnight, and most questions answered in 24 hours. Our forum has 830 messages.
In some sense, helping the community is the best marketing. Experts like Andrew Kutz, William Lam, Matt Cowger and many others just spread the words out on various forums. More testimonials from Michael Ottati and others experts can be found at testimonial page.
Big thanks to all who help the project a success!
Appendix. Key Milestones
May 22, 2008 – 1.0 Alpha
Oct 18, 2008 – 1.0 GA
Dec. 29, 2008 – first 1,000 downloads
Jan 26, 2009 – 1.0 U1
Feb 05, 2009 – 2.0 Beta
April 4, 2009 – 2,000 downloads
June 25, 2009 – 2.0 GA
July 15, 2009 – 3,000 downloads
Oct. 19, 2009 – 4,000 downloads
Dec. 5–2.0 GA update 1
Jan 08, 2010 – 5,000 downloads
March 12, 2010 – 6,000 downloads
May 6, 2010 – 7,000 downloads