Setting Security Certificate: What VMware Did Not Tell You

In my last article, I discussed on the format requirement for Java APIs and how I found out the root cause and its solution. Even more mysterious is the format requirement of VMware vCenter as I discovered in another VMware related project, in which I needed to register an extension with vCenter and set up its certificate.

You may be wondering why certificate is needed for an extension. The short answer is that with certificate, your extension or application can login vCenter with your private key without password, something like SSH to ESXi without password. This feature has been supported in the VMware ESX Agent Management API and the commercial version of vijava API.

Bothered by SLOW Web UI to manage vSphere? Want to manage ALL your VMware vCenters, AWS, Azure, Openstack, container behind a SINGLE pane of glass? Want to search, analyze, report, visualize VMs, hosts, networks, datastores, events as easily as Google the Web? Find out more about vSearch 3.0: the search engine for all your private and public clouds.

It took me quite some time to figure out the specific format requirement for the certificate because it’s not adequately documented anywhere. For that reason, I think others in the community may benefit from my experience and solution.

To set security certificate for a registered extension with vSphere API, you can use the ExtensionManager managed object type, which has a method called setExtensionCertificate() that takes two parameters (the _this is not counted here because it’s hidden completely in vijava API) as follows:

      extensionKey: xsd:string Key of extension to update.
      certificatePem*: xsd:string PEM encoded certificate. If not specified, the certificate passed over SSL handshake is used.

Both parameters are pretty straight forward. But when I called the method to set security certificate for my extension, I got the famous InvalidArgument exception (see vSphere API and SDK FAQ). According to the vSphere API doc, it means “the certificate described by “certificatePem” is not in PEM format, or could not be decoded to an X.509 certificate.”

For the first part, I tried to read the certificate with standard Java API and succeeded by inserting CRLF after the header. So I was pretty sure the certificate I had was a good one. But still, it could not be taken by vSphere API. There left only one choice, which is that the vSphere API may expect a different format than I had supplied.

Given that the certificate string encoded in PEM is pretty long(over 1,000 characters), there could be countless ways to format the certificate string and probably only one way (?) would make vSphere API happy. With this in mind, I had to think differently. I decided to use the DoubleCloud Proxy I had wrote before and record the message from vSphere Client because the same certificate somehow worked with the GUI.

Recording the messages between vSphere Client and vCenter turned out to be quite easy process with the DoubleCloud Proxy. Once the recorded SOAP message showed in the DoubleCloud Proxy GUI, everything becomes clear. There, the certificate is chopped into many lines of 64 characters with many CRLF (“\r\n” in Java/C/C++ programming) delimiter as the following. The last line length can be less than 64 characters. I just made up the last line so don’t assume that is the exact length of last line.


The solution is to insert these CRLF and make it the same structure as shown above. If your PEM based certificate is not like above, just write a few line of code to split lines and insert CRLF delimiters.

This is a good example of how a good API documentation is badly needed. In reality, it’s pretty hard to achieve. I don’t think VMware did not want to tell your about the format but probably the developers had assumed that all the certificates come with that format and might have tested the APIs with that assumption even though the assumption is not necessarily true and not part of the standard. Even better had vCenter been more robust in handling different formats.

This entry was posted in vSphere API and tagged , , , . Bookmark the permalink. Post a comment or leave a trackback: Trackback URL.

Post a Comment

Your email is never published nor shared. Required fields are marked *


You may use these HTML tags and attributes <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>


    My company has created products like vSearch ("Super vCenter"), vijavaNG APIs, EAM APIs, ICE tool. We also help clients with virtualization and cloud computing on customized development, training. Should you, or someone you know, need these products and services, please feel free to contact me: steve __AT__

    Me: Steve Jin, VMware vExpert who authored the VMware VI and vSphere SDK by Prentice Hall, and created the de factor open source vSphere Java API while working at VMware engineering. Companies like Cisco, EMC, NetApp, HP, Dell, VMware, are among the users of the API and other tools I developed for their products, internal IT orchestration, and test automation.