Setting Security Certificate: What VMware Did Not Tell You
In my last article, I discussed on the format requirement for Java APIs and how I found out the root cause and its solution. Even more mysterious is the format requirement of VMware vCenter as I discovered in another VMware related project, in which I needed to register an extension with vCenter and set up its certificate.
You may be wondering why certificate is needed for an extension. The short answer is that with certificate, your extension or application can login vCenter with your private key without password, something like SSH to ESXi without password. This feature has been supported in the VMware ESX Agent Management API and the commercial version of vijava API.
Time to learn how to "Google" and manage your VMware and clouds in a fast and secureHTML5 App
It took me quite some time to figure out the specific format requirement for the certificate because it’s not adequately documented anywhere. For that reason, I think others in the community may benefit from my experience and solution.
To set security certificate for a registered extension with vSphere API, you can use the ExtensionManager managed object type, which has a method called setExtensionCertificate() that takes two parameters (the _this is not counted here because it’s hidden completely in vijava API) as follows:
- extensionKey: xsd:string Key of extension to update.
- certificatePem*: xsd:string PEM encoded certificate. If not specified, the certificate passed over SSL handshake is used.
Both parameters are pretty straight forward. But when I called the method to set security certificate for my extension, I got the famous InvalidArgument exception (see vSphere API and SDK FAQ). According to the vSphere API doc, it means “the certificate described by “certificatePem” is not in PEM format, or could not be decoded to an X.509 certificate.”
For the first part, I tried to read the certificate with standard Java API and succeeded by inserting CRLF after the header. So I was pretty sure the certificate I had was a good one. But still, it could not be taken by vSphere API. There left only one choice, which is that the vSphere API may expect a different format than I had supplied.
Given that the certificate string encoded in PEM is pretty long(over 1,000 characters), there could be countless ways to format the certificate string and probably only one way (?) would make vSphere API happy. With this in mind, I had to think differently. I decided to use the DoubleCloud Proxy I had wrote before and record the message from vSphere Client because the same certificate somehow worked with the GUI.
Recording the messages between vSphere Client and vCenter turned out to be quite easy process with the DoubleCloud Proxy. Once the recorded SOAP message showed in the DoubleCloud Proxy GUI, everything becomes clear. There, the certificate is chopped into many lines of 64 characters with many CRLF (“\r\n” in Java/C/C++ programming) delimiter as the following. The last line length can be less than 64 characters. I just made up the last line so don’t assume that is the exact length of last line.
-----BEGIN CERTIFICATE-----<CRLF> IBAgIJAKMIIDRTCCAi2gAwlwrFcAdHQtMA0GCSqGSIb3DQEBBQUAMCAxHjAcBgNV<CRLF> BAMTFWxvY2FsaG9zdC5sb2NhbGRvbWFpbjAeFw0xNDAyMjExMzM4NTBaFw0yNDAy<CRLF> ... Bp95+uKYTY4bnOixW/3X80t+2zMnJqPJ62UhHuKzw==<CRLF> -----END CERTIFICATE-----<CRLF>
The solution is to insert these CRLF and make it the same structure as shown above. If your PEM based certificate is not like above, just write a few line of code to split lines and insert CRLF delimiters.
This is a good example of how a good API documentation is badly needed. In reality, it’s pretty hard to achieve. I don’t think VMware did not want to tell your about the format but probably the developers had assumed that all the certificates come with that format and might have tested the APIs with that assumption even though the assumption is not necessarily true and not part of the standard. Even better had vCenter been more robust in handling different formats.