Our business team invited me to a phone call with one of our strategic partners days ago. They had a scalability issue with their server application. It turned out to be related to session management. I think they are not the only one who got into this type of problems, and most likely not the last one. So I decide to share it and hopefully you can avoid similar problems in your projects.
The partner has a server implementation on top of vSphere API. Their server takes service requests from their clients. Everything worked fine in the lab. After moving it to the production environment, they saw a big slow down. Their production environment had about 10 ESX servers plus 300 VMs, which are way below one vSphere vCenter can comfortably handle. So what’s the problem?
Bothered by SLOW Web UI to manage vSphere? Want to manage ALL your VMware vCenters, AWS, Azure, Openstack, container behind a SINGLE pane of glass? Want to search, analyze, report, visualize VMs, hosts, networks, datastores, events as easily as Google the Web? Find out more about vSearch 3.0: the search engine for all your private and public clouds.
After questions and a little testing with MOB ( a handy browser based tool every vSphere API developer should know and be familiar with), I found out that they had more sessions than really needed. For each user to their server, it creates a new session to the backend vSphere. When the number of the sessions grows, it slows down the vSphere. It explains why it works fine in lab, but not in production where there are much more sessions are created.
Each session to the vSphere uses some resources and put burden on the server. When your session also uses the property collector’s property filter feature, it uses even more resources. For each session and property filter, the server needs to watch on the changes on your behalf and track the versions!
I don’t have performance data that shows the turning point against the number of sessions. As I heard from others, the response time of vSphere Client gets slow down when there are about 30 concurrent login vSphere Clients. It doesn’t matter you use it or not as long as the vSphere Client is still open. So here is a quick reminder to everyone: logout your vSphere Client when you don’t need it anymore.
Each vSphere Client uses at least one session and it uses property filters extensively. This experiment reveals something about the sessions in general — use as few sessions as possible in your applications.
Now the direction is clear. How to get it right?
In your server applications, you don’t want to create a new session for each new user to your server. You just need one session to the vSphere with the highest combined privileges of all your possible users. This of course gets you more work– you have to handle the user authentication/permission by yourself. You will need to have a database with your server to persist these data. If your server is really busy, you can have one or two more additional sessions, but you have to manage these sessions. For example, you can dedicate one session for one datacenter. Since every project is different and you know your application frameworks better than I do, I won’t elaborate more here.
In VI Java API 2.0, I have developed a caching framework which can be a great help to your application development, especially server applications. You can create a CacheInstance for each ServiceInstance, and tell the framework what properties on which managed objects you want to monitor. The framework then takes care of the rest with a backend thread. When you need to read the properties, you just read from the cached instance like from a hash table. I have created a sample code illustrating how the API can be used.