While building a new server product of its own kind, we chose virtual appliance as the package. During the development phase, we configured thin disk to save space. But for our beta customers to try out, we decided to switch to thick disk for better performance as the product has to do lots of processing and heavy disk I/O. BTW, we still have a few slots for new beta customers with have large vSphere deployments. Please contact me if you are interested.
Then, a little problem came up – we found that the size of the OVA had a big jump of 150MB. Althougth it’s not a big deal, it would be better to keep it small to save customer time. Later on, we realize that it could also be faster with the extra steps.
Bothered by SLOW Web UI to manage vSphere? Want to manage ALL your VMware vCenters, AWS, Azure, Openstack, container behind a SINGLE pane of glass? Want to search, analyze, report, visualize VMs, hosts, networks, datastores, events as easily as Google the Web? Find out more about vSearch 3.0: the search engine for all your private and public clouds.
After asking for help on Twitter, I got two tips immediately from Timo Sugliani (@tsugliani) and Giuseppe Guglielmett (@gguglie): using dd to wipe disk, and use yum to clean cache. Bcause after the OS was installed, we also ran an automated installer which uses yum to install needed packages. It makes a lot sense to clean up what the yum has left on the disk. So that seems a low hanging fruit to try out.
# yum clean all
The export OVA reduced about 8MB in size, or 1% of the overall file size. Good but definitely not enough. The files may have been deleted from filesysstem metadata, but their content data may still remain on the disk. For VMware, it cannot tell whether a non-zero space is used or not on the disk, so it still treats the deleted disk space as data.
So simply yum clean is necessary but not enough.
I moved on with the dd command. It turned out it’s not that easy. For one thing, the virtual machine for build had the un-used files deleted already, thus there no easy way to zero out the disk space that were no longer tracked by filesysstem. Of course, I could re-run the script and use dd command before the rm command. If that is the case, second issue comes up: how to zero out all the files under a folder instead of a single file. Luckily, I got a little shell script from the Internet: ( I didn’t get a chance to try it BTW)
# find /folder/folder1 -type f | while read line; do dd if=/dev/zero of=$line bs=1k count=1024; done # rm -rf /folder/folder1
Still, I wanted a faster way than re-running the installer. A bit more study got me a small Linux program called zerofree. After installing it, I only found out that it supports ext filesystem, but not the xfs filesystem by my appliance.
# rpm -Uvh http://dl.fedoraproject.org/pub/fedora/linux/releases/21/Everything/x86_64/os/Packages/z/zerofree-1.0.3-3.fc21.x86_64.rpm
Then I was distracted by something else, the exploration paused for a couple days…
When I picked it up again, I did a bit more research and found a good solution from this post. The script there is for Ubuntu, so I had to take out only what I needed:
# dd if=/dev/zero of=/0bits bs=16M dd: error in writing '/0bits': no space left on device 805+0 records in 804+0 records out 13500571648 bytes (14GB) copied, 207.568 s, 65.0 MB/s
The command took about 4 minutes to finish. And it worked its magic – the new OVA reduced 100MB in size.
That is not yet the end – we have to verify the OVA actaully works. When running the new deployed appliance from the OVA, I got a new trouble – there is no space left so it cannot even create needed tmp files. Using the df command, I found out that the usage of the filesystem is 100% – no wonder that no new file can be created.
I quickly realized the “/0bits” is a file name. I have to admit that I was confused about what /0bits meant and thought it could be special parameter to tell the dd command to zero out zero sized files or something like that. After browsing a couple of dd tutorials, I decided to run the command anyway – in the worst case I can revert to the snapshot I took earlier.
With the no space problem, the dot got connected: the dd command copy zero bytes to /0bits file until there is no more un-allocated space according to filesystem. That essentially indirectly finds all the spaces used by deleted files. Simply deleting the /0bits file solves the problem easily. When I looked back to original blog post, the rm command was actually there. My bad – I could have been more patient.
This approach is a better way than the explict zero out deleted files. Why? It’s related to another VMware related setting. With the thick disk, there are two differnt option: eager zeroed and lazy zeroed. That does that mean? When a new VMDK is created, the vm disk is guarantted to be filled with all zero bits if you choose eager zero. This of course takes longer to finish, and in most cases it’s not necessary. But it has a clear benefit for building a new virtual appliance to keep its expoerted OVA small. What if you have forgot to pick the right one? No worry. Using the dd command as above will help to solve the problem.
After rebooting the appliance, I noticed that VM seems to run a lot faster than the earlier version. I don’t know there is a direct connection with dd command, but it is definitely a nice-to-have side-product.
Well, that is quite some writing. Time for me to modify the installer and building a smaller and faster appliance for our beta customers.
Last note: although the above technique is tested on VMware, but really it should work for creating Linux based virtual appliance for other hypervisors: Hyper-V, XEN, KVM.