Uncover KVM and Virtual Machines in OpenStack

After installing OpenStack and posting a few articles, I started to dig down a bit more on the KVM hypervisor used in OpenStack. For that, I wrote about the libvirt API and how to remotely manage KVM with it.

In this article, I will introduce how KVM is used in Openstack and what a virtual machine is made of.

Lost VMs or Containers? Too Many Consoles? Too Slow GUI? Time to learn how to "Google" and manage your VMware and clouds in a fast and secure HTML5 App.

How A Virtual Machine Instance Comes to Life?

The KVM hypervisor is installed on Nova compute node. After user request a new virtual machine instance, the nova talks to KVM to create a new virtual machine instance. There are some API/messaging between Horizon and Nova. Here let’s take a close look at the interaction between Nova and KVM.

To get to the details, you can always dig into the source code of Nova and find out how KVM is exactly called there. There is actually an easier way and that is what I will use to show you.

The following is the command I used to list the process for a virtual machine instance. There are a few more instances but I only keep the related output for one virtual machine instance for simplicity.

[root@node1 ~]# ps aux | grep kvm
root      1884  0.0  0.0      0     0 ?        S    Jun14   0:00 [kvm-irqfd-clean]
qemu      4518  6.7  0.6 1995916 690792 ?      Sl   Jun14 2298:13 /usr/libexec/qemu-kvm -name instance-0000000d -S -M rhel6.4.0 -cpu Westmere,+rdtscp,+pdpe1gb,+dca,+pcid,+pdcm,+xtpr,+tm2,+est,+smx,+vmx,+ds_cpl,+monitor,+dtes64,+pclmuldq,+pbe,+tm,+ht,+ss,+acpi,+ds,+vme -enable-kvm -m 512 -smp 1,sockets=1,cores=1,threads=1 -uuid 5a70225f-87c4-42f4-8f95-b12bbae282c7 -smbios type=1,manufacturer=Red Hat Inc.,product=OpenStack Nova,version=2013.1.2-1.el6,serial=c92d59a0-277c-11e2-3210-32100000001e,uuid=5a70225f-87c4-42f4-8f95-b12bbae282c7 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-0000000d.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -no-kvm-pit-reinjection -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/nova/instances/5a70225f-87c4-42f4-8f95-b12bbae282c7/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=24 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:45:5b:67,bus=pci.0,addr=0x3 -chardev file,id=charserial0,path=/var/lib/nova/instances/5a70225f-87c4-42f4-8f95-b12bbae282c7/console.log -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -device usb-tablet,id=input0 -vnc 172.23.45.22:0 -k en-us -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
root      4519  0.0  0.0      0     0 ?        S    Jun14   0:00 [kvm-pit-wq]

As you can find from the output, the process with ID 4518 is the one for a virtual machine instance. It includes the command called by Nova and all the arguments that defines the virtual machine hardware, location, log, VNC connection, etc. Because of many arguments, it’s not quite easy to read all the details. I will show you the information in nice and readable way soon.

What is a Virtual Machine Instance Made Of?

Like VMware virtual machine, a KVM virtual machine instance consists of a few files mostly in a directory. These files can be found from the host operating system with normal file system commands as follows.

[root@node1 ~]# ls -al /var/lib/nova/instances/5a70225f-87c4-42f4-8f95-b12bbae282c7/
total 52428
drwxr-xr-x. 2 nova nova     4096 Jun 14 16:15 .
drwxr-xr-x. 8 nova nova     4096 Jul  8 10:50 ..
-rw-rw----. 1 nova qemu    42824 Jun 17 11:21 console.log
-rw-r--r--. 1 qemu qemu 53870592 Jul  7 16:34 disk
-rw-r--r--. 1 nova nova     1548 Jun 14 16:15 libvirt.xml

Note that there is actually a hidden file called .swp, which I think for swapping virtual machine memory.
The disk file holds the disk image for the virtual machine instance. The console.log tracks the output of the virtual machine instance console, including the commands the users type in the virtual machine instance console. This is the source of what you see in the log of a virtual machine instance from Horizon GUI.

As mentioned earlier, I will show you more readable information for a virtual machine instance. This is the libvirt.xml file as listed in the following. If you are familiar with VMware, this file is very much like the .vmx file for a virtual machine in terms of its purpose. I know XML is not really that readable, but definitely more so than the command line arguments.

[root@node1 ~]# vim /var/lib/nova/instances/5a70225f-87c4-42f4-8f95-b12bbae282c7/libvirt.xml
<domain type="kvm">
  <uuid>5a70225f-87c4-42f4-8f95-b12bbae282c7</uuid>
  <name>instance-0000000d</name>
  <memory>524288</memory>
  <vcpu>1</vcpu>
  <sysinfo type="smbios">
    <system>
      <entry name="manufacturer">Red Hat Inc.</entry>
      <entry name="product">OpenStack Nova</entry>
      <entry name="version">2013.1.2-1.el6</entry>
      <entry name="serial">c92d59a0-277c-11e2-3210-32100000001e</entry>
      <entry name="uuid">5a70225f-87c4-42f4-8f95-b12bbae282c7</entry>
    </system>
  </sysinfo>
  <os>
    <type>hvm</type>
    <boot dev="hd"/>
    <smbios mode="sysinfo"/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <clock offset="utc">
    <timer name="pit" tickpolicy="delay"/>
    <timer name="rtc" tickpolicy="catchup"/>
  </clock>
  <cpu mode="host-model" match="exact"/>
  <devices>
    <disk type="file" device="disk">
      <driver name="qemu" type="qcow2" cache="none"/>
      <source file="/var/lib/nova/instances/5a70225f-87c4-42f4-8f95-b12bbae282c7/disk"/>
      <target bus="virtio" dev="vda"/>
    </disk>
    <interface type="bridge">
      <mac address="fa:16:3e:45:5b:67"/>
      <model type="virtio"/>
      <source bridge="qbr09266687-51"/>
      <target dev="tap09266687-51"/>
    </interface>
    <serial type="file">
      <source path="/var/lib/nova/instances/5a70225f-87c4-42f4-8f95-b12bbae282c7/console.log"/>
    </serial>
    <serial type="pty"/>
    <input type="tablet" bus="usb"/>
    <graphics type="vnc" autoport="yes" keymap="en-us" listen="192.168.45.22"/>
  </devices>
</domain>

Going Beyond Individual Virtual Machine Instances?

There are of course more than one virtual machine instances that run on KVM at a time. To accommodate this, a structure of folders is there for metadata of virtual machines. The compute_nodes file in the instances folder is such a file for that purpose as shown below. The more virtual machine instances you have, the more entries in the file.

[root@node1 ~]# vim /var/lib/nova/instances/compute_nodes
{"node1.openstack": 1373304730.1942029, "localhost.localdomain": 1371232588.703598}:q
 
Under the same instances folder is a special _base folder which seems to be the home of virtual machine images (think it as virtual machine templates in VMware world).
[root@node1 ~]# ls /var/lib/nova/instances/_base/
174a6098fea68ea2de1220de7e4680051c2fd06c
[root@node1 ~]# ls -al /var/lib/nova/instances/_base/174a6098fea68ea2de1220de7e4680051c2fd06c
-rw-r--r--. 1 nova qemu 41126400 Jul  8 10:32 /var/lib/nova/instances/_base/174a6098fea68ea2de1220de7e4680051c2fd06c

More on Nova with KVM

Going beyond the KVM virtual machines, there are several more folders in the /var/lib/nova folder as show below. If you are familiar with Openstack already, you easily guess out the intended usage of these folders.

[root@node1 nova]# ls -al /var/lib/nova
total 32
drwxr-xr-x.  8 nova nova 4096 Jun 13 12:15 .
drwxr-xr-x. 52 root root 4096 Jun 19 10:51 ..
drwxr-xr-x.  2 nova nova 4096 Jun 10 03:32 buckets
drwxr-xr-x.  2 nova nova 4096 Jun 10 03:32 images
drwxr-xr-x.  8 nova nova 4096 Jul  8 10:50 instances
drwxr-xr-x.  2 nova nova 4096 Jun 10 03:32 keys
drwxr-xr-x.  2 nova nova 4096 Jun 10 03:32 networks
drwxr-xr-x.  2 nova nova 4096 Jul  8 11:26 tmp

The only tricky one is the tmp folder which is for temporary files. To find out, just issue a command and everything is clear:

[root@node1 tmp]# ls -al /var/lib/nova/tmp
total 8
drwxr-xr-x. 2 nova nova 4096 Jul  8 11:26 .
drwxr-xr-x. 8 nova nova 4096 Jun 13 12:15 ..
-rw-r--r--. 1 nova nova    0 Jul 12 13:57 nova-storage-registry-lock
This entry was posted in Cloud Computing, Virtualization and tagged , , . Bookmark the permalink. Post a comment or leave a trackback: Trackback URL.

4 Comments

  1. nido
    Posted March 11, 2014 at 3:55 pm | Permalink

    hi

    I need some help if possible for you guide me about it. I am working
    on suspended VM migration , can you tell me where i can find all
    related files in openstack of some instance and then move those to
    other openstack and run it .. do you know how can i do this? or some assistance or guidance from your side

  2. Posted March 11, 2014 at 5:19 pm | Permalink

    You may want to check out the OpenStack REST APIs, or go directly with libvirt which can be programmed in Java. Search for the article I wrote on it.

    Steve

  3. nido
    Posted March 11, 2014 at 11:17 pm | Permalink

    Can you explain what i need to check in openstack REST API.
    can you share link of article here please

  4. Posted March 12, 2014 at 5:02 pm | Permalink

    I think each OpenStack component has its own REST API. You may want to check the NOVA (compute) one in particular.

    Good luck!

    Steve

Post a Comment

Your email is never published nor shared. Required fields are marked *

*
*

You may use these HTML tags and attributes <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

  • NEED HELP?


    My company has created products like vSearch ("Super vCenter"), vijavaNG APIs, EAM APIs, ICE tool. We also help clients with virtualization and cloud computing on customized development, training. Should you, or someone you know, need these products and services, please feel free to contact me: steve __AT__ doublecloud.org.

    Me: Steve Jin, VMware vExpert who authored the VMware VI and vSphere SDK by Prentice Hall, and created the de factor open source vSphere Java API while working at VMware engineering. Companies like Cisco, EMC, NetApp, HP, Dell, VMware, are among the users of the API and other tools I developed for their products, internal IT orchestration, and test automation.