Using libvirt

I (nwf) highly suggest against it if at all possible. What a pain in the rump. You’ll probably be spending a lot of time on over here learning how to write domain files. Here’s a collection of site-specific notes. The only reason it’s here is that this turns out to be the best easy-and-mostly-standard way of wrangling QEMUs into running at boot for the world’s benefit, which unfortunately we must do in order for afs to be reasonably manageable. Use virsh autostart ${DOMAIN} once you’ve got the domain created, to make that auto-run happen.

Debugging XML

Egads this is painful. You can get a good/no-good bit result out of the tool virt-xml-validate, but you should ignore the error messages it prints, most of the time, as it is a remarkably poor producer. I suggest the usual divide-and-conquer approach of commenting out half the file and narrowing from there.

You can also compare a round-trip through libvirt: use virsh define ${YOUR_XML_DOCUMENT} and virsh dumpxml ${YOUR_DOMAIN_NAME}.

Using Ceph

http://ceph.com/docs/master/rbd/libvirt/ will tell you what’s up, but for ease of administration, here are the steps:

• You’ll want a ceph principal for your host; may I suggest client.$(HOSTNAME).libvirt. Go use Creating a New Ceph User to make one and extract its key. For caps, you’ll probably want something like:

  "osd" "allow class-read object_prefix rbd_children, \
         allow rwx pool=volumes_2, allow rw pool=images_2,
         allow rwx pool=scratch"
  "mon" "allow r"
  

If you’re standing up an AFS server using Ceph partitions, you probably need to augment those pool lists.

• You need to register this secret with libvirt. Create an XML file (of course):

  cat >virsh-secret.xml <<HERE
  <secret ephemeral='no' private='yes'>
      <usage type='ceph'>
              <name>client.$(HOSTNAME).libvirt secret</name>
      </usage>
  </secret>
  HERE
  

  Then, virsh secret-define --file virsh-secret.xml which will print out a UUID for you, which you can also retrieve with virsh secret-list. Then virsh secret-set-value ${UUID} --base64 ${EXTRACTED_KEY}. Then rm ${EXTRACTED_KEY} virsh-secret.xml.

• Now you need to actually use that key with your RBD stanzas in domain XML. Try something like:

  <disk type='network' device='disk'>
    <driver name='qemu' type='raw' cache='writeback'/>
    <auth username='magellan.libvirt'>
      <secret type='ceph' usage='client.magellan.libvirt secret'/>
    </auth>
    <source protocol='rbd' name='...'/>
    <target dev='vd...' bus='virtio'/>
  </disk>
  

  It appears that the virtio drivers are more robustly tested with Ceph than, for example, the IDE drivers. We have had I/O timeouts and guest kernel panics with the latter but have not seen them with the former. Modern QEMU BIOSes are up to snuff enough to boot off of virtio devices.

Serial Console

In your domain XML, stick:

<serial type="pty"><target port="0"/></serial>
<console type="pty"><target type="serial" port="0"/></console>

And on the guest, follow the instructions in Linux Serial Console. Then you can use virsh console ${DOMAIN} once it’s up and running to attach to the console from the host.

Note

It’d be better if we could run -curses mode inside screen or something, but this is not to be, so here we are. This probably means that you should do the install of a libvirt guest separately from the libvirt side of things, which necessitates learning not only how to write a domain file but how to write QEMU arguments directly.

Libvirt Events

Your domain file should probably contain these hooks:

<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>restart</on_crash>
<on_lockfailure>poweroff</on_lockfailure>