XenSource has some pretty good GUI tools. There’s XenCenter and the xsconsole, both of which are pretty adequate in a free sense and get pretty darn interesting when you actually pay Citrix. But today I want to take a little look under the hood of XenSource. I had previously written about Xen. But note that this is a different beast.

Before I get started talking about how to do some tasks in XenSource, I first want to throw out there a few terms. The first, is virtual machine. This is exactly what it sounds like, an operating system that runs on a virtual host rather than a physical hosts. So take that Dell PowerEdge that’s about 9 years old that you just use to access your Golden Girls collection on the NT server you used napster to steal it on and virtualize it. Sweet, now your mp3s can still run too!

The next term is hypervisor. This is just a really, really dumb physical machine that is super beefy and which is going to run some VMs for ya’. I like to think of a Hypervisor like the old pictures of the Governator back when Arnold weightlifted. He’s the hypervisor. They’re the weights. Illegal instructions is what happened when he made ill-fated attempts to meet ladies (I apologize in advance for any ill fated attempts at humor, it’s late…).

DomU

So the hypervisor loads these virtual machines. Those are the basic virtualization terms that most people likely know. But the term domain is probably different for many. This is the context in which a VM runs. Think of it kinda’ like reserved resources (I use the term reserved very loosely here). To see your domains, we’ll look at our first command. Many of the XenSource commands sit in /opt/xensource/bin (so I’ll include the dirname of commands in this article). In here, you’ll find list_domains which interestingly shows you domains (e.g. dom0, dom1, dom2, etc):

/opt/xensource/bin/list_domains

Note: In this article I use the default location for commands. You should too. If you put the binaries somewhere else then there’s a chance you aren’t bright. Unless you have a good reason. Then you’re definitely not bright ’cause you’ve clearly invoked something I call “xen stoner logic.” It is however, possible to resolve this issue: stop doing drugs.

If you’ve got a lot of domains running, you can grep out the list for a given UUID. Let’s say that I have a UUID of BACHMAN-TURNER-OVERDRIVE. I’m going to use this UUID throughout the article as my example UUID. Any time you see this, replace it with your hex-based UUID. If you note, my UUID example is not hex based. This is something you must overlook because I’m writing this article for me and while I will be on to other bands in my iTunes playlist by the end of this article, that’s where I am now… Anyway, you take the UUID (also referred to as the dom-id of the VM ) and use list_domains, then pipe out your output to grep with a first position of the UUID after the grep, as follows:

/opt/xensource/bin/list_domains | grep BACHMAN-TURNER-OVERDRIVE

Now, when I referred to the “context” of the virtual machine, think about this. The process of the VM is basically the dom. You can transfer that process to another host in a pool, which is basically a cluster. You can also end up having a failed shut down of a virtual machine and end up with something in an unresponsive state. So to restart your domains, use the id of the bad domain and

/opt/xensource/bin/destroy_domain –dom-id 0

Remote Connectivity

Pretty much, each of the commands we’re about to use can be bookended with options that allow for remote connectivity. We’re basically running these locally using the xe command. Any time you use xe, you can throw a -s -u -pw option before the commands we go through, defining server username and password for that username respectively. For example, a very basic command that will show you optical media and ISOs available to your host is xe cd-list. We’re going to take that command and run it against a remote server:

/opt/xensource/bin/xe -s 10.0.0.2 -u root -pw B0ston cd-list

The options can go in front of or behind most of the commands or arguments. I find that I end up using screen a lot more than using all this, if anyone cares… Except backup, as I’ll explain in that section of this article.

Managing Tasks

Sometimes, you have a task stuck or sucking up too many resources. This can cause a VM to become completely unresponsive (like, can’t ping it) or otherwise run very poorly. You see a pattern across multiple doms on the same host and so decide to check to see if XenSource has any tasks eating up resources (like a backup). To see the task list for a given box, check the task-list. The xe task-list command will show you what tasks Xen is doing:

/opt/xensource/bin/xe task-list

You can then stop a task if it doesn’t stoke you out, using the task-cancel command:

/opt/xensource/bin/xe task-cancel uuid=BACHMAN-TURNER-OVERDRIVE

Now, not all tasks cancel gracefully (just ask anyone who’s tried taking candy from a toddler in mid-munch), so the task-cancel (and most other commands in XenSource) comes with a nifty force=true option:

/opt/xensource/bin/xe task-cancel force=true uuid=BACHMAN-TURNER-OVERDRIVE

The xe task-list command doesn’t show you what VMs are running per se, so I find that xentop is a useful command for checking on VMs, helping to show when one is spinning out of control or aggregate information about VMs. Sure, you can see this information with XenCenter but you don’t seriously trust GUIs do you?!?! Especially not one that can’t connect to the open source versions of Xen!!! Anyway, xentop is like top but for Xen (go figure, right?!?!). Xentop comes with a few options and interactive commands when in the xentop interface. The options are:

• -h: See a help page
• -i: I always use this one these days as I may otherwise forget to kill my session, which just takes up resources. The -i indicates how many iterations before the interactive environment ends.
• -d: Delay, waits a few more or less seconds between refresh (the default value here is 3 seconds). The interactive command for this while xentop is running is D.
• -V: Shows the version of xentop. Can be enabled during interactive mode using the V key.
• -n: Includes information about networks. This can be turned on while xentop is running by holding down the N key.
• -x: includes block data.
• -r: This is useful if you have a lot of domains or a lot of wrapping as it repeats the header for each.
• S: Changes the order to sort to the next tab/header
• Q: The interactive key to stop xentop
• Arrow: Scrolls the screen

xentop is one of the more common commands I run as I also use it to find some pretty basic information. Given the above options, to fire it up to refresh every 1 second for 5 minutes, show network information, repeat headers and show block data:

/opt/xensource/bin/xentop -i 300 -d 1 -n -x -r

Managing Virtual Machines

If you have a domain that goes unresponsive but shows running in xentop, once you’ve confirmed that it’s an issue with the VM itself, go ahead and try to restart it:

/opt/xensource/bin/xe vm-reboot vm-name=bachman

You can also reboot based on UUID of the VM (you don’t really trust the translation of names to addresses without a static hosts file do you?!?!):

/opt/xensource/bin/xe vm-reboot vm-id=vm_UUID=BACHMAN-TURNER-OVERDRIVE

Or just reset the power state, like when a dom still exists for a VM called badcompany:

/opt/xensource/bin/xe vm-reset-powerstate vm=badcompany --force

You can also fire up a nice interactive DOS-style GUI. To do so, run xsconsole, which can be run via ssh and is uber-fast. The other thing I like about xsconsole is you can see things like the serial number of the computer, networking information, pool information and most other things that can be queried from xe:

/opt/xensource/bin/xsconsole

Managing Networking Within Xen

Every time I think about running any command regarding networking I always grab a quick ifconfig so I understand what’s going on with the actual host:

ifconfig -a

Then I look at the networking within the VMs. Before you break stuff, you want to make users can get to it. After all, why else break things… Sticking with our xe command, we’re gonna’ take a peak at the vm-vif-list command, for a vm called badcompany:

/opt/xensource/bin/xe vm-vif-list vm-name=badcompany

To add a network interface, use the gif-create command along with a vm-uuid for the VM and a network-uuid for the interface. The NIC will be the virtual NIC:

/opt/xensource/bin/xe vif-create vm-uuid=BACHMAN-TURNER-OVERDRIVE network-uuid=AAAA-BBBBB-CCCCC-DDDDD device=2

Once created, “plug” your vif into a host using the vif-plug command:

/opt/xensource/bin/xe vif-plug uuid=BACHMAN-TURNER-OVERDRIVE

For the most part, the above vif-* commands can be replaced with pif. To see your physical interfaces:

/opt/xensource/bin/xe pif-list

To then the physical interfaces available to xe on another boxer:

/opt/xensource/bin/xe pif-scan host-uuid=BACHMAN-TURNER-OVERDRIVE

To forget an interface use pif-forget and a uuid argument which would be a pif from the list earlier:

/opt/xensource/bin/xe pif-forget uuid=_tmp1234567890

Pif-plug works the same as vif-plug within xe. Once an interface is installed then you can also set a number of arguments. These include autonegotiation, duplexing and speed, done using other-config:ethtool-autoneg=”off”, other-config:ethtool-speed=”1000” and other-config-duplex=”full” respectively, resulting in:

/opt/xensource/bin/xe pif-param-set uuid=BACHMAN-TURNER-OVERDRIVE other-config:ethtool-autoneg=”off” other-config:ethtool-speed=”1000” other-config-duplex=”full”

Once changed, check ethtool for the options we configured earlier.

One of the more annoying elements of networking with VMs is how to handle virtual switching. You can also enable openvswitch or bridge mode (use the built in network stack aka OVS or bridge connections respectively, likely more common). To do so, use the xe-switch-network-backend command along with either of these two. The first of the following commands the hypervisor into OVS and the second leverages bridging:

/opt/xensource/bin/xe-switch-network-backend openvswitch
/opt/xensource/bin/xe-switch-network-backend bridge

Shows you debugging information about what Xen is doing using the xen-debugtool command:

/opt/xensource/bin/xen-bugtool -yestoall

Managing Storage

You’ll also need to manage disks. While domUs are divided into PVs via LVM (PV=Physical Volume and LVM=Logical Volume). Each logical volume (LV) then gets a nice long UUID attached to it. Storage repositories can be local, fibre channeled, iSCSI, etc. Don’t try to mount them via carrier pigeon though. There’s latency there… Rather than use vgs to manage these volumes, use the sr* options within xe. We can start with sr-list, to see the storage available (sr is short for storage repository I assume, although it could be short for :

/opt/xensource/bin/xe sr-list

Now let’s say that you would like to take a drive you popped into /dev/scs0. There are a lot of options here. But let’s look at a basic incantation that’s going to create a volume called “Foreigner” with a type of lvm:

/opt/xensource/bin/xe sr-create name-label'"Foreigner" type=lvm device-config-device=/dev/scs0

The UUID is then output on success. In this case let’s say your sr uuid is BACHMAN-TURNER-OVERDRIVE. Re-run your sr-list to see if your storage is available. Now let’s say you change your mind and want something a little more modern, like a volume at /dev/scs1. You can delete Foreigner in favor of Black Keys:

/opt/xensource/bin/xe sr-forget uuid=BACHMAN-TURNER-OVERDRIVE
/opt/xensource/bin/xe sr-create name-label'"Black Keys" type=lvm device-config-device=/dev/scs0

iSCSI is likely to be one of the most common types of storage you’ll work with in a XenSource environment. The iscsiadm command can be used to discover, login and logout of storage. We’ll start with the discovery of storage. Here, we’ll use iscsiadm and set the mode using the -mode operator. The mode will be discovery. We’ll query for a -type of sendtargets and use a portal address of 10.0.0.1:

iscsiadm -mode discovery -type sendtargets -portal 10.0.0.1

Once you see the iSCSI LUN you want to mount, use iscsiadm to login to the node the LUN resides on. To do so, use iscsiadm in the node mode, along with a target name. The target is the IQN reported back from the previous command (let’s call it qn.0000.com.isilon:sn.123456) and then the same portal address from before along with the port and finally a -login operator

iscsiadm –mode node –targetname qn.0000.com.isilon:sn.123456 –portal 10.0.0.1:3260 –login

The pdb-list options are used to work with what the XenCenter GUI shows as LVM over iSCSI. Instead of having a type of lvm, we’re going to then work on a type of lvmoiscsi. This type has options of device-config as before, but now with a target of an IP address and additional options of an iSCSI target iQN. Let’s say we have a target IP of 10.0.0.1 and a IQN (given by your network admin usually unless you are that individual when you will be giving yourself an IQN which is very different than trying to give yourself… never mind, that was not going anywhere good) of iqn.0000.com.isilon:sn.123456

/opt/xensource/bin/xe sr-probe type=lvmoiscsi device-config:target=10.0.0.1 device-config:targetIQN=qn.0000.com.isilon:sn.123456

Once mounted, you should be able to see any UUIDs of iSCSI based LVMs using se-list:

/opt/xensource/bin/xe sr-list type=lvmoiscsi

You should also be able to use pdb-list to see any that are in /etc/iscsi/send_targets:

/opt/xensource/bin/xe pdb-list sr-uuid BACHMAN-TURNER-OVERDRIVE

Note: When using iSCSI, for HA you will need a dedicated Heartbeat Storage LUN for your pool. iscsiadm can be used to manage iSCSI itself.

Sometimes a disk will become detached, or not appear to Xen. When this happens, use pvdisplay to see if the disk is attached to the Xen server itself:

pvdisplay

Which outputs something as follows:

--- Physical volume ---
PV Name /dev/sda3
VG Name VG_XenStorage-AAAAAAA-BBBB-CCCC-DDDD-EEEEEEEEEE
PV Size 2.1 TB / not usable 8.25 MB
Allocatable yes
PE Size (KByte) 4096
Total PE 512254
Free PE 512254
Allocated PE 254
PV UUID uRowsc-slAI-33dG-0Cln-UtQ9-d7eb-uTx6aI

— Physical volume —
PV Name /dev/sdb1
VG Name VG_XenStorage-VVVVVV-WWWW-XXXX-YYYY-ZZZZZZZZZ
PV Size 2.1 TB / not usable 8.25 MB
Allocatable yes
PE Size (KByte) 4096
Total PE 512254
Free PE 512254
Allocated PE 254
PV UUID YmRdHu-tAbB-zz0C-V20t-2AKN-fV5Z-rRkGNa

Grab the string after VG_XenStorage-, and attempt to “introduce” the storage again:

/opt/xensource/bin/xe sr-introduce uuid=AAAAAAA-BBBB-CCCC-DDDD-EEEEEEEEEE type=iscsiolvm AAAAAAA-BBBB-CCCC-DDDD-EEEEEEEEEE

This makes the LUN appear in XenCenter as detached. You can then use pdb-create to attach the storage (which can be seen using ls -l on /dev/disk/by-id) to the UUID of a host (BACHMAN-TURNER-OVERDRIVE):

/opt/xensource/bin/xe pbd-create host-uuid=BACHMAN-TURNER-OVERDRIVE sr-uuid=AAAAAAA-BBBB-CCCC-DDDD-EEEEEEEEEE device-config:device=/dev/disk/by-id/scsi-SATA_SAMSUNG_HDZZZZZZZZZZZZZZ-part1
MMMMMMM-NNNNNNN-OOOOO-PPPP-QQQQQQQQQ

Note: Some mounting options for pdb-create: multihomed=true shared=true

Then “plug” it in using pdb-plug along with that last string in the above command:

/opt/xensource/bin/xe pdb-plug uuid=MMMMMMM-NNNNNNN-OOOOO-PPPP-QQQQQQQQQ

You can then see the storage using pdb-list, which outputs the following:

uuid ( RO) : MMMMMMM-NNNNNNN-OOOOO-PPPP-QQQQQQQQQ
host-uuid ( RO): BACHMAN-TURNER-OVERDRIVE
sr-uuid ( RO): AAAAAAA-BBBB-CCCC-DDDD-EEEEEEEEEE
device-config (MRO): device: /dev/disk/by-id/scsi-SATA_SAMSUNG_HDHDZZZZZZZZZZZZZZ-part1
MMMMMMM-NNNNNNN-OOOOO-PPPP-QQQQQQQQQ-part1
currently-attached ( RO): true

To detach, unplug using the same string:

/opt/xensource/bin/xe pbd-unplug uuid=MMMMMMM-NNNNNNN-OOOOO-PPPP-QQQQQQQQQ

You can then destroy the connection using sf-destroy and the same uuid:

/opt/xensource/bin/xe pdb-destroy uuid=MMMMMMM-NNNNNNN-OOOOO-PPPP-QQQQQQQQQ

Once “unplugged” forget the storage using sr-forget along with the same uuid:

/opt/xensource/bin/xe sr-forget uuid=MMMMMMM-NNNNNNN-OOOOO-PPPP-QQQQQQQQQ

Finally, if needed you can logout of the node using iscsiadm, which basically uses the same command used to login, but with the -logout operator at the end rather than -login:

iscsiadm –mode node –targetname qn.0000.com.isilon:sn.123456 –portal 10.0.0.1:3260 –logout

As with xm, xe supports using block tap disks. Support for TapDisks is provided in the form of tap-ctl. Use the list command to see your disks:

tap-ctl list

You can also ls against se-mounts:

ls -al /var/run/sr-mount

Managing A Virtual Machine

Now that you’ve got networking and some storage, let’s check out how to install a new OS. One thing to know is that since you don’t really have optical media on these physical hosts running hypervisors, you usually mount up iso’s to make them available to clients. To do so, drop an iso into either the local or shared repository for isos. The local is /opt/xensource/packages/iso. The shared is /var/opt/xen/iso_import. You can also use mount to fire up a cifs share and use that:

mount –t cifs //10.0.1.1/share /tmp/isos –o username=chuck,password=negron

You will need to restart xapi when you put isos into the repositories. To do so, run the service command, identifying the xapi service and the restart command for it:

service xapi restart

The easiest way to fire up a VM is going to be to use a template to do so. To see a list of templates available to XenSource, use the template-list command with xe:

/opt/xensource/bin/xe template-list

You can then list by defining a characteristic, such as name-label. As most admins in XenSource environments that I’ve seen use special characters, you’ll have to escape those out when listing by names. for example, use the \ character before any spaces as follows:

/opt/xensource/bin/xe template-list name-label=Windows\ Server\ 2003 params=all

You can also move a vm from one host to another. As with many tasks, we’ll start that one with a list command, in this case for sr:

/opt/xensource/bin/xe sr-list

Then use the vm-copy command within the xe environment to define the old VM, the new VM name and the uuid of the SR that houses the VM and optionally a new-name-description argument (which should be self-explanatory):

/opt/xensource/bin/xe vm-copy vm=3DogNight sr-uuid=NOT-BACHMAN-TURNER-OVERDRIVE new-name-label=NegronsPosse new-name-description=”Joy To The World”

Or to import from a vxa use the vm-import xe command:

/opt/xensource/bin/xe vm-import filename=/ToImport/70s.vxa

Then install it:

/opt/xensource/bin/xe vm-install template cat_stephens new-name-label=yusuf

To start the vm, then run vm-list, grab the name and then start/power it up by name. If my VM name is yusuf then the command to do so would be:

/opt/xensource/bin/xe vm-start vm yusuf

Once installed, the virtual machine should be set to automatically start if it’s a VM that should automatically start. To do so, use xe along with the vm-param-set command and identify the uuid that will automatically start (which we’ll call BACHMAN-TURNER-OVERDRIVE for giggles. You will also need to set the option other-config:auto_poweron to the boolean true state:

/opt/xensource/bin/xe vm-param-set uuid=BACHMAN-TURNER-OVERDRIVE other-config:auto_poweron=true

To delete a vm, just use the vm-destroy command along with the UUID:

/opt/xensource/bin/xe vm-destroy uuid=BACHMAN-TURNER-OVERDRIVE

Managing Pools

Find information about how a XenServer pool is configured, view the contents of the /etc/xensource/pool.conf file:

cat /etc/xensource/pool.conf

To configure a pool to automatically start, use xe along with the pool-param-set command. The uuid for the pool will need to be included and the other-config:auto_poweron option that we used with a vm earlier will also need to be set to true (it’s boolean):

/opt/xensource/bin/xe pool-param-set uuid=BACHMAN-TURNER-OVERDRIVE other-config:auto_poweron=true

You’ll find information about the master and slave pool members in the pool.conf. Armed with that information, there are a number of commands you can run to perform tasks on the pool. For example, you may need to remove a server from the pool. To do so, use the host-list option and then make note of the UUID of the host to remove:

/opt/xensource/bin/xe host-list

Then, run the pool-sync-database command to update the state.db file, which contains the virtual machine metadata for the pool and acts as the control domain:

/opt/xensource/bin/xe pool-sync-database

Next, run a host-forget for the bad UUID:

/opt/xensource/bin/xe host-forget uuid=BACHMAN-TURNER-OVERDRIVE

And finally run a pool-eject to get rid of the host you just forgot:

/opt/xensource/bin/xe pool-eject host-uuid

The pool is smart enough to pick a new master when needed. However, it typically won’t pick one just because, so you need to force the task at hand. To do so, first force the election of the new master. To do so, run xe followed by the pool-emergency-transition-to-master operator, as follows:

/opt/xensource/bin/xe pool-emergency-transition-to-master

Because it’s an emergency force, you then need to recover your slaves in the pool. To do so, run:

/opt/xensource/bin/xe pool-recover-slaves

And then check the list of hosts, noting any UUIDs where needed:

/opt/xensource/bin/xe host-list

Finally, resync the database for the pool:

/opt/xensource/bin/xe pool-sync database

And then remove the host you just failed the master from using host-forget, unless you’re using High Availability:

/opt/xensource/bin/xe host-forget

If you’re using High Availability, instead you might want to grab the UUIDs for any hosts that are not functioning in the pool and then run an xe se-list for each failing UUID to clean up the Xapi database:

/opt/xensource/bin/xe se-forget uuid=BACHMAN-TURNER-OVERDRIVE

If there are any hosts you can’t forget or remove from XenCenter then you might find that the power state on a VM must be forced down first. Keep in mind that Xen isn’t gonna’ want to let you do something you shouldn’t, here. So let’s check for virtual machines running using vm-list, resident on the UUID of the slave:

/opt/xensource/bin/xe vm-list resident-on=BACHMAN-TURNER-OVERDRIVE

Then let’s force down the UUID:

/opt/xensource/bin/xe vm-reset-powerstate uuid=BACHMAN-TURNER-OVERDRIVE –force

In XenCenter the bad host should then be gone.

Managing Services

The /etc/init.d domains several scripts used to start Xen (xend) and a variety of tools. One such tool is not XenSource. In the XenSource world, they think they know better than most of the rest of the entire *nix community. Therefore, they use xapi as their service name and use xe instead of xm as their binary (although honestly from my point of view it is nice that I quickly know what’s what running a quick ps or top). Xapi has a few nice attributes, such as the fact that they use SSL to securely connect to the server.

p2v-legacy

Backup

As mentioned earlier, the pool.db contains the metadata for the virtual machines as well as the configuration data for the system. Objects in the database, much as with all of the commands we’ve been running, are tracked by UUIDs. The database is mirrored to all slaves in a given pool. This allows masters and slaves to replace one another if need be. While synchronized between hosts, the database should still be backed up. Virtual machines are simple virtual disk images (VDIs) but the metadata is necessary to fully restore operations as well. Metadata and data are stored in a variety of locations. Therefore, there are a few different commands used to back up data.

Running backups is one of those places where I like to run the backup from another host. This allows me to easily just pull the file down locally. Let’s say I want to backup the pool database. That can be done with xe and the pool-dump-database command, using the file-name option followed by a path, -h with an IP or hostname of the server you’re pulling the pool from and -u and -pw which assumably mean username and password respectively. String it all together and it runs as follows:

/opt/xensource/bin/xe pool-dump-database file-name=/xenbak/pooldb -h 10.0.0.2 -u root -pw B0ston

We’ll continue on with that and grab the host backups from each server in the pool using the host-backup command and the same pretty much options:

/opt/xensource/bin/xe host-backup file-name=/xenbak/pooldb -h 10.0.0.2 -u root -pw B0ston

And then restoration of the host:

/opt/xensource/bin/xe host-restore file-name=/xenbak/pooldb -h 10.0.0.2 -u root -pw B0ston

When you restore data it’s just extracting backups and placing them into a partition and not overwriting running or current data. To restore VM metadata you’d then do an xe pool-database-restore command. When restoring a pool, you’ll also need to do a host-forget followed by a pool-join command. TO backup individual metadata on VMs use xe backup-metadata and xe restore-metadata or xe vm-export -metadata and xe vm-import -metadata. For example to import a VM using xe:

/opt/xensource/bin/xe vm-import filename=/tmp/cifsshare/myVM.xml force=true sr-uuid=BACHMAN-TURNER-OVERDRIVE preserve=true

Dealing with Snapshots

Once a snapshot is removed, you will need to use the coalesce leaf plugin to reclaim any space previously used by that snapshot. To do so, use the host-call-plugin command with xe, define the host-uuid and then call the coalesce-leaf plugin, defining the leaf-coalesce args for the vm_uuid:

/opt/xensource/bin/xe host-call-plugin host-uuid=AAAAA-BBBBB-CCCCC-DDDDD plugin=coalesce-leaf fn=leaf-coalesce args:vm_uuid=BACHMAN-TURNER-OVERDRIVE

Finding More Help

By the way, I never said I was very smart. Some of this crap is uber-dangerous. Ergo, the most important command xe has is help:

/opt/xensource/bin/xe help -s 127.0.0.1 --all

And so castles made of sand fall in the sea, eventually…