Show Line Numbers When Viewing A File

The nl command is used to show line numbers when viewing a file (unless you use the -b option along with an n, which seems to be one of the more pointless things to ever do at the command line, but then what do I know…). So if you’d like to see the line numbers for a file called nl The output would look like this: 1 # 2 #!/bin/bash 3 # 4 # Script Name: Or at least, that’s how I used to do it. For decades I never noticed that cat had a -b option. So if you’d like to use cat to see line numbers in your script, just run the command, along with a -b. cat -b Cat also allows output lines (-b skips lines with whitespace). Cat also has a squeeze option for sequential lines. Overall, not absolutely everything in nl, but enough that I doubt I’ll need to use nl much in the future.

Use Netstat To Locate What Process Is Using A Port

You’re installing software on some host. The installation goes well and then you go to access the information you need or connect to the service from another host. Wait, what’s that? Port is already in use? Crap. We’ve all been there. The quick and dirty answer: netstat. Let’s say you’re trying to use port 8080: netstat -tuln | grep 8080 Let’s say the response is httpd. OK, let’s see where that’s located using whereis: whereis httpd And what kind of file is httpd: file /usr/sbin/httpd Which responds with: /usr/sbin/httpd: Mach-O 64-bit executable x86_64 I guess we knew that since it had a port open, but what type of executable is this httpd you speak of, pray tell? whatis httpd httpd(8) – Apache Hypertext Transfer Protocol Server Apache2::Resource(3pm) – Limit resources used by httpd children CGI::Carp(3pm) – CGI routines for writing to the HTTPD httpd(8) – Apache Hypertext Transfer Protocol Server Oooohhhhh, I see now…

Sort ls Entries By Modification Times

I find that when I’m trying to quickly sort directory listings in a terminal screen, the last thing I want is to have to open . to get a Finder screen and then sort that way. Luckily, I’m not the only one who needs to sort data by time stamp from time to time. To do so using ls, simply use the -alt options: ls -alt ~/Desktop Or to see the information in reverse: ls -altr ~/Desktop

5 Ways To Manage Background Jobs In A Shell Environment

When running commands that are going to take awhile, I frequently start them with the nohup command, disown the command from the current session or queue them for later execution. The reason is that if I’m running them from a Terminal or SSH session and the session is broken I want to make sure they complete. To schedule a job for later execution, use at. For example, if I want to perform a simple command, I can schedule it in a minute by running it as an echo piped to at: echo "" | at now + 2 minutes Note, if using 1 minute, you’ll need that to be singular. But you can also disown the job. To do so, end a command with an & symbol. So, running a command or script that will take awhile with an ampersand at the end displays the job number for the command and then you can disown it by running disown followed by -h at the end. for example: du -d 0 & disown -h If you choose not to disown the job, you can check running jobs using the jobs command at any time: jobs Nohup runs a command or script in the background even after a shell has been stopped: nohup cvfsck -nv goldengirls & The above command runs the command between nohup and the & symbol in the background. By default, you’ll then have the output to the command run in the nohup.out file in your home directory. So if your username were krypted, you could tail the output using the following command: tail -f /Users/krypted/nohup.out You can also use screen and then reconnect to that screen. For example, use screen with a -t to create a new screen: screen -t sanconfigchange Then run a command: xsanctl sanConfigChanged Then later, reconnect to your screen: screen -x And you can control-n or control-a to scroll through running background processes this way, provided each is in its own screen. Finally, in AIX you can actually use the bg command. I used to really like this as I could basically move an existing job into the background if I’d already invoked it from a screen/session. For example, you have pid 88909 running and you want to put it into the background. You can just run bg 88909 and throw it into the background, allowing you to close a tty. But then if you’d like to look at it later, you can always pop it back using, you guessed it, fg. This only worked in AIX really, but is a great process management tool.

Scripting Azure On A Mac

Microsoft Azure is Microsoft’s cloud services. Azure can host virtual machines and act as a location to store files. However, Azure can do much more as well, providing an Active Directory instance, provide SQL database access, work with hosted Visual Studio, host web sites or provide BizTalk services. All of these can be managed at windows_azure_logo6 You can also manage Windows Azure from the command line on Linux, Windows or Mac. To download command line tools, visit Once downloaded, run the package installer. Screen Shot 2013-11-29 at 10.51.01 PMWhen the package is finished installing, visit /usr/local/bin where you’ll find the azure binary. Once installed, you’ll need to configure your account from the site to work with your computer. To do so, log into the portal. Screen Shot 2013-12-01 at 8.25.57 PM Once logged in, open Terminal and then use the azure command along with the account option and the download verb: azure account download This account downloads the .publishsettings file for the account you’re logged in as in your browser. Once downloaded, run azure with the account option and the import verb, dragging the path to your .publishsettings file from azure account import /Users/krypted/Downloads/WindowsAzure-credentials.publishsettings The account import then completes and your user is imported into azure. Once imported, run azure with the account option and then storage list: azure account storage list You might not have any storage configured yet, but at this point you should see the following to indicate that the account is working: info: No storage accounts defined info: account storage list command OK You can also run the azure command by itself to see some neat ascii-art (although the azure logo doesn’t really come through in this spiffy cut and paste job): info: _ _____ _ ___ ___________________ info:        /_\  |__ / | | | _ \ __| info: _ ___ / _ \__/ /| |_| |   / _|___ _ _ info: (___ /_/ \_\/___|\___/|_|_\___| _____) info: (_______ _ _) _ ______ _)_ _ info: (______________ _ ) (___ _ _) info: info: Windows Azure: Microsoft's Cloud Platform info: info: Tool version 0.7.4 help: help: Display help for a given command help: help [options] [command] help: help: Open the portal in a browser help: portal [options] help: help: Commands: help: account to manage your account information and publish settings help: config Commands to manage your local settings help: hdinsight Commands to manage your HDInsight accounts help: mobile Commands to manage your Mobile Services help: network Commands to manage your Networks help: sb Commands to manage your Service Bus configuration help: service Commands to manage your Cloud Services help: site Commands to manage your Web Sites help: sql Commands to manage your SQL Server accounts help: storage Commands to manage your Storage objects help: vm Commands to manage your Virtual Machines help: help: Options: help: -h, --help output usage information help: -v, --version output the application version Provided the account is working, you can then use the account, config, hdinsight, mobile, network, sb, service, site, sql, storage or vm options. Each of these can be invoked along with a -h option to show a help page. For example, to see a help page for service: azure service -h You can spin up resources including sites, storage containers and even virtual machines (although you might need to create templates for VMs first). As an example, let’s create a new site using the git template: azure site create --git Overall, there are a lot of options available in the azure command line interface. The web interface is very simple, with options in the command line interface mirroring the options in the web interface. Running and therefore scripting around these commands is straight forward. I wrote up some Amazon stuff previously at, but the azure controls are really full featured and I’m really becoming a huge fan of the service itself the more I use it (which likely means I’ll post more articles on it soon).

Correcting SR_BACKEND_FAILURE_46 Error in XenSource

If you try to start a VM in XenSource and you see an error that SR_BACKEND_FAILURE_46 then one way to resolve is to detach the storage and reattach while forgetting and resetting the VM’s uuid. To do so, first locate the VM in question and go to the Storage tab to find the SR. Click Properties and add _temp to the end of the name so that you can easily find it to reattach. Then click on the Detach button for the storage. From the Console for the host, run the following (assuming the VM is called JACKED): xe vdi-list name-label= “JACKED_temp” Note the VDI UUID and SR UUID, which we’ll call $VDI_UUID and $SR_UUID respectively. THen run these commands: xe vdi-forget uuid=$VDI_UUID xe sr-scan uuid=$SR_UUID xe vdi-param-set name-label=JACKED uuid=$VDI_UUID Open XenCenter and go to the VM > Storage tab and click Attach. Choose JACKED_temp and then click Attach Disk. Finally, choose the VDI that you were working on and click Start. The VM should fire up now. Good luck!

ESX: Configure Virtual Machine's To Startup Automatically

You shouldn’t have to reboot your ESX servers very often. But when you do, you might want the virtual machines to start up automatically. To configure a virtual machine to start up (or shut down) automatically select the host and click on the Configuration tab. Then click on virtual machine Startup/Shutdown and click on Properties, selecting “Allow virtual machines to start and stop automatically with the system”. As I mentioned in a previous article, you can also configure the operating system to start after a brief delay by providing a Default Startup Delay time, allowing time for booting systems to run scripts or to throw them into Safe Mode. You can also configure automated shutdown options at this screen as well. Also use the Move Up and Move Down options to indicate what order virtual machines start when the system starts. Click Save and the configs are written to the system.

List All DNS Records For A Domain

Sometimes you want to move a domain but you don’t have a copy of the zone file in order to recreate records. The easy way to do this is to grab a zone transfer. To do so, dig is your friend: dig -tAXFR Sometimes though (and actually more often than not) a zone transfer is disabled. In that case you’ll need to dig the domain a bit differently. I like to use +nocmd, query for any and list the results (+answer): dig +nocmd any +answer Which results in the following:
;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 39183 ;; flags: qr rd ra; QUERY: 1, ANSWER: 6, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ; IN ANY ;; ANSWER SECTION: 1262 IN A 3600 IN MX 0 3600 IN MX 10 3600 IN NS 3600 IN NS 3600 IN SOA 2010010400 28800 7200 604800 3600 ;; Query time: 127 msec ;; SERVER: ;; WHEN: Tue May 7 22:31:15 2013 ;; MSG SIZE rcvd: 207
The above shows the naked domain name entry (yes, I still giggle every time I write the word naked so it’s ok if you giggled when you read it), all of the mail (which btw I don’t actually use that mail so please don’t try and send any at this time) and the ns servers. Now, the serial and refresh information isn’t included in this output. Actually, it is but it might not make sense, so we’ll just add the +multiline option which will make this look strangely like a zone file: dig +nocmd any +multiline +answer Notice the serial, refresh, retry, expire and minimum options are now listed in a much more fashionable way:
;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 10965 ;; flags: qr rd ra; QUERY: 1, ANSWER: 6, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ; IN ANY ;; ANSWER SECTION: 3225 IN A 3225 IN MX 0 3225 IN MX 10 3225 IN NS 3225 IN NS 3225 IN SOA ( 2010010400 ; serial 28800 ; refresh (8 hours) 7200 ; retry (2 hours) 604800 ; expire (1 week) 3600 ; minimum (1 hour) ) ;; Query time: 22 msec ;; SERVER: ;; WHEN: Tue May 7 22:32:20 2013 ;; MSG SIZE rcvd: 207
And there ya’ go. You’ve basically done a zone transfer on a box, even though zone transfers are disabled. Silly DNS admins, disabling zone transfers and all that… Yes, I disable zone transfers on most of my DNS boxen as well, or at least only allow them for specific IPs… 😉

Using the XenSource Command Line Interface

XenSource has some pretty good GUI tools. There’s XenCenter and the xsconsole, both of which are pretty adequate in a free sense and get pretty darn interesting when you actually pay Citrix. But today I want to take a little look under the hood of XenSource. I had previously written about Xen. But note that this is a different beast. Before I get started talking about how to do some tasks in XenSource, I first want to throw out there a few terms. The first, is virtual machine. This is exactly what it sounds like, an operating system that runs on a virtual host rather than a physical hosts. So take that Dell PowerEdge that’s about 9 years old that you just use to access your Golden Girls collection on the NT server you used napster to steal it on and virtualize it. Sweet, now your mp3s can still run too! The next term is hypervisor. This is just a really, really dumb physical machine that is super beefy and which is going to run some VMs for ya’. I like to think of a Hypervisor like the old pictures of the Governator back when Arnold weightlifted. He’s the hypervisor. They’re the weights. Illegal instructions is what happened when he made ill-fated attempts to meet ladies (I apologize in advance for any ill fated attempts at humor, it’s late…).


So the hypervisor loads these virtual machines. Those are the basic virtualization terms that most people likely know. But the term domain is probably different for many. This is the context in which a VM runs. Think of it kinda’ like reserved resources (I use the term reserved very loosely here). To see your domains, we’ll look at our first command. Many of the XenSource commands sit in /opt/xensource/bin (so I’ll include the dirname of commands in this article). In here, you’ll find list_domains which interestingly shows you domains (e.g. dom0, dom1, dom2, etc): /opt/xensource/bin/list_domains Note: In this article I use the default location for commands. You should too. If you put the binaries somewhere else then there’s a chance you aren’t bright. Unless you have a good reason. Then you’re definitely not bright ’cause you’ve clearly invoked something I call “xen stoner logic.” It is however, possible to resolve this issue: stop doing drugs. If you’ve got a lot of domains running, you can grep out the list for a given UUID. Let’s say that I have a UUID of BACHMAN-TURNER-OVERDRIVE. I’m going to use this UUID throughout the article as my example UUID. Any time you see this, replace it with your hex-based UUID. If you note, my UUID example is not hex based. This is something you must overlook because I’m writing this article for me and while I will be on to other bands in my iTunes playlist by the end of this article, that’s where I am now… Anyway, you take the UUID (also referred to as the dom-id of the VM ) and use list_domains, then pipe out your output to grep with a first position of the UUID after the grep, as follows: /opt/xensource/bin/list_domains | grep BACHMAN-TURNER-OVERDRIVE Now, when I referred to the “context” of the virtual machine, think about this. The process of the VM is basically the dom. You can transfer that process to another host in a pool, which is basically a cluster. You can also end up having a failed shut down of a virtual machine and end up with something in an unresponsive state. So to restart your domains, use the id of the bad domain and /opt/xensource/bin/destroy_domain –dom-id 0

Remote Connectivity

Pretty much, each of the commands we’re about to use can be bookended with options that allow for remote connectivity. We’re basically running these locally using the xe command. Any time you use xe, you can throw a -s -u -pw option before the commands we go through, defining server username and password for that username respectively. For example, a very basic command that will show you optical media and ISOs available to your host is xe cd-list. We’re going to take that command and run it against a remote server: /opt/xensource/bin/xe -s -u root -pw B0ston cd-list The options can go in front of or behind most of the commands or arguments. I find that I end up using screen a lot more than using all this, if anyone cares… Except backup, as I’ll explain in that section of this article.

Managing Tasks

Sometimes, you have a task stuck or sucking up too many resources. This can cause a VM to become completely unresponsive (like, can’t ping it) or otherwise run very poorly. You see a pattern across multiple doms on the same host and so decide to check to see if XenSource has any tasks eating up resources (like a backup). To see the task list for a given box, check the task-list. The xe task-list command will show you what tasks Xen is doing: /opt/xensource/bin/xe task-list You can then stop a task if it doesn’t stoke you out, using the task-cancel command: /opt/xensource/bin/xe task-cancel uuid=BACHMAN-TURNER-OVERDRIVE Now, not all tasks cancel gracefully (just ask anyone who’s tried taking candy from a toddler in mid-munch), so the task-cancel (and most other commands in XenSource) comes with a nifty force=true option: /opt/xensource/bin/xe task-cancel force=true uuid=BACHMAN-TURNER-OVERDRIVE The xe task-list command doesn’t show you what VMs are running per se, so I find that xentop is a useful command for checking on VMs, helping to show when one is spinning out of control or aggregate information about VMs. Sure, you can see this information with XenCenter but you don’t seriously trust GUIs do you?!?! Especially not one that can’t connect to the open source versions of Xen!!! Anyway, xentop is like top but for Xen (go figure, right?!?!). Xentop comes with a few options and interactive commands when in the xentop interface. The options are:
  • -h: See a help page
  • -i: I always use this one these days as I may otherwise forget to kill my session, which just takes up resources. The -i indicates how many iterations before the interactive environment ends.
  • -d: Delay, waits a few more or less seconds between refresh (the default value here is 3 seconds). The interactive command for this while xentop is running is D.
  • -V: Shows the version of xentop. Can be enabled during interactive mode using the V key.
  • -n: Includes information about networks. This can be turned on while xentop is running by holding down the N key.
  • -x: includes block data.
  • -r: This is useful if you have a lot of domains or a lot of wrapping as it repeats the header for each.
  • S: Changes the order to sort to the next tab/header
  • Q: The interactive key to stop xentop
  • Arrow: Scrolls the screen

xentop is one of the more common commands I run as I also use it to find some pretty basic information. Given the above options, to fire it up to refresh every 1 second for 5 minutes, show network information, repeat headers and show block data:

/opt/xensource/bin/xentop -i 300 -d 1 -n -x -r

Managing Virtual Machines

If you have a domain that goes unresponsive but shows running in xentop, once you’ve confirmed that it’s an issue with the VM itself, go ahead and try to restart it: /opt/xensource/bin/xe vm-reboot vm-name=bachman You can also reboot based on UUID of the VM (you don’t really trust the translation of names to addresses without a static hosts file do you?!?!): /opt/xensource/bin/xe vm-reboot vm-id=vm_UUID=BACHMAN-TURNER-OVERDRIVE Or just reset the power state, like when a dom still exists for a VM called badcompany: /opt/xensource/bin/xe vm-reset-powerstate vm=badcompany --force You can also fire up a nice interactive DOS-style GUI. To do so, run xsconsole, which can be run via ssh and is uber-fast. The other thing I like about xsconsole is you can see things like the serial number of the computer, networking information, pool information and most other things that can be queried from xe: /opt/xensource/bin/xsconsole

Managing Networking Within Xen

Every time I think about running any command regarding networking I always grab a quick ifconfig so I understand what’s going on with the actual host: ifconfig -a Then I look at the networking within the VMs. Before you break stuff, you want to make users can get to it. After all, why else break things… Sticking with our xe command, we’re gonna’ take a peak at the vm-vif-list command, for a vm called badcompany: /opt/xensource/bin/xe vm-vif-list vm-name=badcompany To add a network interface, use the gif-create command along with a vm-uuid for the VM and a network-uuid for the interface. The NIC will be the virtual NIC: /opt/xensource/bin/xe vif-create vm-uuid=BACHMAN-TURNER-OVERDRIVE network-uuid=AAAA-BBBBB-CCCCC-DDDDD device=2 Once created, “plug” your vif into a host using the vif-plug command: /opt/xensource/bin/xe vif-plug uuid=BACHMAN-TURNER-OVERDRIVE For the most part, the above vif-* commands can be replaced with pif. To see your physical interfaces: /opt/xensource/bin/xe pif-list To then the physical interfaces available to xe on another boxer: /opt/xensource/bin/xe pif-scan host-uuid=BACHMAN-TURNER-OVERDRIVE To forget an interface use pif-forget and a uuid argument which would be a pif from the list earlier: /opt/xensource/bin/xe pif-forget uuid=_tmp1234567890 Pif-plug works the same as vif-plug within xe. Once an interface is installed then you can also set a number of arguments. These include autonegotiation, duplexing and speed, done using other-config:ethtool-autoneg=”off”, other-config:ethtool-speed=”1000” and other-config-duplex=”full” respectively, resulting in: /opt/xensource/bin/xe pif-param-set uuid=BACHMAN-TURNER-OVERDRIVE other-config:ethtool-autoneg=”off” other-config:ethtool-speed=”1000” other-config-duplex=”full” Once changed, check ethtool for the options we configured earlier. One of the more annoying elements of networking with VMs is how to handle virtual switching. You can also enable openvswitch or bridge mode (use the built in network stack aka OVS or bridge connections respectively, likely more common). To do so, use the xe-switch-network-backend command along with either of these two. The first of the following commands the hypervisor into OVS and the second leverages bridging: /opt/xensource/bin/xe-switch-network-backend openvswitch /opt/xensource/bin/xe-switch-network-backend bridge Shows you debugging information about what Xen is doing using the xen-debugtool command: /opt/xensource/bin/xen-bugtool -yestoall

Managing Storage

You’ll also need to manage disks. While domUs are divided into PVs via LVM (PV=Physical Volume and LVM=Logical Volume). Each logical volume (LV) then gets a nice long UUID attached to it. Storage repositories can be local, fibre channeled, iSCSI, etc. Don’t try to mount them via carrier pigeon though. There’s latency there… Rather than use vgs to manage these volumes, use the sr* options within xe. We can start with sr-list, to see the storage available (sr is short for storage repository I assume, although it could be short for : /opt/xensource/bin/xe sr-list Now let’s say that you would like to take a drive you popped into /dev/scs0. There are a lot of options here. But let’s look at a basic incantation that’s going to create a volume called “Foreigner” with a type of lvm: /opt/xensource/bin/xe sr-create name-label'"Foreigner" type=lvm device-config-device=/dev/scs0 The UUID is then output on success. In this case let’s say your sr uuid is BACHMAN-TURNER-OVERDRIVE. Re-run your sr-list to see if your storage is available. Now let’s say you change your mind and want something a little more modern, like a volume at /dev/scs1. You can delete Foreigner in favor of Black Keys: /opt/xensource/bin/xe sr-forget uuid=BACHMAN-TURNER-OVERDRIVE /opt/xensource/bin/xe sr-create name-label'"Black Keys" type=lvm device-config-device=/dev/scs0 iSCSI is likely to be one of the most common types of storage you’ll work with in a XenSource environment. The iscsiadm command can be used to discover, login and logout of storage. We’ll start with the discovery of storage. Here, we’ll use iscsiadm and set the mode using the -mode operator. The mode will be discovery. We’ll query for a -type of sendtargets and use a portal address of iscsiadm -mode discovery -type sendtargets -portal Once you see the iSCSI LUN you want to mount, use iscsiadm to login to the node the LUN resides on. To do so, use iscsiadm in the node mode, along with a target name. The target is the IQN reported back from the previous command (let’s call it and then the same portal address from before along with the port and finally a -login operator iscsiadm –mode node –targetname –portal –login The pdb-list options are used to work with what the XenCenter GUI shows as LVM over iSCSI. Instead of having a type of lvm, we’re going to then work on a type of lvmoiscsi. This type has options of device-config as before, but now with a target of an IP address and additional options of an iSCSI target iQN. Let’s say we have a target IP of and a IQN (given by your network admin usually unless you are that individual when you will be giving yourself an IQN which is very different than trying to give yourself… never mind, that was not going anywhere good) of /opt/xensource/bin/xe sr-probe type=lvmoiscsi device-config:target= Once mounted, you should be able to see any UUIDs of iSCSI based LVMs using se-list: /opt/xensource/bin/xe sr-list type=lvmoiscsi You should also be able to use pdb-list to see any that are in /etc/iscsi/send_targets: /opt/xensource/bin/xe pdb-list sr-uuid BACHMAN-TURNER-OVERDRIVE Note: When using iSCSI, for HA you will need a dedicated Heartbeat Storage LUN for your pool. iscsiadm can be used to manage iSCSI itself. Sometimes a disk will become detached, or not appear to Xen. When this happens, use pvdisplay to see if the disk is attached to the Xen server itself: pvdisplay Which outputs something as follows: --- Physical volume --- PV Name /dev/sda3 VG Name VG_XenStorage-AAAAAAA-BBBB-CCCC-DDDD-EEEEEEEEEE PV Size 2.1 TB / not usable 8.25 MB Allocatable yes PE Size (KByte) 4096 Total PE 512254 Free PE 512254 Allocated PE 254 PV UUID uRowsc-slAI-33dG-0Cln-UtQ9-d7eb-uTx6aI — Physical volume — PV Name /dev/sdb1 VG Name VG_XenStorage-VVVVVV-WWWW-XXXX-YYYY-ZZZZZZZZZ PV Size 2.1 TB / not usable 8.25 MB Allocatable yes PE Size (KByte) 4096 Total PE 512254 Free PE 512254 Allocated PE 254 PV UUID YmRdHu-tAbB-zz0C-V20t-2AKN-fV5Z-rRkGNa Grab the string after VG_XenStorage-, and attempt to “introduce” the storage again: /opt/xensource/bin/xe sr-introduce uuid=AAAAAAA-BBBB-CCCC-DDDD-EEEEEEEEEE type=iscsiolvm AAAAAAA-BBBB-CCCC-DDDD-EEEEEEEEEE This makes the LUN appear in XenCenter as detached. You can then use pdb-create to attach the storage (which can be seen using ls -l on /dev/disk/by-id) to the UUID of a host (BACHMAN-TURNER-OVERDRIVE): /opt/xensource/bin/xe pbd-create host-uuid=BACHMAN-TURNER-OVERDRIVE sr-uuid=AAAAAAA-BBBB-CCCC-DDDD-EEEEEEEEEE device-config:device=/dev/disk/by-id/scsi-SATA_SAMSUNG_HDZZZZZZZZZZZZZZ-part1 MMMMMMM-NNNNNNN-OOOOO-PPPP-QQQQQQQQQ Note: Some mounting options for pdb-create: multihomed=true shared=true Then “plug” it in using pdb-plug along with that last string in the above command: /opt/xensource/bin/xe pdb-plug uuid=MMMMMMM-NNNNNNN-OOOOO-PPPP-QQQQQQQQQ You can then see the storage using pdb-list, which outputs the following: uuid ( RO) : MMMMMMM-NNNNNNN-OOOOO-PPPP-QQQQQQQQQ host-uuid ( RO): BACHMAN-TURNER-OVERDRIVE sr-uuid ( RO): AAAAAAA-BBBB-CCCC-DDDD-EEEEEEEEEE device-config (MRO): device: /dev/disk/by-id/scsi-SATA_SAMSUNG_HDHDZZZZZZZZZZZZZZ-part1 MMMMMMM-NNNNNNN-OOOOO-PPPP-QQQQQQQQQ-part1 currently-attached ( RO): true To detach, unplug using the same string: /opt/xensource/bin/xe pbd-unplug uuid=MMMMMMM-NNNNNNN-OOOOO-PPPP-QQQQQQQQQ You can then destroy the connection using sf-destroy and the same uuid: /opt/xensource/bin/xe pdb-destroy uuid=MMMMMMM-NNNNNNN-OOOOO-PPPP-QQQQQQQQQ Once “unplugged” forget the storage using sr-forget along with the same uuid: /opt/xensource/bin/xe sr-forget uuid=MMMMMMM-NNNNNNN-OOOOO-PPPP-QQQQQQQQQ Finally, if needed you can logout of the node using iscsiadm, which basically uses the same command used to login, but with the -logout operator at the end rather than -login: iscsiadm –mode node –targetname –portal –logout As with xm, xe supports using block tap disks. Support for TapDisks is provided in the form of tap-ctl. Use the list command to see your disks: tap-ctl list You can also ls against se-mounts: ls -al /var/run/sr-mount

Managing A Virtual Machine

Now that you’ve got networking and some storage, let’s check out how to install a new OS. One thing to know is that since you don’t really have optical media on these physical hosts running hypervisors, you usually mount up iso’s to make them available to clients. To do so, drop an iso into either the local or shared repository for isos. The local is /opt/xensource/packages/iso. The shared is /var/opt/xen/iso_import. You can also use mount to fire up a cifs share and use that: mount –t cifs // /tmp/isos –o username=chuck,password=negron You will need to restart xapi when you put isos into the repositories. To do so, run the service command, identifying the xapi service and the restart command for it: service xapi restart The easiest way to fire up a VM is going to be to use a template to do so. To see a list of templates available to XenSource, use the template-list command with xe: /opt/xensource/bin/xe template-list You can then list by defining a characteristic, such as name-label. As most admins in XenSource environments that I’ve seen use special characters, you’ll have to escape those out when listing by names. for example, use the character before any spaces as follows: /opt/xensource/bin/xe template-list name-label=Windows Server 2003 params=all You can also move a vm from one host to another. As with many tasks, we’ll start that one with a list command, in this case for sr: /opt/xensource/bin/xe sr-list Then use the vm-copy command within the xe environment to define the old VM, the new VM name and the uuid of the SR that houses the VM and optionally a new-name-description argument (which should be self-explanatory): /opt/xensource/bin/xe vm-copy vm=3DogNight sr-uuid=NOT-BACHMAN-TURNER-OVERDRIVE new-name-label=NegronsPosse new-name-description=”Joy To The World” Or to import from a vxa use the vm-import xe command: /opt/xensource/bin/xe vm-import filename=/ToImport/70s.vxa Then install it: /opt/xensource/bin/xe vm-install template cat_stephens new-name-label=yusuf To start the vm, then run vm-list, grab the name and then start/power it up by name. If my VM name is yusuf then the command to do so would be: /opt/xensource/bin/xe vm-start vm yusuf Once installed, the virtual machine should be set to automatically start if it’s a VM that should automatically start. To do so, use xe along with the vm-param-set command and identify the uuid that will automatically start (which we’ll call BACHMAN-TURNER-OVERDRIVE for giggles. You will also need to set the option other-config:auto_poweron to the boolean true state: /opt/xensource/bin/xe vm-param-set uuid=BACHMAN-TURNER-OVERDRIVE other-config:auto_poweron=true To delete a vm, just use the vm-destroy command along with the UUID: /opt/xensource/bin/xe vm-destroy uuid=BACHMAN-TURNER-OVERDRIVE To show the domains for a given label use vm-list (assuming the label is still  yusuf): /opt/xensource/bin/xe vm-list name-label=yusuf To then force reboot, use vm-reboot along with the force option (still using yusuf): /opt/xensource/bin/xe vm-reboot force=true vm=yusuf And to power down instead of reboot: /opt/xensource/bin/xe vm-reset-powerstate vm=yusuf

Managing Pools

Find information about how a XenServer pool is configured, view the contents of the /etc/xensource/pool.conf file: cat /etc/xensource/pool.conf To configure a pool to automatically start, use xe along with the pool-param-set command. The uuid for the pool will need to be included and the other-config:auto_poweron option that we used with a vm earlier will also need to be set to true (it’s boolean): /opt/xensource/bin/xe pool-param-set uuid=BACHMAN-TURNER-OVERDRIVE other-config:auto_poweron=true You’ll find information about the master and slave pool members in the pool.conf. Armed with that information, there are a number of commands you can run to perform tasks on the pool. For example, you may need to remove a server from the pool. To do so, use the host-list option and then make note of the UUID of the host to remove: /opt/xensource/bin/xe host-list Then, run the pool-sync-database command to update the state.db file, which contains the virtual machine metadata for the pool and acts as the control domain: /opt/xensource/bin/xe pool-sync-database Next, run a host-forget for the bad UUID: /opt/xensource/bin/xe host-forget uuid=BACHMAN-TURNER-OVERDRIVE And finally run a pool-eject to get rid of the host you just forgot: /opt/xensource/bin/xe pool-eject host-uuid The pool is smart enough to pick a new master when needed. However, it typically won’t pick one just because, so you need to force the task at hand. To do so, first force the election of the new master. To do so, run xe followed by the pool-emergency-transition-to-master operator, as follows: /opt/xensource/bin/xe pool-emergency-transition-to-master Because it’s an emergency force, you then need to recover your slaves in the pool. To do so, run: /opt/xensource/bin/xe pool-recover-slaves And then check the list of hosts, noting any UUIDs where needed: /opt/xensource/bin/xe host-list Finally, resync the database for the pool: /opt/xensource/bin/xe pool-sync database And then remove the host you just failed the master from using host-forget, unless you’re using High Availability: /opt/xensource/bin/xe host-forget If you’re using High Availability, instead you might want to grab the UUIDs for any hosts that are not functioning in the pool and then run an xe se-list for each failing UUID to clean up the Xapi database: /opt/xensource/bin/xe se-forget uuid=BACHMAN-TURNER-OVERDRIVE If there are any hosts you can’t forget or remove from XenCenter then you might find that the power state on a VM must be forced down first. Keep in mind that Xen isn’t gonna’ want to let you do something you shouldn’t, here. So let’s check for virtual machines running using vm-list, resident on the UUID of the slave: /opt/xensource/bin/xe vm-list resident-on=BACHMAN-TURNER-OVERDRIVE Then let’s force down the UUID: /opt/xensource/bin/xe vm-reset-powerstate uuid=BACHMAN-TURNER-OVERDRIVE –force In XenCenter the bad host should then be gone.

Managing Services

The /etc/init.d domains several scripts used to start Xen (xend) and a variety of tools. One such tool is not XenSource. In the XenSource world, they think they know better than most of the rest of the entire *nix community. Therefore, they use xapi as their service name and use xe instead of xm as their binary (although honestly from my point of view it is nice that I quickly know what’s what running a quick ps or top). Xapi has a few nice attributes, such as the fact that they use SSL to securely connect to the server. p2v-legacy


As mentioned earlier, the pool.db contains the metadata for the virtual machines as well as the configuration data for the system. Objects in the database, much as with all of the commands we’ve been running, are tracked by UUIDs. The database is mirrored to all slaves in a given pool. This allows masters and slaves to replace one another if need be. While synchronized between hosts, the database should still be backed up. Virtual machines are simple virtual disk images (VDIs) but the metadata is necessary to fully restore operations as well. Metadata and data are stored in a variety of locations. Therefore, there are a few different commands used to back up data. Running backups is one of those places where I like to run the backup from another host. This allows me to easily just pull the file down locally. Let’s say I want to backup the pool database. That can be done with xe and the pool-dump-database command, using the file-name option followed by a path, -h with an IP or hostname of the server you’re pulling the pool from and -u and -pw which assumably mean username and password respectively. String it all together and it runs as follows: /opt/xensource/bin/xe pool-dump-database file-name=/xenbak/pooldb -h -u root -pw B0ston We’ll continue on with that and grab the host backups from each server in the pool using the host-backup command and the same pretty much options: /opt/xensource/bin/xe host-backup file-name=/xenbak/pooldb -h -u root -pw B0ston And then restoration of the host: /opt/xensource/bin/xe host-restore file-name=/xenbak/pooldb -h -u root -pw B0ston When you restore data it’s just extracting backups and placing them into a partition and not overwriting running or current data. To restore VM metadata you’d then do an xe pool-database-restore command. When restoring a pool, you’ll also need to do a host-forget followed by a pool-join command. TO backup individual metadata on VMs use xe backup-metadata and xe restore-metadata or xe vm-export -metadata and xe vm-import -metadata. For example to import a VM using xe: /opt/xensource/bin/xe vm-import filename=/tmp/cifsshare/myVM.xml force=true sr-uuid=BACHMAN-TURNER-OVERDRIVE preserve=true

Dealing with Snapshots

Once a snapshot is removed, you will need to use the coalesce leaf plugin to reclaim any space previously used by that snapshot. To do so, use the host-call-plugin command with xe, define the host-uuid and then call the coalesce-leaf plugin, defining the leaf-coalesce args for the vm_uuid: /opt/xensource/bin/xe host-call-plugin host-uuid=AAAAA-BBBBB-CCCCC-DDDDD plugin=coalesce-leaf fn=leaf-coalesce args:vm_uuid=BACHMAN-TURNER-OVERDRIVE

Finding More Help

By the way, I never said I was very smart. Some of this crap is uber-dangerous. Ergo, the most important command xe has is help: /opt/xensource/bin/xe help -s --all And so castles made of sand fall in the sea, eventually…

Some Basic Xen Commands

The most important command for managing pretty much anything in Linux is vi. So if you only learn one command, learn that one. But if you want to learn another, the second most important command for managing Xen is then xm (well, once you’ve apt-gotten or yummied up the installation that is). The xm command has a number of easy verbs, each used for managing the Xen environment.
  • xm info – Shows information about the Xen host
  • xm list – Shows information about doms (states include r for running, b for blocked, c for crashed, p for paused and the worse, d for dying).
  • xm network-list – Shows virtual interfaces for doms
  • xm log – Shows information from the Xen logs
  • xm reboot – Reboots a VM
  • xm vcpu-list – Shows dom virtual processors
  • xm top – Shows hosts and domains similar to how top works in *nix
  • xm uptime – Shows uptime
  • xm dmesg – Shows the send message buffer
  • xm create – Create a node called
  • xm console – Switch to that new node
  • xm destroy – Deletes that newly created node
  • xm shell – Invoke an interactive shell environment of your xend
  • xm shutdown – Turn off a VM
  • xm pause – Rather than shut the VM down, just pause it (starts back up much faster), but if the host is rebooted then state is lost (otherwise use suspend)
  • xm suspend – Suspends a VM, which writes the data to disk, so changes wouldn’t be lost on restart.
  • xm rename – Rename installed VMs
  • xm resume – If a VM is paused, fire it up
  • xm save  – Similar to suspend except with user definable state file
  • xm restore – Similar to resume except restoreable with exports that used the save verb
  • xm dump-core – Dumps core per domain
  • xm sysrq – Sends system requests per domain
  • xm block-list – Lists block devices per domain
  • xm mem-max – Configure the maximum memory for a domain
  • xm mem-set – Configure the current memory allowance for a domain
  • xm vcpu-set – Configure active processors for a domain
  • xm migrate – Move a domain to another server (e.g. using the -l operator to do so live)
Virt-manager and virt-install can be used to manage and create virtual machines for use with Xen. Virsh can also be of assistance:
  • virsh nodeinfo – Shows information about each node
  • virsh vcpuinfo – Shows information about virtual processors
  • virsh dominfo – Shows information about domains
  • virsh dumpxml  – Dumps the same information just in parseable XML