Automated Regression Testing Mac OS X Clients Using Sikuli

Imaging can be a complicated task. Many imaging environments have a lot of scripts, packages, base images and other aspects of automation. The more of these that you have, the more potential combinations you have for the state of a system once they’ve been run. This gets complicated when you want to make sure that each possible combination of images will have a consistent result when installed. For example, take something simple, like a property list. Each possible combination of packages, scripts images and even managed preferences might have a different impact on that poor property list. A simple defaults command can often give administrators the ability to see what settings have been applied to a system. For simple tasks that can be programatically trapped for, this might be enough. You can argue that everything can be programatically discovered, but I would argue that is an immature way to look at such things. The example I used in the Enterprise Mac Administrator’s Guide was looking at Microsoft Word. Let’s say that you install 24 fonts. When you open Word, you end up needing to check and make sure that each font loads. You can verify fonts test clean, but you can’t verify that they show up in Word. Microsoft gives you the ability to list fonts that appear in the font menu programatically. But you would literally need to open Word and look in the fonts menu on each regression of each system image in order to tell if the fonts actually appear in the list and if so, whether they look right. One small example. In many environments, each application will have 10 or more tests, with a lot of applications potentially being deployed – and the list grows as failures are discovered year over year… Now let’s say you’ve got 80 packages in your self service library. At this point, in order to tell that every combination is cool, you’d need to manually open Word on a lot of image combinations in order to verify that your fonts load properly. EggPlant is a tool that enables administrators to run a script that clicks on things for us and reports back on what happens when we do. Now, AppleScript can do some of this as well. But eggPlant gets very in depth in terms of scripting logic and is cross platform. EggPlant leverages a Vine server that gets installed on clients. Screen shots are then taken through a Vine client and compared to screen shots you make and save in eggPlant to compare to the ones that are captured from Vine. That’s a bit of overs implication, but at the end of the day, bolt some simple scripting on top of that and you’ve got a pretty advanced solution. EggPlant now also supports iOS by jail breaking devices to load the Vine server. EggPlant is one of the best tools for this kind of thing for the Mac platform. It’s more popular amongst those who do a lot of web testing, but it shoehorns nicely into post-flight imaging regression testing. I had a good conversation with @tvsutton at the Penn State Mac Admins conference about eggPlant and he mentioned a tool called Sikuli that I hadn’t used before, so hat tip to him for the rest of this article. While eggPlant has way more logic in it, for simple regression testing, you can leverage Sikuki in its place, which wouldn’t cause some of the potential conflicts that running a localized Vine server might cause on client systems. Sikuli is a simple tool that allows administrators to script graphical events. You can effectively automate anything you see on a screen using the same type of screen shots; however, with Sikuli, you dump the screen shots directly into the script via the scripting interface, known as the Sikuli IDE. The first step to using Sikuli is to download the IDE from their site and run the installer. Once installed, open Sikuli. The options along the left side are some common commands that can be run, in serial from top to bottom. Not all of the possible methods are exposed in the GUI sidebars, but many of the more common ones are. Let’s go ahead and manually enter switchApp followed by an application you’d like to open with your script. In my example, I’m going to use switchApp("") While the command says switchApp, it might as well say openApp, as the application will open when you run the command. Go ahead and see for yourself (assuming you have a Lion Server and can therefore open Now, let’s look at some graphical interfacing, go ahead and click on click in the sidebar, which will dim the screen and bring up crosshairs for taking a screenshot. Here, select some part of the screen. I’m using Users for my example: Once you take the screenshot, it should appear in the parenthesis after click. Next, we’re going to send a screenshot command to the computer itself. To do so, we’re going to use the type command, which conveniently types content into the screen. You can also use this for dialog boxes and other such things, but we’re going to just use it to send a screenshot command to the server. To do so, we’re going to tell it to type the 3 key, along with KeyModifier.SHIFT and KeyModifier.CMD: type("3", KeyModifier.SHIFT + KeyModifier.CMD) Now your screen will look something like this: Running the workflow should result in a screenshot on your desktop, of the users on the system. Next, let’s just repeat this process by clicking on all the screens, grabbing a little screenshot of each and producing results on the desktop. All we’re going to do is cut/copy/paste the actual commands and then loop through each of the services and screens in Server Admin until we’ve gotten a screenshot of each: Now is when things can start getting a little more interesting. We could have done everything we did up to now using Automator. But Sikuli has some built in logic. Here, we’re going to use the wait option in the Find section of the sidebar to go ahead and wait each screen that can be latent to show content. After we click the services, we’ll wait for a pattern on the screen not specific to any given system but that only appears when the settings appear, as follows: This is a basic configuration of Sikuli, with a simple task, take some screenshots of the Server application. It can then be anchored to an imaging workflow by invoking the workflow from the command line. The command line is just a shell script sitting in the Sikuli-IDE, which is by default dropped into Applications. Because you won’t need it for standard systems, you can use it in special regression testing workflows from your imaging solution. Save the workflow you’ve been running as some title (I’ll use ServerApp) with the .sikuli extension. Then, to run it from the command line, fire up sikuli along with the actual project and a -r operator to tell it to run: /Applications/ ~/Desktop/ServerApp.sikuli -r You can also use the sikuli-ide.jar or sikuli-script.jar to get an even lighter install, documented at on the site. On that page, it also explains how to pass arguments to Jython’s sys.argv using the –args option as well as using -stderr. The scripting environment from there becomes jython, a mashup language that takes python and java and uses a little duct tape to hold them together. Overall, it’s an interesting concept. There isn’t a lot of logic, unless you’re willing to script things. But if you are willing to script things then you can do pretty much anything you might want. For example, you can have it play bejeweled for you, so you can actually get some work done (although perhaps you’d rather play bejeweled and script your meetings…):

Defaults & symbolichotkeys in Mac OS X

Front Row is awesome. Hot keys are awesome. Typos are not. While zipping along, typing my fool heart out, I tend to fat finger about enough to drop my words per minute in half at times. Occasionally, my typos will land me in an annoying spot, with some application opening: often that application is Front Row. Which led me to unmapping the hot key. But then of course, since I reimage my machines a lot, I wanted to put that into my image… Hot keys are stored in, in a users ~/Library/Preferences. You could setup a system with the exact key mappings that you wish to have, use managed preferences to send specific key strokes into the property list or script the augmentation of the file with the defaults command. Within the file a dictionary called AppleSymbolicHotKeys and a number of options that correspond to options in the Keyboard Shortcuts defined in the Keyboard System Preference pane, under the Keyboard Shortcuts tab. Each shortcut entry is a nested dictionary, with a boolean key for whether it is enabled or not and a value dictionary nested even further within that. The value dictionary has a parameters array for the keys that invoke the action and a key that is a string for type (that is always set to standard). That parameters array is comprised of three items, aptly named: Item 0, Item 1 and Item 2. Those items are filled with numbers that map to a key on the keyboard or a 0 if the key is not used. Some of these keys include:
  • 50: ~
  • 52: Enter
  • 53: Escape
  • 122: F1
  • 120: F2
  • 99: F3
  • 118: F4
  • 96: F5
  • 97: F6
  • 98: F7
  • 100: F8
  • 101: F9
  • 109: F10
  • 103: F11
  • 111: F12
  • 105: F13
  • 107: F14
  • 113: F15
  • 131072: Shift
  • 262144: Control
  • 524288: Option
  • 1048576: Command
Note: A useful tool for finding key codes for combinations of keys is Full Key Codes, from Some of the entries in the list that can be controlled, along with their corresponding numeric identifier in the file:
  • 32: All Windows
  • 33: Application Windows
  • 36: Desktop
  • 62: Dashboard
  • 73: Front Row (my goddess of the fat finger)
Note: If you have one that is not in this list, then simply copy the plist, make the change, open both in your favorite property list editor and either eyeball them for the delta or use the diff command. Now, let’s look at setting that symbolichotkeys property list to set the Front Row (Dictionary 73 within Dictionary AppleSymbolicHotKeys to disable, by changing the enabled key to 0, and then leaving the value dictionary as is by copying it back in with the same values (if you care: delimited from the enabled key with a ; and defined as a a dictionary based on the content between the {} with an array inside of it, defined using parenthesis to start and stop the array, followed with another semicolon to delimit the end of that key followed by the type keypair followed by yet another semicolon to end each open key).
defaults write AppleSymbolicHotKeys -dict-add 73 “{enabled = 0; value = { parameters = (65535, 53, 1048576); type = ‘standard’; }; }”
To then re-enable:
defaults write AppleSymbolicHotKeys -dict-add 73 “{enabled = 1; value = { parameters = (65535, 53, 1048576); type = ‘standard’; }; }”
You could also map different keystrokes by sending different numerical values (some are shown above) into the parameters array.

New PlistCheck Application

PlistCheck is a very small, very fast application that checks preferences (aka – property lists) to make sure that they are properly formatted. There are a few other tools out there that do this and it’s easier to just use the command line, but if you are not command line savvy (or working with someone who isn’t) then you can use this tool to check property lists (preferences) rather than using the command line to do so.

Click here to Download PlistCheck

PlistCheck will be made available on the Apps page as well.