Logs, Scripts and OS X Mountain Lion Server

OS X Mountain Lion has a lot of scripts used for enabling services, setting states, changing hostnames and the like. Once upon a time there was a script for OS X Server called server setup. It was a beautiful but too simplistic kind of script. Today, much of that logic has been moved out into more granular scripts, kept in /Applications/Server.app/Contents/ServerRoot/System/Library/ServerSetup, used by the server to perform all kinds of tasks. These scripts are, like a lot of other things in Mountain Lion Server. Some of these include the configuration of amavisd, docecot and alerts. These scripts can also be used for migrating services and data, such as /Applications/Server.app/Contents/ServerRoot/System/Library/ServerSetup/MigrationExtras/30-ipfwmigrator. Sometimes the scripts are in bash, sometimes ruby, sometimes perl and other times even python.

Additionally, there’s a directory /Applications/Server.app/Contents/ServerRoot/System/Library/ServerSetup/MigrationExtras/ that is full of scripts for migrating services in OS X Server, helpful for even services that have been seemingly deprecated.

One of the things that can can be useful about the scripts scattered throughout the Server app is to learn how the developers of OS X Server intend for certain tasks to occur. One such example is /Applications/Server.app/Contents/ServerRoot/System/Library/ServerSetup/loggather.sh, used to grab logs. Here, you can learn the locations of certain logs as well as rudimentary stackshot commands. This is where I started calling stackshot before I did Server installs (or during), using the following command, which creates a custom text file containing :

/usr/libexec/stackshot -i -f /Library/Logs/ServerSetup_StackShot_KRYPTED.txt

This is also where I learned that I can tail /tmp/SetupLogs.tgz during some installs to be able to watch what’s going on during the installation process:

tail -f /tmp/SetupLogs.tgz

Looking At Each Service

This is also where I learned that Apple had put an Open Directory backup script in /Applications/Server.app/Contents/ServerRoot/usr/libexec/server_backup/opendirectorybackup (that still requires a password). But what I haven’t seen in all of these logs is bumping up the logging level for services before performing tasks, so that you can see a verbose output of what’s going on. To do this, it looks like we’re going service-by-service. So let’s look alphabetically, starting with Address Book:

sudo serveradmin settings addressbook:DefaultLogLevel = "warn"

This by defualt logs to /var/log/caldavd/error.log, which is built based on the following, which sets the base:

sudo serveradmin settings addressbook:LogRoot=/var/log/caldavd

And the following, which sets the file name in that directory:

sudo serveradmin settings addressbook:ErrorLogFile=error.log

You can change either by changing what comes after the = sign.

Next is afp. This service logs output to two places. The first is with errors to the service, using /Library/Logs/AppleFileService/AppleFileServiceError.log, the path designated in the following:

sudo serveradmin settings afp:errorLogPath = "/Library/Logs/AppleFileService/AppleFileServiceError.log"

The second location logs activities (open file, delete file, etc) rather than errors and is /Library/Logs/AppleFileService/AppleFileServiceAccess.log, defined using:

sudo serveradmin settings afp:activityLogPath = "/Library/Logs/AppleFileService/AppleFileServiceAccess.log"

The activity log is disabled by default and enabled using the command:

sudo serveradmin settings afp:activityLog = yes

The events that trigger log entries are in the afp:loggingAttributes array and are all enabled by default. There are no further controls for the verbosity of the afp logs.

The next service is calendar. Similar to address book, the caldav server uses DefaultLogLevel to set how much data gets placed into logs:

sudo serveradmin settings calendar:DefaultLogLevel = "warn"

This by defualt logs to /var/log/caldavd/error.log, which is built based on the following, which sets the base:

sudo serveradmin settings calendar:LogRoot=/var/log/caldavd

And the following, which sets the file name in that directory:

sudo serveradmin settings calendar:ErrorLogFile=error.log

You can changing either by changing what comes after the = sign.

Profile Manager is called devicemgr in the serveradmin interface and I’ve found no way to augment the logging levels. Nor does its migration script ( /Applications/Server.app/Contents/ServerRoot/System/Library/ServerSetup/MigrationExtras/80-devicemgrmigration.sh ) point to any increased logging during migration.

The dirserv (aka Open Directory) uses the slapconfig back-end, so I use slapconfig to increase logging:

sudo slapconfig -enableslapdlog

The DNS service uses named.conf, located in /etc to set log levels and has no serveradmin settings for doing so. Here, use the logging section and look for both the file setting (by default /Library/Logs/named.log) for where the log is stored as well as the severity setting, which can set the logging levels higher or lower.

By default Messages, or iChat Server, logs a lot. See the following for what is logged:

sudo serveradmin settings jabber:logLevel = "ALL"

Adding the -D option to the LaunchDaemon that invokes jabber will increase the logs. Logging long-term is handled in each of the xml files that make up the features of jabber. See the Logconfiguration section of the c2s file via:

cat /Applications/Server.app/Contents/ServerRoot/private/etc/jabberd/c2s.xml

The mail service has a number of options for logging, much of which has to do with the fact that it’s a patchy solution made up of postfix, etc. To see the virus database logging levels (which should usually be set to warn):

sudo serveradmin settings mail:postfix:virus_db_log_level

To see the spamassassin logging levels:

sudo serveradmin settings mail:postfix:spam_log_level

To see the actual postfix logging level:

sudo serveradmin settings mail:postfix:log_level

To enable timestamps on logs:

sudo serveradmin settings mail:imap:logtimestamps = yes

To set the dovecot logging:

sudo serveradmin settings mail:imap:log_level = "warn"

To set increased logging per function that dovecot performs, see the config files in /Applications/Server.app/Contents/ServerRoot/private/etc/dovecot/default/conf.d, each of which has a logging section to do so.

The NetBoot service is simple to configure logging for, simply set the netboot:logging_level to HIGH (by default it’s MEDIUM):

sudo serveradmin settings netboot:logging_level = "HIGH"

The Postgres service uses a log directory, configured with postgres:log_directory:

sudo serveradmin settings postgres:log_directory = "/Library/Logs/PostgreSQL"

The /private/etc/raddb/radiusd.conf has a section (log {}) dedicated to configuring how the radius service logs output.

The san service (Xsan) logs output per volume to both the System Log and volume-based log files, stored in /Library/Preferences/Xsan/data.

The smb service has a file /Library/Preferences/SystemConfiguration/com.apple.smb.server.plist
with a key for log level that can be used for more verbose output of the service.

The PPTP VPN service logs output to the file specified in vpn:Servers, configured with these:

sudo serveradmin settings vpn:Servers:com.apple.ppp.pptp:Server:LogFile = "/var/log/ppp/vpnd.log"
sudo serveradmin settings vpn:Servers:com.apple.ppp.pptp:PPP:LogFile = "/var/log/ppp/vpnd.log"
sudo serveradmin settings vpn:Servers:com.apple.ppp.l2tp:Server:LogFile = "/var/log/ppp/vpnd.log"
sudo serveradmin settings vpn:Servers:com.apple.ppp.l2tp:PPP:LogFile = "/var/log/ppp/vpnd.log"

By default, verbose logging is enabled, which you can see with:

sudo serveradmin settings vpn:Servers:com.apple.ppp.pptp:Server:VerboseLogging
sudo serveradmin settings vpn:Servers:com.apple.ppp.pptp:PPP:VerboseLogging
sudo serveradmin settings vpn:Servers:com.apple.ppp.l2tp:Server:VerboseLogging
sudo serveradmin settings vpn:Servers:com.apple.ppp.l2tp:PPP:VerboseLogging

The last service is web (Apache). The default access logs are per-site, with a key called customLogPath existing for each. The defaultSite uses the following for its logs:

sudo serveradmin settings web:defaultSite:customLogPath

Swap out the defaultSite with another site to see its log paths. There’s also a key for errorLogPath that shows errors. These are per-site so that administrators can provide access to logs for the owners of each site and not fear them having access to logs for other users. Global error logs are stored in /private/var/log/apache2/error_log as defined in /private/etc/apache2/httpd.conf. Find LogLevel in this file and set it to configure how in depth the logs will be, using debug for the most verbose and info, notice, warn, error, crit, alert, and emerg to get incrementally less information.

Additionally the log formats can be set in /private/etc/apache2/httpd.conf, allowing administrators to configure Mountain Lion Server’s built-in web service to conform to the standards of most modern web log analyzers.

Conclusion

Overall, there’s a lot of information in these logs and administrators can spend as much time reviewing logs as they want. But other than standard system logs, the output is typically configured on a service-by-service basis. Some services offer a lot of options and others offering only a few. Some services also offer options within the serveradmin environment while others use their traditional locations in their configuration files. I’ll end this with a warning. There can also be a lot of output in these logs. Therefore, if you set the logging facilities high, make sure to keep a watchful eye on the capacity of the location you’re writing logs out to. The reason I looked at paths to logs where applicable was because you might want to consider redirecting logs to an external volume when debugging so as not to fill up a boot volume and cause even more problems than what you’re likely parsing through logs looking to fix…

One comment

  • Matt Domenici
    August 21, 2012 - 10:42 pm | Permalink

    Thanks Charles. As far as I can tell, somehow half of my mail service log files are being directed at an empty directory. What’s odd is that they work just fine after the server is bounced. Looking in terminal at the root locations now though, shows empty. Any tips as to why?

  • Comments are closed.