krypted.com

Tiny Deathstars of Foulness

OS X Mavericks has a lot of scripts used for enabling services, setting states, changing hostnames and the like. Once upon a time there was a script for OS X Server called server setup. It was a beautiful but too simplistic kind of script. Today, much of that logic has been moved out into more granular scripts, kept in /Applications/Server.app/Contents/ServerRoot/System/Library/ServerSetup, used by the server to perform all kinds of tasks. These scripts are, like a lot of other things in Mountain Lion Server. Some of these include the configuration of amavisd, docecot and alerts. These scripts can also be used for migrating services and data, such as /Applications/Server.app/Contents/ServerRoot/System/Library/ServerSetup/MigrationExtras/30-ipfwmigrator. Sometimes the scripts are in bash, sometimes ruby, sometimes perl and other times even python. Additionally, there’s a directory /Applications/Server.app/Contents/ServerRoot/System/Library/ServerSetup/MigrationExtras/ that is full of scripts for migrating services in OS X Server, helpful for even services that have been seemingly deprecated. One of the things that can can be useful about the scripts scattered throughout the Server app is to learn how the developers of OS X Server intend for certain tasks to occur. One such example is /Applications/Server.app/Contents/ServerRoot/System/Library/ServerSetup/loggather.sh, used to grab logs. Here, you can learn the locations of certain logs as well as rudimentary stackshot commands. This is where I started calling stackshot before I did Server installs (or during), using the following command, which creates a custom text file containing : “/usr/libexec/stackshot -i -f /Library/Logs/ServerSetup_StackShot_KRYPTED.txt” This is also where I learned that I can tail /tmp/SetupLogs.tgz during some installs to be able to watch what’s going on during the installation process: tail -f /tmp/SetupLogs.tgz Looking At Each Service This is also where I learned that Apple had put an Open Directory backup script in /Applications/Server.app/Contents/ServerRoot/usr/libexec/server_backup/opendirectorybackup (that still requires a password). But what I haven’t seen in all of these logs is bumping up the logging level for services before performing tasks, so that you can see a verbose output of what’s going on. To do this, it looks like we’re going service-by-service. So let’s look alphabetically, starting with Address Book: sudo serveradmin settings addressbook:DefaultLogLevel = “warn” This by defualt logs to /var/log/caldavd/error.log, which is built based on the following, which sets the base: sudo serveradmin settings addressbook:LogRoot=/var/log/caldavd And the following, which sets the file name in that directory: sudo serveradmin settings addressbook:ErrorLogFile=error.log You can change either by changing what comes after the = sign. Next is afp. This service logs output to two places. The first is with errors to the service, using /Library/Logs/AppleFileService/AppleFileServiceError.log, the path designated in the following: sudo serveradmin settings afp:errorLogPath = “/Library/Logs/AppleFileService/AppleFileServiceError.log” The second location logs activities (open file, delete file, etc) rather than errors and is /Library/Logs/AppleFileService/AppleFileServiceAccess.log, defined using: sudo serveradmin settings afp:activityLogPath = “/Library/Logs/AppleFileService/AppleFileServiceAccess.log” The activity log is disabled by default and enabled using the command: sudo serveradmin settings afp:activityLog = yes The events that trigger log entries are in the afp:loggingAttributes array and are all enabled by default. There are no further controls for the verbosity of the afp logs. The next service is calendar. Similar to address book, the caldav server uses DefaultLogLevel to set how much data gets placed into logs: sudo serveradmin settings calendar:DefaultLogLevel = “warn” This by defualt logs to /var/log/caldavd/error.log, which is built based on the following, which sets the base: sudo serveradmin settings calendar:LogRoot=/var/log/caldavd And the following, which sets the file name in that directory: sudo serveradmin settings calendar:ErrorLogFile=error.log You can changing either by changing what comes after the = sign. Profile Manager is called devicemgr in the serveradmin interface and I’ve found no way to augment the logging levels. Nor does its migration script ( /Applications/Server.app/Contents/ServerRoot/System/Library/ServerSetup/MigrationExtras/80-devicemgrmigration.sh ) point to any increased logging during migration. The dirserv (aka Open Directory) uses the slapconfig back-end, so I use slapconfig to increase logging: sudo slapconfig -enableslapdlog The DNS service uses named.conf, located in /etc to set log levels and has no serveradmin settings for doing so. Here, use the logging section and look for both the file setting (by default /Library/Logs/named.log) for where the log is stored as well as the severity setting, which can set the logging levels higher or lower. By default Messages, or iChat Server, logs a lot. See the following for what is logged: sudo serveradmin settings jabber:logLevel = “ALL” Adding the -D option to the LaunchDaemon that invokes jabber will increase the logs. Logging long-term is handled in each of the xml files that make up the features of jabber. See the Logconfiguration section of the c2s file via: cat /Applications/Server.app/Contents/ServerRoot/private/etc/jabberd/c2s.xml The mail service has a number of options for logging, much of which has to do with the fact that it’s a patchy solution made up of postfix, etc. Global log locations are controlled using the mail:global:service_data_path key, which indicates a path that logs are stored in (as usual many of these are in /Library/Server): sudo serveradmin settings mail:global:service_data_path = "/Library/Server/Mail" To see the virus database logging levels (which should usually be set to warn): sudo serveradmin settings mail:postfix:virus_db_log_level To see the spamassassin logging levels: sudo serveradmin settings mail:postfix:spam_log_level To see the actual postfix logging level: sudo serveradmin settings mail:postfix:log_level To enable timestamps on logs: sudo serveradmin settings mail:imap:logtimestamps = yes To set the dovecot logging to info: sudo serveradmin settings mail:imap:log_level = “info” To set increased logging per function that dovecot performs, see the config files in /Applications/Server.app/Contents/ServerRoot/private/etc/dovecot/default/conf.d, each of which has a logging section to do so. The NetBoot service is simple to configure logging for, simply set the netboot:logging_level to HIGH (by default it’s MEDIUM): sudo serveradmin settings netboot:logging_level = “HIGH” The Postgres service uses a log directory, configured with postgres:log_directory: sudo serveradmin settings postgres:log_directory = “/Library/Logs/PostgreSQL” The /private/etc/raddb/radiusd.conf has a section (log {}) dedicated to configuring how the radius service logs output. The Xsan service logs output per volume to both the System Log and volume-based log files, stored in /Library/Preferences/Xsan/data. The smb service has a file /Library/Preferences/SystemConfiguration/com.apple.smb.server.plist with a key for log level that can be used for more verbose output of the service. The PPTP VPN service logs output to the file specified in vpn:Servers, configured with these: sudo serveradmin settings vpn:Servers:com.apple.ppp.pptp:Server:LogFile = “/var/log/ppp/vpnd.log” sudo serveradmin settings vpn:Servers:com.apple.ppp.pptp:PPP:LogFile = “/var/log/ppp/vpnd.log” sudo serveradmin settings vpn:Servers:com.apple.ppp.l2tp:Server:LogFile = “/var/log/ppp/vpnd.log” sudo serveradmin settings vpn:Servers:com.apple.ppp.l2tp:PPP:LogFile = “/var/log/ppp/vpnd.log” By default, verbose logging is enabled, which you can see with: sudo serveradmin settings vpn:Servers:com.apple.ppp.pptp:Server:VerboseLogging sudo serveradmin settings vpn:Servers:com.apple.ppp.pptp:PPP:VerboseLogging sudo serveradmin settings vpn:Servers:com.apple.ppp.l2tp:Server:VerboseLogging sudo serveradmin settings vpn:Servers:com.apple.ppp.l2tp:PPP:VerboseLogging The last service is web (Apache). The default access logs are per-site, with a key called customLogPath existing for each. The defaultSite uses the following for its logs: sudo serveradmin settings web:defaultSite:customLogPath Swap out the defaultSite with another site to see its log paths. There’s also a key for errorLogPath that shows errors. These are per-site so that administrators can provide access to logs for the owners of each site and not fear them having access to logs for other users. Global error logs are stored in /private/var/log/apache2/error_log as defined in /private/etc/apache2/httpd.conf. Find LogLevel in this file and set it to configure how in depth the logs will be, using debug for the most verbose and info, notice, warn, error, crit, alert, and emerg to get incrementally less information. Additionally the log formats can be set in /private/etc/apache2/httpd.conf, allowing administrators to configure Mountain Lion Server’s built-in web service to conform to the standards of most modern web log analyzers. Conclusion Overall, there’s a lot of information in these logs and administrators can spend as much time reviewing logs as they want. But other than standard system logs, the output is typically configured on a service-by-service basis. Some services offer a lot of options and others offering only a few. Some services also offer options within the serveradmin environment while others use their traditional locations in their configuration files. I’ll end this with a warning. There can also be a lot of output in these logs. Therefore, if you set the logging facilities high, make sure to keep a watchful eye on the capacity of the location you’re writing logs out to. The reason I looked at paths to logs where applicable was because you might want to consider redirecting logs to an external volume when debugging so as not to fill up a boot volume and cause even more problems than what you’re likely parsing through logs looking to fix…

October 22nd, 2013

Posted In: Mac OS X Server

Tags: , , , , , , , , , , , , , , , ,

Mac OS X Server 10.7, Lion Server, comes with a few substantial back-end changes. One of these is the move from SQLite3 to PostgreSQL for many of the back-end databases, including Wiki and Podcast Producer (collab), Webmail (roundcubemail), iCal Server and Address Book Server (caldav) and as the back-end to the newest service in Lion Server, Profile Manager (device_management). As such, it’s now important to be able to use PostgreSQL the way we once used SQLite3, when trying to augment the data that these databases contains, as there currently aren’t a lot of options for editing this data (aside from manually of course). Postgres has a number of commands that can be used to interact with databases. The most important is probably psql. Many of the other commands simply provide automated options to psql, and over time I’ve started using psql for most everything. For example, PostgreSQL comes with a command /user/bin/createuser. However, as it’s usually more verbose with errors, I like to use psql to do this. In Lion Server, the only user that can access the Postgres databases is _postgres, installed by default with Lion Server. Because a lot of commands require passwords and we might not always want to provide write access to the databases, we’re going to create a new SuperUser, called krypted with a password of daneel. To do so, we will have to use the _postgres user to invoke psql. Any time you want to invoke psql with a different user than the user you are currently logged in as, use the -U option. To define a database, use the -d option (device_management providing access to Profile Manager data, caldav to iCal Server data roundcubemail to WebMail data and collar to Wiki data). To string this together, for accessing the device_management database as _postgres: psql -U _postgres -d device_management To then create a new user called krypted with a password of daneel we’ll use the create option, defining a user as the type of object to create, followed by the user name and then with password followed by the password (single quoted) and then createuser; as follows: device_management=# create user krypted with password 'daneel' create user; Now that there’s a valid user, let’s see what else we can do. To see all of the tables, use d: device_management=# d As you can tell, there are a bunch of them. Run the help command to see a list of SQL commands that can be run and ? for a list of psql options. To put some SQL commands into action, we’re going to look at the tasks that have been performed by Profile Manager. These are stored in the tasks table (aptly named), so we’re going to run the following SQL query (note a space followed by a semi-colon is required at the end of this thing): device_management=# select * from "public"."tasks" limit 1000 offset 0 ; Or to make it a bit simpler if you don’t have a lot of data in there yet: device_management=# select * from "public"."tasks" ; After seeing the output, you’ll probably be a little appreciative of Apple’s formatting. Next, let’s look at dumping the databases. We’re going to create a folder on the root of the volume called db_backups first: sudo mkdir /db_backups This is where these backups will end up getting stored. We’ll continue using the _postgres user for now. To do our database dumps, we’re going to use pg_dump, located at /usr/bin. First, we’ll dump the device_management database (but first we’ll stop the service and after we’ll start it – all commands from here on out also assume you’re sudo’d): serveradmin stop devicemgr pg_dump -U _postgres device_management -c -f /db_backups/device_management.sql serveradmin start devicemgr And the other 3 (stopping and starting each in the process): serveradmin stop web pg_dump -U _postgres roundcubemail -c -f /db_backups/roundcubemail.sql serveradmin start web serveradmin stop wiki pg_dump -U _postgres collab -c -f /db_backups/collab.sql serveradmin start wiki serveradmin stop addressbook serveradmin stop calendar pg_dump -U _postgres caldav -c -f /db_backups/caldav.sql serveradmin start addressbook serveradmin start calendar I haven’t had any problems running the dumps with the services running, but it’s better safe than sorry I guess. I’d probably also add some logging and maybe dump the output of full status for each service to try and track if all is well with each. Any time a service didn’t fire back up I’d then build in a sanity check for that event. There’s also a database for postgres itself, so let’s back that up as well since we’re here: pg_dump -U _postgres postgres -c -f /db_backups/postgres.sql These can then be restored using psql with the -d option to define the database being restored into and the -f option to define the file being restored from. For example, to restore collab: psql -U _postgres -d collab -f /db_backups/collab The databases are all dumped daily using pg_dumpall. These are stored in /var/pgsql but can be changed using serveradmin settings (for example, to move them to /var/pgsql1): serveradmin settings postgres:dataDir = "/var/pgsql1" If you mess up the Profile Manager database (before you put any real data into it) you can always use the /usr/share/devicemgr/backend/wipeDB.sh script to trash the database and start anew (although I’d just use a snapshot of a VM for all this and restore from that). You can also connect to Postgres remotely, or locally through a network socket (common in Apache uses) by adding a listener. To do so, we’ll need to restart the Postgres LaunchDaemon. First, back up the file, just in case: cp org.postgresql.postgres.plist org.postgresql.postgres.plist.OLD_CSE Then stop postgres: serveradmin stop postgres Then edit the org.postgresql.postgres.plist file to change the following line: listen_addresses= To read: listen_addresses=127.0.0.1 Then fire up postgres again: serveradmin start postgres And now let’s scan port 5432 (the default TCP and UDP port used for postgres) for localhost: /Applications/Utilities/Network Utility.app/Contents/Resources/stroke 127.0.0.1 5432 5432 We could have used another IP address for the listen_addresses as well, but with that _postgres user not requiring a password it didn’t really seem prudent to do so. Once you’ve enabled a socket, you’ll then be able to use one of the many GUI tools to manage postgres. Navicat is available on the Mac App Store for $5 and PGnJ is a nice, easy to use, free one. There are tons of others, but I don’t spend a lot of time in a SQL GUI and so don’t need more than a cheap app will get me. One nice thing about most of these is that they help you to form SQL queries (or they help me). This can get really nice if you are, for example, trying to get some good reporting on Profile Manager (a feature it’s a bit light on right now). Finally, don’t do any of this stuff on a production box, except maybe if you want more than nightly backups unless you think pretty hard about what you’re doing and know the exact impact of doing something. If you were to edit the databases on a live boxen, then you can safely assume that with how all of the objects in those databases use GUIDs that you’re probably going to break something, if not bring the whole house of cards tumbling down.

January 4th, 2012

Posted In: iPhone, Mac OS X Server, Mac Security, Mass Deployment, SQL

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,