Fixing Service Issues When Upgrading to 10.7.3 Server

The 10.7.2 to 10.7.3 update for Lion Server has introduced a few issues in some environments that I’ve seen. It just so happens that the update corrects a lot of behavior with Lion Server while also introducing new features, so it’s something you’re gonna’ need to do eventually. Therefore, before I update, I would strongly recommend backing up all of your services, your service data and Open Directory.

Once you’ve run the 10.7.3 update, there are a few things that I’ve seen happen. The first is that the web server won’t start. If this happens, reset the web server back to factory default:

serveradmin command web:command=restoreFactorySettings

Once it’s reset, you should be able to import any data that was backed up before and get things back to normal. The second is calendar data. On a few different systems I’ve seen users have to nuke iCal and then reimport data. To nuke and pave iCal, see this post: http://krypted.com/mac-os-x-server/nukepave-ical-server-in-lion-server. Once iCal Server has been restored to full working order (after the last step in that article) you can use psql to restore your data from the location of your backups (here called /backup/caldav.sql):

psql -U _postgres -d caldav -f /backup/caldav.sql

There’s also a script located in /usr/share/caldavd/lib/python/calendarserver/tools
called fix calendardata.py that can be used to scan and possibly fix any issues with the data itself. If that doesn’t not work though, you may be starting over. The script does not give root execute permissions by default and so you will need to chmod it to provide execute and then run it.

If you nuke CalDAV and you nuke OD and then restore them both, the GeneratedUIDs can be mismatched. Use the Directory Editor in the new Directory Utility to browse users and attach the GeneratedUID back to the correct entry in CalDAV. To locate all of the entries in CalDAV, run:

psql -U _postgres caldav -c “select * from calendar_home"

If Profile Manager won’t load it could be one of three issues (in the following order seemingly). The first is the web server, which the first command will fix. Another issue I’ve seen is that Open Directory gets a little messed up. The fix for this is to use Server Admin (not slapconfig) to burn OD down and set it back up. You can then promote replicas and finally restore the archive you did before upgrading the server. The third is to reset the Profile Manager database using wipeDB.sh:

sudo /usr/share/devicemgr/backend/wipeDB.sh

After wiping the data, you can re-run the setup in Server app for the Profile Manager service to restore an empty Profile Manager instance to working order. You can restore data into the empty Profile Manager database using the same commands I showed earlier for CalDAV, just use devicemgrd instead.

Note: I am pretty sure you need sudo for most every command I use on this site, but more specifically you need it with this stuff. So sudo is assumed if not explicitly stated.

Finally, be on the lookout for custom designs in the Wiki interface. OS updates are known to change things, but more specifically when things are not documented they can easily change. Hacking the pages nested within /usr/share/collabd is basically not supported any more. Each OS update to 10.7 has broken some of the hacks we’ve done to collabd, making me wonder whether it’s a good idea any more…

Note2: I have had little issues running these updates in walled gardens. It’s production data that is the problem. It seems that most of the issues are data driven (the opposite of data driven design is not devops driven design).

6 Comments

  • Martin
    March 6, 2012 - 6:55 am | Permalink

    This really sounds very scaring. Even more because upgrading to 10.7.3 is still on my todo list..

    • March 6, 2012 - 7:03 am | Permalink

      It’s not a big deal. I’m sure problems w/ the upgrade are pretty rare. They all seem to revolve around services that use Postgres (Profile Manager, Calendar & Wiki) and Apache/SproutCore.

  • Kevin
    May 30, 2012 - 1:02 pm | Permalink

    If I do this : hightechlights:~ kevin$ sudo /usr/share/devicemgr/backend/wipeDB.sh
    i get this : devicemgr:state = “STOPPED”
    postgres:error = “CANNOT_START_SERVICE_TIMEOUT_ERR”
    (in /usr/share/devicemgr/backend)
    Couldn’t drop device_management : #
    (in /usr/share/devicemgr/backend)
    fe_sendauth: no password supplied
    /usr/share/devicemgr/backend/vendor/rails/activerecord/lib/active_record/connection_adapters/postgresql_adapter.rb:941:in `initialize’
    /usr/share/devicemgr/backend/vendor/rails/activerecord/lib/active_record/connection_adapters/postgresql_adapter.rb:941:in `connect’
    /usr/share/devicemgr/backend/vendor/rails/activerecord/lib/active_record/connection_adapters/postgresql_adapter.rb:941:in `connect’
    /usr/share/devicemgr/backend/vendor/rails/activerecord/lib/active_record/connection_adapters/postgresql_adapter.rb:217:in `initialize’
    /usr/share/devicemgr/backend/vendor/rails/activerecord/lib/active_record/connection_adapters/postgresql_adapter.rb:37:in `new’
    /usr/share/devicemgr/backend/vendor/rails/activerecord/lib/active_record/connection_adapters/postgresql_adapter.rb:37:in `postgresql_connection’
    /usr/share/devicemgr/backend/vendor/rails/activerecord/lib/active_record/connection_adapters/abstract/connection_pool.rb:223:in `send’
    /usr/share/devicemgr/backend/vendor/rails/activerecord/lib/active_record/connection_adapters/abstract/connection_pool.rb:223:in `new_connection’
    /usr/share/devicemgr/backend/vendor/rails/activerecord/lib/active_record/connection_adapters/abstract/connection_pool.rb:245:in `checkout_new_connection’
    /usr/share/devicemgr/backend/vendor/rails/activerecord/lib/active_record/connection_adapters/abstract/connection_pool.rb:188:in `checkout’
    /usr/share/devicemgr/backend/vendor/rails/activerecord/lib/active_record/connection_adapters/abstract/connection_pool.rb:184:in `loop’
    /usr/share/devicemgr/backend/vendor/rails/activerecord/lib/active_record/connection_adapters/abstract/connection_pool.rb:184:in `checkout’
    /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/monitor.rb:242:in `synchronize’
    /usr/share/devicemgr/backend/vendor/rails/activerecord/lib/active_record/connection_adapters/abstract/connection_pool.rb:183:in `checkout’
    /usr/share/devicemgr/backend/vendor/rails/activerecord/lib/active_record/connection_adapters/abstract/connection_pool.rb:98:in `connection’
    /usr/share/devicemgr/backend/vendor/rails/activerecord/lib/active_record/connection_adapters/abstract/connection_pool.rb:326:in `retrieve_connection’
    /usr/share/devicemgr/backend/vendor/rails/activerecord/lib/active_record/connection_adapters/abstract/connection_specification.rb:123:in `retrieve_connection’
    /usr/share/devicemgr/backend/vendor/rails/activerecord/lib/active_record/connection_adapters/abstract/connection_specification.rb:115:in `connection’
    /usr/share/devicemgr/backend/vendor/rails/railties/lib/tasks/databases.rake:70:in `create_database’
    /usr/share/devicemgr/backend/vendor/rails/railties/lib/tasks/databases.rake:31
    /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/rake.rb:636:in `call’
    /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/rake.rb:636:in `execute’
    /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/rake.rb:631:in `each’
    /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/rake.rb:631:in `execute’
    /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/rake.rb:597:in `invoke_with_call_chain’
    /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/monitor.rb:242:in `synchronize’
    /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/rake.rb:590:in `invoke_with_call_chain’
    /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/rake.rb:583:in `invoke’
    /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/rake.rb:2051:in `invoke_task’
    /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/rake.rb:2029:in `top_level’
    /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/rake.rb:2029:in `each’
    /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/rake.rb:2029:in `top_level’
    /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/rake.rb:2068:in `standard_exception_handling’
    /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/rake.rb:2023:in `top_level’
    /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/rake.rb:2001:in `run’
    /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/rake.rb:2068:in `standard_exception_handling’
    /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/rake.rb:1998:in `run’
    /usr/bin/rake:31
    Couldn’t create database for {“adapter”=>”postgresql”, “database”=>”device_management”, “encoding”=>”UTF8″, “pool”=>5, “username”=>”_devicemgr”}
    (in /usr/share/devicemgr/backend)
    May 30 12:59:18 hightechlights.private ProfileManager[36640] : Unable to create ProfileManager log file at ‘/var/log/devicemgr/profilemanager.log’ (No such file or directory)
    rake aborted!
    fe_sendauth: no password supplied

    (See full trace by running task with –trace)
    devicemgr:state = “STARTING”

    Sweet, huh ?

    • May 30, 2012 - 2:58 pm | Permalink

      I’ve had two different things fix this. The first and simplest is to run it a second time. The second way, which is a bit more complicated would be to nuke and pave the OD database, then re-run this command. Neither seem great, but I’ve had both fix this issue.

  • ZtF
    July 26, 2012 - 5:29 pm | Permalink

    btw. wipeDB.sh location (path) in Mountain Lion is /Applications/Server.app/Contents/ServerRoot/usr/share/devicemgr/backend/

  • Matt Domenici
    October 6, 2012 - 4:11 pm | Permalink

    Hi Charles, this didn’t work for me. I went ahead and ran the wipeDB script. After that, trying to reconfigure the Profile Manager service results in the assistant in Server.app hanging during configuration at about 75% of the way through. Do you have any ideas?

  • Comments are closed.