Troubleshooting Apache, Proxies, and Tomcat in OS X Server 5

OS X Server 5 dropped last week. It’s the first time I’ve seen an OS X Server version drop before an OS release. I’m guessing there was an impetus to get it out the door before OS X 10.11 ships, so that caching and software update servers can facilitate quicker adoption and tools like Profile Manager will work on 0-day. But, there are some funny issues that are popping up. One of these is OS X Server usurping some ports that would otherwise potentially be used by other tools. Notably for Casper administrators, this includes port 8443. So here are some issues I’ve seen with Apache in the latest OS X Server. Ports are in use that shouldn’t be This is of particular interest to people running Tomcat sites (e.g. Casper admins). If you have a 3rd party service that isn’t loading, you may find that a port is already in use. For example, let’s say that you’re trying to start a JSS on port 8443. Well, let’s say you run stroke and you see this (when the JSS is stopped): /System/Library/CoreServices/Applications/Network\ 8443 8443 And let’s say you get this response (again, with the JSS stopped): Open TCP Port:      8443      pcsync-https Well, that means that the server has probably just totally ganked port 8443 for that funky new proxy thing. In /Library/Server/Web/Config/Proxy/apache_serviceproxy.conf there are a few new funny things due to proxy services (that whole proxy folder is new btw). One of which is the fact that the server listens on some ports you might not mean for it to listen on, by default including 80, 443, 8008, 8800, 8443, and 8843. The server always had a default site listening on ports 80 and 443, but now Caldav response is using 8443 for a Virtual Host for the CalendarServer that redirects to /webcal on port 443. Arg. There are a few things you can do to correct this. One would be to comment out one of the lines for the listeners. For this, find the line that reads:
listen 8443
And replace it with:
#listen 8443
This would likely spawn some errors in your apache logs when the virtual hosts that also use 8443 try and load. So you’ll likely also want to comment out the virtual host section of the file. For this, look for <VirtualHost *:8443> to that virtual hosts </VirtualHost> and comment out the whole section. Another option, if you do actually want to use the server as a calendar server as well, might be to replace the asterisk in the definition with an IP address or hostname, which would bind that port to a specific IP address or hostname. This would be true if you have something using 8008, 8800 (think Kerio), etc. Also, consider that there’s a /Library/Server/Web/Config/Proxy/apache_serviceproxy_customsites*.conf entry. For 5.03 and 5.04, this isn’t an issue, but any time you see an include like that, you could be loading up multiple includes in the future. Which could introduce additional tasks. Also, keep in mind that you’ll want to keep a backup of this file handy. It’s in a place in your system where Apple can change things in the file without any concern around customizations you previously made in the file. Therefore, in a subsequent software update, you may need to restore that file. You don’t get prompted that there’s a new version of OS X Server When you install OS X Server 5, the next time you open the Server app, you should get prompted that the Server app has been replaced and then go through a little assistant. If you don’t, reboot, throw the in the trash, redownload and reopen the app. That should take care of that issue. Certificates don’t get migrated The /Library/Server/Web/Config/Proxy/apache_serviceproxy.conf file will have a number of certificates. These include SSLCertificateFile, SSLCertificateKeyFile, and SSLCertificateChainFile. In /etc/certificates, you’ll have some certificates. For example, on my server, I have: 4A94D0AE-7DD6-4D8D-A721-D62DE2AAE092.C174963A4CB567837EE8B5FD7EC8DCBE03143CCB.cert.pem 4A94D0AE-7DD6-4D8D-A721-D62DE2AAE092.C174963A4CB567837EE8B5FD7EC8DCBE03143CCB.chain.pem 4A94D0AE-7DD6-4D8D-A721-D62DE2AAE092.C174963A4CB567837EE8B5FD7EC8DCBE03143CCB.concat.pem Server Fallback SSL Certificate.AD80FE0DDF4D16419A158AAA901594FF15D48A2A.cert.pem Server Fallback SSL Certificate.AD80FE0DDF4D16419A158AAA901594FF15D48A2A.chain.pem Server Fallback SSL Certificate.AD80FE0DDF4D16419A158AAA901594FF15D48A2A.concat.pem Server Fallback SSL Certificate.AD80FE0DDF4D16419A158AAA901594FF15D48A2A.key.pem One is built based on the promotion of OD, another is a fallback, and the one with the funny GUID in front of it is usually the one that you’d use when defining these fields. If OS X Server doesn’t see the correct pem files that it’s expecting it will just create new ones. The old ones are still there. So, if a service like Profile Manager is totally busted, you can backup the /Library/Server/Web/Config/Proxy/apache_serviceproxy.conf and edit the path to the certificates in the file to correct them. Reboot and see if Profile Manager fires up. On one machine, I also had to trash the Server app again and install it again, but just pointing the paths to the correct location worked for the most part (also, note that I had to use the full path of a file rather than just the name of the file). Oh, don’t forget, this would need to be done for each virtual host with an offending certificate chain.   Apache binds ports to all IPs A final issue I’ll point out is that servers that I’d customized the IP that Apache listens on needed to be reconfigured. This is done in the see /Library/Server/Web/Config/Apache2/httpd_server_app.conf configuration file. Here, look for a line for Listen. It will be commented out as so:
If you want to only have a given port listen on a given IP, use that section of that file to customize how the listener should operate. For example, if you have an IP on your machine of and you only want port 80 listening on that port, use the following
Conclusion Overall, I would say that if you haven’t upgraded to Server 5 on a Yosemite system, that I’d hold off. There are some funny kinks that need to be worked out and I’d hate to be the one figuring some of this out if I wasn’t planning on a funky upgrade session (e.g. if I had a limited downtime window).

10 thoughts on “Troubleshooting Apache, Proxies, and Tomcat in OS X Server 5”

  1. After updating I found myself having to find every config file with RewriteRule ^/$ /xcode or RewriteRule ^/$ /wiki or similar (sometimes the ^/$ were in quotes) and remove the lines as they overrode the previous behaviour where all I had to do was configure the site to not treat /xcode or /wiki as index files. once remove the websites started to behave again.

  2. Thanks for your article and for shedding some light into the new Server 5 web services! To hold off the Server 5 update was easier said than done. Contrary to earlier updates of the server app this one was downloaded automatically on some of my servers, urging me to immediate action. I am not impressed with Apple on this one (as well as their non-existent documentation).

    1. I’m wondering why any mission critical servers would be configured to auto-update app store apps (or worse the operating system).

      1. You’re absolutely right. In my case those machines have been running OS X Server since 10.5 or even earlier and were migrated to new system versions over the years. And since version 1.0 Apple never automatically updated their – apparently until version 5. So it’s critical to switch off auto-updates for apps and system in System Preferences – App Store.

        1. Until now, versions have required a specific OS X major version which would cause the App Store to not auto-install the update until the OS X version was updated. Version 5 is the first to have a lower OS X requirement, making it easier for the App Store to update the without the OS X update happening first.

          I would guess no one has checked the update settings on those updated since 10.5 servers, so as they got updated, auto-update settings that automatically got turned on were never turned off. In-turn, when Apple finally produced a version of that didn’t require a major OS X update, the servers updated unexpectedly, catching their server admins unaware.

  3. I’m having a real nightmare with virtual hosts, in server 4 I used proxy pass in the virtual host file for services like Kerio and Rumpus but now that doesn’t work, the read me file and apple discussions strongly recommend that you use webapps and create a includes conf file and plist for each site, but when I try and start the webapp using webappctl start I get an error

  4. Nice article, I upgraded OSX server 4 to 5 this morning and maybe I should have read your conclusion before even trying to update…

    I’m facing a strange problem with the “getsslpassphrase” executable being killed by the kernel, thus preventing my SSL websites that uses certificates generated by from starting. On the same machine, I have another SSL site that uses a certificate provided by a well known certificates issuer that do not have any issues (maybe its private key is not password protected, I did not checked).

    After digging into the migration process, I can see that /Library/Server/Web/Config/apache2/getsslpassphrase is actually a copy of /Applications/

    Using the terminal, I started investigating a bit, and a strange thing happened : Starting /Library/Server/Web/Config/apache2/getsslpassphrase from the CLI, the process gets killed by the kernel (you can see the logs below). Starting the same executable (checked with diff) from /Applications/, it is not killed…

    Unfortunately, apache config files are using the first path, and I’m pretty sure bad things will happen if I change for the one that works.

    Seems to be a code signing issue. Any idea ?


    Oct 3 09:27:41 albertville kernel[0]: proc 11247: load code signature error 2 for file “getsslpassphrase”
    Oct 3 09:27:41 albertville kernel[0]: CODE SIGNING: cs_invalid_page(0x1052a0000): p=11247[getsslpassphrase] final status 0x3000200, denying page sending SIGKILL
    Oct 3 09:27:41 albertville kernel[0]: CODE SIGNING: process 11247[getsslpassphrase]: rejecting invalid page at address 0x1052a0000 from offset 0x0 in file “/Library/Server/Web/Config/apache2/getsslpassphrase” (cs_mtime:1435693648.0 != mtime:1443852688.0) (signed:1 validated:1 tainted:1 wpmapped:0 slid:0)

Comments are closed.