JAMF,  Java,  Mac OS X,  Mac OS X Server,  Mac Security

Troubleshooting Apache, Proxies, and Tomcat in OS X Server 5

OS X Server 5 dropped last week. It’s the first time I’ve seen an OS X Server version drop before an OS release. I’m guessing there was an impetus to get it out the door before OS X 10.11 ships, so that caching and software update servers can facilitate quicker adoption and tools like Profile Manager will work on 0-day. But, there are some funny issues that are popping up. One of these is OS X Server usurping some ports that would otherwise potentially be used by other tools. Notably for Casper administrators, this includes port 8443. So here are some issues I’ve seen with Apache in the latest OS X Server.

Ports are in use that shouldn’t be

This is of particular interest to people running Tomcat sites (e.g. Casper admins). If you have a 3rd party service that isn’t loading, you may find that a port is already in use. For example, let’s say that you’re trying to start a JSS on port 8443. Well, let’s say you run stroke and you see this (when the JSS is stopped):

/System/Library/CoreServices/Applications/Network\ Utility.app/Contents/Resources/stroke 127.0.0.1 8443 8443

And let’s say you get this response (again, with the JSS stopped):

Open TCP Port:      8443      pcsync-https

Well, that means that the server has probably just totally ganked port 8443 for that funky new proxy thing. In /Library/Server/Web/Config/Proxy/apache_serviceproxy.conf there are a few new funny things due to proxy services (that whole proxy folder is new btw). One of which is the fact that the server listens on some ports you might not mean for it to listen on, by default including 80, 443, 8008, 8800, 8443, and 8843. The server always had a default site listening on ports 80 and 443, but now Caldav response is using 8443 for a Virtual Host for the CalendarServer that redirects to /webcal on port 443. Arg. There are a few things you can do to correct this. One would be to comment out one of the lines for the listeners. For this, find the line that reads:

listen 8443

And replace it with:

#listen 8443

This would likely spawn some errors in your apache logs when the virtual hosts that also use 8443 try and load. So you’ll likely also want to comment out the virtual host section of the file. For this, look for <VirtualHost *:8443> to that virtual hosts </VirtualHost> and comment out the whole section. Another option, if you do actually want to use the server as a calendar server as well, might be to replace the asterisk in the definition with an IP address or hostname, which would bind that port to a specific IP address or hostname.

This would be true if you have something using 8008, 8800 (think Kerio), etc.

Also, consider that there’s a /Library/Server/Web/Config/Proxy/apache_serviceproxy_customsites*.conf entry. For 5.03 and 5.04, this isn’t an issue, but any time you see an include like that, you could be loading up multiple includes in the future. Which could introduce additional tasks. Also, keep in mind that you’ll want to keep a backup of this file handy. It’s in a place in your system where Apple can change things in the file without any concern around customizations you previously made in the file. Therefore, in a subsequent software update, you may need to restore that file.

You don’t get prompted that there’s a new version of OS X Server

When you install OS X Server 5, the next time you open the Server app, you should get prompted that the Server app has been replaced and then go through a little assistant. If you don’t, reboot, throw the Server.app in the trash, redownload and reopen the app. That should take care of that issue.

Certificates don’t get migrated

The /Library/Server/Web/Config/Proxy/apache_serviceproxy.conf file will have a number of certificates. These include SSLCertificateFile, SSLCertificateKeyFile, and SSLCertificateChainFile. In /etc/certificates, you’ll have some certificates. For example, on my server, I have:

4A94D0AE-7DD6-4D8D-A721-D62DE2AAE092.C174963A4CB567837EE8B5FD7EC8DCBE03143CCB.cert.pem
4A94D0AE-7DD6-4D8D-A721-D62DE2AAE092.C174963A4CB567837EE8B5FD7EC8DCBE03143CCB.chain.pem
4A94D0AE-7DD6-4D8D-A721-D62DE2AAE092.C174963A4CB567837EE8B5FD7EC8DCBE03143CCB.concat.pem
Server Fallback SSL Certificate.AD80FE0DDF4D16419A158AAA901594FF15D48A2A.cert.pem
Server Fallback SSL Certificate.AD80FE0DDF4D16419A158AAA901594FF15D48A2A.chain.pem
Server Fallback SSL Certificate.AD80FE0DDF4D16419A158AAA901594FF15D48A2A.concat.pem
Server Fallback SSL Certificate.AD80FE0DDF4D16419A158AAA901594FF15D48A2A.key.pem
odr.krypted.com.00EA9C581A8C85D48D295807946C0703DAF88F67.cert.pem
odr.krypted.com.00EA9C581A8C85D48D295807946C0703DAF88F67.chain.pem
odr.krypted.com.00EA9C581A8C85D48D295807946C0703DAF88F67.concat.pem
odr.krypted.com.00EA9C581A8C85D48D295807946C0703DAF88F67.key.pem

One is built based on the promotion of OD, another is a fallback, and the one with the funny GUID in front of it is usually the one that you’d use when defining these fields. If OS X Server doesn’t see the correct pem files that it’s expecting it will just create new ones. The old ones are still there. So, if a service like Profile Manager is totally busted, you can backup the /Library/Server/Web/Config/Proxy/apache_serviceproxy.conf and edit the path to the certificates in the file to correct them. Reboot and see if Profile Manager fires up. On one machine, I also had to trash the Server app again and install it again, but just pointing the paths to the correct location worked for the most part (also, note that I had to use the full path of a file rather than just the name of the file). Oh, don’t forget, this would need to be done for each virtual host with an offending certificate chain.

 

Apache binds ports to all IPs

A final issue I’ll point out is that servers that I’d customized the IP that Apache listens on needed to be reconfigured. This is done in the see /Library/Server/Web/Config/Apache2/httpd_server_app.conf configuration file. Here, look for a line for Listen. It will be commented out as so:

#Listen 12.34.56.78:80

If you want to only have a given port listen on a given IP, use that section of that file to customize how the listener should operate. For example, if you have an IP on your machine of 10.0.0.100 and you only want port 80 listening on that port, use the following

Listen 10.0.0.100:80

Conclusion

Overall, I would say that if you haven’t upgraded to Server 5 on a Yosemite system, that I’d hold off. There are some funny kinks that need to be worked out and I’d hate to be the one figuring some of this out if I wasn’t planning on a funky upgrade session (e.g. if I had a limited downtime window).