I originally posted this at http://www.318.com/TechJournal
Introduction
Xsan requires a dedicated ethernet network in the supported architecture by Apple. For systems that are obtaining directory information or need to be wired into the corporate network of many organizations this can cause issues. Namely that Xsan will attempt to use the corporate network for connectivity with clients. We see this in many configurations and it can cause dropped packets, unmountable volumes and other intermittent issues.
One way to fix this for metadata controllers is to choose the network adapter that you would like to use on the metadata network in Server Admin. This can be done by:
Open Xsan Admin
Click on the Xsan listed under SAN Components
Clicking on Setup
Click on Computers
Click on the metadata controller
Choose the network controller you would like to connect using
This doesn’t always work, and selecting a network controller to use is not an available option with Xsan clients and older versions of the Xsan software. A fix to get around this issue for these systems is to block metadata traffic on the production network interface. Here is an example of how to do this using ipfw. This example assumes that you are using Tiger Server for your metadata controller. Once we step through this using Server Admin we will explain the same thing using the configuration files which should help users of old versions of Mac OS X Server or of Mac OS X Workstation.
Through this entire article we are going to assume that the IP address of our metadata controllers production IP is 192.168.50.5. The IP address of our metadata controllers metadata IP is 10.0.0.5. If metadata goes over the 192.168.50.x network then it will likely cause issues, so our goal throughout is going to be blocking metadata traffic on 192.168.50.5. There are also many ways to do this. We are going to focus on blocking outgoing traffic for Xsan on the production IP. Another way to accomplish this might be to block all traffic for metadata for the network range of 192.168.50.x. There are many different ways to do this (including using managed switches) but this way has been working for us.
DISCLAIMER : Be very careful playing with the firewall on a headless server. The last thing you want to do is block yourself from being able to ARD or SSH into a system you are trying to administer. Also try to make sure that you do this stuff while you have the opportunity for a little down-time as you might need a little time to troubleshoot. Also, be sure to backup the service settings before making any changes. To do this, drag the small icon in the lower right-hand corner of the firewall service screen to the desktoop. When you see the + icon let go of the icon and it will save the service settings. If you need to restore this later you can drag the file back over to the Server Admin screen.
Using Firewall in Server Admin
First we’re going to enable Firewall (ipfw):
Open Server Admin from /Applications/Server
Authenticate to the server you are configuring
Click on Firewall under the Computers and Servers List
Click on Settings
Click on Services
Click on the Start Service button in the toolbar of Server Admin
Now we’re going to define the ports to block (see Figure 1):
Click on the + sign to add a service
Enter Xsan under the Name field
Enter “536, 537†(without the quotes) under the Port field
Leave the Protocol set to TCP
Click on OK
Figure 1 – Defining Ports
Now we’re going to define the addresses to block this port for (see Figure 2):
Click on Address Groups
Click on the + sign to add an address group
Name the address group Production IP
Click on the + sign beside Addresses in group
Remove Any if it is present
Type the IP address of your internal IP for the production network
Click OK
Figure 2 – Creating the Address Group
Now we’re going to create the rule for blocking the Xsan ports (see Figure 3):
Click on the Advanced tab
Uncheck all of the rules that deny if you want to first test your base config (you can always go back and add rules once you’ve become more familiar with the Firewall service)
Click on the + button
In the Action drop-down choose the Deny option
In the Protocol drop-down choose the TCP option
In the Service drop-down choose the newly created Xsan service
In the Address drop-down choose the newly created Production IP address
In the Destination drop-down select Any
The Ports should read 536, 537 as this is what we defined earlier
In the Interface drop-down choose Out
Click OK
Figure 3 – Configuring the Rule
Using the IPFW from the Command Line
Personally I find it much easier to do most of this using the command line. For systems not running Mac OS X Server you will need to use the command-line. To do this you would use the ipfw.conf file that is created to define what types of traffic are allowed. The ipfw.conf file is located at /private/etc/ipfilter/ipfw.conf.
The default ipfw.conf file looks like something like this:
# ipfw.conf.default – Installed by Apple, never modified by Server Admin app
#
# ipfw.conf – The servermgrd process (the back end of Server Admin app)
# creates this from ipfw.conf.default if it’s absent, but does not modify it.
#
# Administrators can place custom ipfw rules in ipfw.conf.
#
# Whenever a change is made to the ipfw rules by the Server Admin application and saved:
# 1. All ipfw rules are flushed
# 2. The rules defined by the Server Admin app (stored as plists) are exported to
# /etc/ipfilter/ipfw.conf.apple and loaded into the firewall via ipfw.
# 3. The rules in /etc/ipfilter/ipfw.conf are loaded into the firewall via ipfw.
# Note that the rules loaded into the firewall are not applied unless the firewall is enabled.
#
# The rules resulting from the Server Admin app’s IPFirewall and NAT panels are numbered:
# 10 – from the NAT Service – this is the NAT divert rule, present only
# when he NAT service is started via the Server Admin app.
# 1000 – from the “Advanced†panel – the modifiable rules, ordered by their
# relative position in the drag-sortable rule list
# 12300 – from the “General†panel – “allow— rules that punch specific holes
# in the firewall for specific services
# 63200 – from the “Advanced†panel – the non-modifiable rules at the bottom
# of the panel’s rule list
#
# Refer to the man page for ipfw(8) for more information.
#
# The following rules are already added by default:
# #add 01000 allow all from any to any via en0
#add 01010 deny all from any to 127.0.0.0/8
#add 01020 deny ip from 224.0.0.0/4 to any in
#add 01030 deny tcp from any to 224.0.0.0/4 in
#add 12300 (â€allow†rules from the “General†panel)
#…
#add 65534 deny ip from any to any
First, we’re going to allow all traffic for both controllers. For this example doing this will make sure that we don’t have any problems with connecting to Xsan Admin (port 311) or ARD. To do this, take out the # for the line that reads:
#add 01000 allow all from any to any via en0
By removing a # from the beginning of a line you are uncommenting the line or enabling the rule. I like to go through and put a commented line for some rules that are complicated so that other techs at my company can figure out what they do if they need to troubleshoot something that I’ve done. To do this you just begin the line with a #.
Also the en0 here might be en1 or lo1 for some metadata controllers. You can use the Network Utility to determine which adapter is using which ethernet port name.
Next you will create another rule that adds a line that denies outgoing traffic for ports 536 and 537 (Sun Grid Engine Qmaster) over the ethernet adapter being used for the metadata network. This adapter can easily be identified using the Network System Preference. This rule looks something like this assuming that 192.168.50.5 is the IP address the server is using for its metadata network interface:
add 65534 deny tcp from 192.168.50.1 to any dst-port 536, 537 out
Reading from left to right we are telling ipfw to add a rule with a unique ID that denies TCP traffic from the IP address of the metadata network interface that is running on ports 536 or 537 for outgoing traffic. The unique ID number also acts as a priority that rules should be run in. A full sample file could be as short as:
#The following line enables network traffic on the Production Network
add 01000 allow all from any to any via en0
#The following line enables network traffic on the Metadata Network
add 01000 allow all from any to any via en1
#The following line disables Xsan Traffic for the Production Network
add 65534 deny tcp from 192.168.50.5 to any dst-port 536, 537 out
Finally, once you are sure that your configuration is good, use Server Admin to enable the Firewall service as we described at the beginning of this article. Remember not to block port 311 or you will not be able to use Xsan Admin to administer the client.
Just in case you use Linux clients
For Linux clients of Xsan I find that using iptables is a great way to accomplish this same task. The two commands to create the deny rule using iptables would be something like this assuming that eth0 is your production network:
Iptables –A PREROUTING –o eth0 -s any –p tcp –dport 536 -j DROP
Iptables –A PREROUTING –o eth0 -s any –p tcp –dport 537 -j DROP
If iptables is not already started you can type:
Service iptables start
The command to view active rules for iptables is:
Chkconfig –list iptables
Saving Windows for Last
Windows XP and Server 2003 have equally as robust firewall features. To limit traffic over ports you would use Windows Firewall for Windows XP and Routing and Remote Access for Windows Server 2003. While Windows Server 2003 is a little beyond the scope of this document, I can help you get started. For Windows XP:
Click on Start
Click on Connect To
Click on Show All Network Connections
Click on Change Windows Firewall Settings (see Figure 4)
Figure 4 – Network Connections
Once you have your Windows Firewall control panel open, click On and make sure that the Don’t allow exceptions check-box is unchecked (see Figure 5)
Click on Advanced
Use the check-boxes under Network Connection Settings to disable the Firewall for the Metadata Controller (see Figure 6)
Figure 5 – Enabling Windows Firewall
Figure 6 – Windows Firewall Advanced Settings