As of OS X 10.9 (and in many cases more importantly in OS X Server for 10.9 and higher), OS X now performs ARP cache validation when trying to pass traffic over a router. If you are double NAT’d/use redundant gateways then the traffic can be interpreted as network redirection and cause some pretty bad packet loss/latency. You can disable this feature by turning off net.link.ether.net.arp_unicast_lim using sysctl:
sysctl -w net.link.ether.inet.arp_unicast_lim=0
That will only disable unicast arp validation until the next reboot. If it fixes a latency problem you’re having then you can go ahead and make it permanent by adding the following line into /etc/sysctl.conf:
net.link.ether.inet.arp_unicast_lim=0
If you’re still having issues with latency, you should turn it back on. To enable it again, repeat, swapping the 0 with a 1.
