I get a lot of questions about why a Windows computer cannot see a Mac OS X Server on the network when Macs can – or put differently why a Windows computer cannot browse the network for a Mac OS X Server. The first thing I always ask is can you ping it by IP address AND by name. If you can ping the host by IP address then you know the host can communicate with the client. If you cannot then you could be having DNS issues or NetBIOS issues. If you use DNS then that’s easy enough to fix, create or edit the record for the server and verify that your client is using the right server for DNS. As with the Mac there are two names for Windows hosts: an SMB name (NetBIOS name) and a DNS name. Fixing your DNS naming issues might not fix a NetBIOS naming issue. You can use NBTStat to troubleshoot these issues further.
NetBIOS attempts to resolve names by first checking the local cache. If the name is not found the system will then check an LMHosts file (similar to a hosts file on a Mac but using the single-line domain naming scheme common in LanMan environments – located at %SYSTEMROOT%SYSTEM32Driversetc) and if a name is not found it then sends a broadcast that looks for the name. This is the default behavior of NetBIOS and can be changed, which we will get into later when we discuss nodes. But for now it is worth pointing out that the issues that you might be encountering could be with the broadcast message. This message is received and processed by all machines on the network, requesting that each system identify itself. Once upon a time it was effective for small networks but since it’s broadcast it’s non-routable (doesn’t work over routers and firewalls or usually across subnets). Additionally, NetBIOS broadcasts can be problematic as they can cause a lot of network traffic if you have a lot of hosts running SMB on your network. NetBIOS broadcasts can also cause a security risk, which is one reason that the Windows Firewall will typically block this type of traffic, causing hosts to not answer the broadcast.
Another issue that can arise with NetBIOS is Master Browsers. If you’re having browsing issues (which we once called browser wars) then chances are, somewhere on your network you’re going to find a client that’s throwing errors about master browser elections… Sometimes thousands of these errors. Browser services maintain a list of the domain or workgroup a computer is in and the protocol being used for each computer on the network. On each subnet a master browser is elected between the systems. If there is a Windows domain controller then it will typically win the election; beyond that is a crap shoot where the newest OS patch/version typically wins (there are other factors as well, but that’s the probably-too-simplified version of it). Frequent elections can cause NetBIOS storms on the network, failed NetBIOS name resolution and potentially corrupt data in the LMHOSTS file.
The quickest and easiest way to resolve these NetBIOS naming issue is to deploy a WINS server and a quick and easy way to solve the browsing issue is to deploy a Domain Master Browser. This is one place where old school Microsoft architecture was actually really nice (although not truly scalable which is why it was replaced with Active Directory Integrated DNS) and to integrate it into Mac OS X Server typically takes less than 10 minutes, about the same amount of time it usually takes in Windows NT Server, 2000 Server and 20003 Server (Mixed). To get started choose the box you’re gonna’ use and fire up Server Admin. Click on the SMB service for the host in question in the Servers list and then click on the Settings icon in the Server Admin Toolbar. Next, go ahead and check the box for Domain master Browser and set the Enable WINS server bullet in the WINS Registration: field as you can see in the following screen.
Now click on the Save button. You can now set the WINS setting on the Windows clients (in the TCP/IP settings). Now, this could end up being cumbersome given a large number of clients. So you might want to go ahead and roll it using DHCP, which might even give you more options from a GUI (getting back to that node thing mentioned earlier). To set it up, stay in Server Admin, click on the DHCP service and then click on the Subnets icon in the Toolbar. Here, click on the subnet to edit (remember you might need to do this on each subnet to hit all the clients) and then click on the WINS tab as seen here:
Now you can enter the IP for the WINS server and if you have the resources the backup server into the WINS/NBNS Primary Server: and WINS/NBNS Secondary Server: fields respectively. You can also enter an IP for a server to perform NBDD (NetBIOS Datagram Distribution – aka Option 46), which I’ve actually never seen implemented correctly once. I think MS may not have gotten this part right and ergo the Samba team might not have either (but feel free to correct me if you have a screen shot to prove it), but it was finalized in the RFC and not really implemented.
Finally, you can select the NBT Node Type. This is where you can actually control the order that NetBIOS uses for name resolution. As can be seen below there are four types of name resolution, b-node (broadcast), p-node (peer), m-node (mixed) and h-node (hybrid). B-node essentially disables WINS, first attempting to resolve from LMHOSTS cache and then using broadcast traffic and then using the LMHOSTS file. P-node uses WINS, LMHOSTS and cache but will not perform any broadcast lookups on the network. M-node is not common with other implementations of WINS, but allows you to set naming to try broadcasts first and then look to WINS. Finally, h-node will try to resolve over WINS and then revert back to broadcasts. 
Why all the choices? Well, it gives you options. For example, in very large networks or very saturated subnets of large networks eliminating as much broadcast traffic can be a very effective means of reducing load on switches. Either way, feel free to just leave the Node Type as the fifth option not previously mentioned, which is Not Set. This will leave it to Windows defaults.
Once you’ve deployed all this you might get a really pissy call from a Windows admin somewhere that you’re causing their WINS server to have a browser election issue. Or you might be proactive and ask if they have a WINS server you can piggy back off of. If so then back in the SMB screen you might notice that whole Register with WINS server: option – use that instead of rolling your own.
Finally, systems will typically simply (did I just use two adverbs back to back, ick) register it’s name and address with the lmhosts file. However, it is possible to customize your lmhosts file, just like it’s possible to customize your hosts file. To do so read the lmhosts man page and then touch an lmhosts file into /etc (not /etc/samba) and customize per the instructions in the man page.
