A few people have hit me up about issues getting Windows machines to play nice with the SMB built into Yosemite Server and Windows. Basically, the authentication dialog keeps coming up even when a Mac can connect. So there are two potential issues that you might run into here. The first is that the authentication method is not supported. Here, you want to enable only the one(s) required. NTLMv2 should be enabled by default, so try ntlm:
sudo serveradmin settings smb:ntlm auth = "yes"
If that doesn’t work (older and by older I mean old as hell versions of Windows), try Lanman:
sudo serveradmin settings smb:lanman auth = “yes"
The second is that the authentication string (can be seen in wireshark) doesn’t include the workgroup/domain. To resolve this, simply include the Server name or workgroup in the beginning of the username followed by a backslash(\). So you might do this as a username if your NetBios name were kryptedserver:
kryptedserver\charles
To get that exact name, use serveradmin again, to look at the smb:NetBIOSName attribute:
smb:NetBIOSName = "kryptedserver"
