I originally wrote this back in 2015 as an article for troubleshooting APNs traffic on a Profile Manager server. But it turns out that troubleshooting push notification communications between macOS Server and Apple’s Push Notification is basically the same as troubleshooting the apsd client on macOS. Basically, we’re gonna’ put the APNs daemon, apsd, into debug mode. To enable APNS debug logging, run these commands:
defaults write /Library/Preferences/com.apple.apsd APSLogLevel -int 7
defaults write /Library/Preferences/com.apple.apsd APSWriteLogs -bool TRUE
Then use tail -f to watch the apsd.log file at /Library/Logs/apsd.log. Be wary, as this can fill up your system. So to disable, use these commands:
defaults write /Library/Preferences/com.apple.apsd APSWriteLogs -bool FALSE
defaults delete /Library/Preferences/com.apple.apsd APSLogLevel
krypted February 9th, 2017
Apple Configurator 2 is a great tool. But you need to debug things from time to time. This might mean that a profile is misconfigured and not installing, or that a device can’t perform a task you are sending it to be performed. This is about the time that you need to enable some debug logs. To do so, quit Apple Configurator and then write a string of ALL into the ACULogLevel key in ~/Library/Containers/com.apple.configurator.ui/Data/Library/Preferences/com.apple.configurator.ui.plist:
defaults write ~/Library/Containers/com.apple.configurator.ui/Data/Library/Preferences/com.apple.configurator.ui.plist ACULogLevel -string ALL
To disable, quit Apple Configurator and then delete that ACULogLevel key:
defaults delete ~/Library/Containers/com.apple.configurator.ui/Data/Library/Preferences/com.apple.configurator.ui.plist ACULogLevel
krypted April 19th, 2016
Spotlight just kinda’ works. Except when it doesn’t. Which is luckily pretty rare, for the use cases that Spotlight was designed for. But when it doesn’t work, you have a few tools that I’ve highlighted over the years to help you out, including articles on shared volumes, manually indexing, disabling Spotlight, and a few others. But what if you need to go in more depth to isolate an issue? For this, Apple has provided us with a tool called mddiagnose, in /usr/bin. In the following command, we’ll run an mddiagnose to dump a bunch of system statistics that we can then look at. Here, we’ll do that to a folder called test in our current working directory:
/usr/bin/mddiagnose -f test
The output is then test.mdsdiagnostic, a directory with a CrashReporter, spindump, Samples, DiagnosticReports, a few system.log exports, and a diagnostic.log.
You can then view your log using the more command (or cat or less or whatevers)
Here, you’ll see the output of a bunch of scripts that were run. I find that this is the most informational aspect of what I get from the mddiagnose output. Every time I’ve actually fixed an issue here, it’s been with this output.
The other aspect of mddiagnose that I’ve found useful is checking permissions and paths. Here, you can answer the simple question of whether mdutil has permissions to check a path. We’ll do so using the -p option:
mddiagnose -p /Library/Application\ Support/Appifitizer
krypted December 15th, 2015
I’ve seen a few instances where an upgrade caused Final Cut to run kinda’ strangely. To resolve, I’ve just been doing a quick reinstall of Final Cut. To do so:
Once done, go back to the Mac App Store and reinstall Final Cut and open it. Those folders you just tossed out will get re-created. Your toolbars and other customizations are likely to be gone, so you’ll have to spend a few minutes getting your workspace back to the way you had it, but if Final Cut was acting oddly it should be back to normal.
krypted January 21st, 2015
I recently purchased a new TV (actually won, but that’s aside from the point). I put the DirecTV receiver on there and it worked like a charm. Then I put the Apple TV on and it appeared to work like a charm. But when the screensaver kicked in, the colors inverted. Sometimes I’d see lines across the screen and other times the Apple TV would get weird and just be blurry. I knew immediately that I was sending it too much. Turns out the new TV couldn’t do less than 1080p and the old Apple TV couldn’t do anything higher than 720p. To confirm, I looked up the serial number. All Apple TVs have Wi-Fi (up to 802.11n), 10/100 Ethernet, optical audio and an Infrared receiver for the remote control. So, here’s some information on model-specific connectivity to your other equipment:
krypted April 27th, 2014
Powermetrics is a command that shows very specific information about what’s using your systems power. This makes it handy for troubleshooting what processes are taking up CPU, GPU, etc. To run a basic iteration of the command, we’re going to look at a usage summary report, using the –show-usage-summary option:
powermetrics -a --show-usage-summary
The output is as follows:
Machine model: MacBookAir6,2
OS version: 13A598
*** Sampled system activity (Sun Dec 1 23:04:13 2013 -0600) (5006.04ms elapsed) ***
*** Running tasks ***
Name PID CPU ms/s User% Deadlines (<2 ms, 2-5 ms) Wakeups (Intr, Pkg idle) java 42754 541.18 12.92 0.00 0.00 12.39 0.00 kernel_task 0 452.41 0.00 141.63 0.00 386.13 0.00 blued 45 39.58 94.31 0.00 0.00 0.00 0.00 cfprefsd 48518 23.67 1.31 0.00 0.00 0.00 0.00 Knock 44431 16.41 90.28 0.00 0.00 0.00 0.00 com.apple.WebKit 16795 15.29 92.11 0.00 0.00 2.20 0.00 UserEventAgent 385 15.00 91.51 0.00 0.00 0.00 0.00 UserEventAgent 13 14.47 88.45 0.00 0.00 0.20 0.00 com.apple.WebKit 1048 11.61 95.80 0.00 0.00 2.40 0.00 com.apple.WebKit 94216 8.98 90.29 0.00 0.00 6.19 0.00 com.apple.WebKit 997 8.24 82.89 0.00 0.00 40.55 0.00 WindowServer 194 6.04 59.63 1.20 0.80 2.20 0.00 com.apple.WebKit 1154 5.52 68.49 0.00 0.00 42.15 0.00 hidd 90 5.49 72.13 0.00 0.00 0.00 0.00 com.apple.WebKit 1027 5.40 65.94 0.00 0.00 41.95 0.00 Terminal 44349 5.31 87.13 0.00 0.00 1.40 0.00 com.apple.WebKit 1054 5.01 67.33 0.00 0.00 41.35 0.00 prl_disp_service 369 3.15 97.11 0.00 0.00 2.20 0.00 distnoted 386 3.14 97.03 0.00 0.00 0.00 0.00 com.apple.WebKit 16780 1.86 60.00 0.00 0.00 16.78 0.00 com.apple.WebKit 55388 1.75 69.71 0.00 0.00 0.20 0.00 com.apple.WebKit 1505 1.71 87.44 0.00 0.00 3.00 0.00 Remote Desktop C 48336 1.69 32.47 0.00 0.00 98.88 0.00 Dropbox 508 1.36 79.03 0.00 0.00 0.60 0.00 distnoted 22 1.13 87.55 0.00 0.00 0.20 0.00 com.apple.WebKit 64843 1.07 93.07 0.00 0.00 1.00 0.00 powermetrics 48592 0.96 28.21 0.00 0.00 0.20 0.00 Safari 431 0.90 44.27 0.20 0.00 1.00 0.00 com.apple.WebKit 16799 0.88 90.05 0.00 0.00 1.60 0.00 com.apple.WebKit 93194 0.55 67.81 0.00 0.00 4.59 0.00 networkd 44 0.45 11.35 0.00 0.00 0.00 0.00 fseventsd 41 0.42 44.20 0.00 0.00 3.40 0.00 mds 81 0.37 44.11 0.00 0.00 2.20 0.00 com.apple.WebKit 1215 0.28 73.38 0.00 0.00 2.00 0.00 IPSecuritasDaemo 348 0.28 55.84 0.00 0.00 1.00 0.00 com.apple.WebKit 93143 0.27 68.84 0.00 0.00 2.00 0.00 PluginProcess 3573 0.26 77.60 0.00 0.00 1.40 0.00 com.apple.WebKit 16781 0.23 76.76 0.00 0.00 1.00 0.00 com.apple.WebKit 1679 0.23 69.04 0.00 0.00 1.60 0.00 usbmuxd 66 0.20 29.81 1.40 0.00 1.40 0.00 dbfseventsd 43670 0.19 26.12 0.00 0.00 0.00 0.00 Finder 434 0.18 22.45 0.00 0.00 0.00 0.00 com.apple.WebKit 16763 0.17 69.84 0.00 0.00 1.60 0.00 notifyd 17 0.16 21.54 0.00 0.00 0.00 0.00 galileod 107 0.15 41.56 0.00 0.00 1.00 0.00 dbfseventsd 43671 0.12 8.20 0.00 0.00 0.00 0.00 mds_stores 237 0.11 34.95 0.00 0.00 0.40 0.00 distnoted 585 0.09 67.90 0.00 0.00 0.00 0.00 com.apple.WebKit 33839 0.08 69.56 0.00 0.00 0.60 0.00 pacemaker 78 0.07 28.13 0.00 0.00 1.00 0.00 com.apple.WebKit 3572 0.05 67.81 0.00 0.00 0.40 0.00 Mail 428 0.05 65.29 0.00 0.00 0.40 0.00 dbfseventsd 43669 0.05 7.35 0.00 0.00 1.60 0.00 com.apple.MailSe 89428 0.04 57.83 0.00 0.00 2.00 0.00 loginwindow 84 0.02 65.43 0.00 0.00 0.00 0.00 com.apple.WebKit 33836 0.02 39.42 0.00 0.00 0.20 0.00 ALL_TASKS -2 1204.27 19.05 144.43 0.80 734.51 0.00 **** Battery and backlight usage **** Backlight level: 563 (range 0-1024) **** Network activity **** out: 1.40 packets/s, 399.12 bytes/s in: 1.40 packets/s, 265.28 bytes/s **** Disk activity **** read: 2.40 ops/s 291.28 KBytes/s write: 4.00 ops/s 1867.16 KBytes/s **** Interrupt distribution **** CPU 0: Vector 0x49(MacBookAir6,2): 68.92 interrupts/sec Vector 0x92(IGPU): 113.46 interrupts/sec Vector 0x94(XHC1): 14.18 interrupts/sec Vector 0x97(ARPT): 5.79 interrupts/sec Vector 0x9e(SSD0): 5.59 interrupts/sec Vector 0xdd(TMR): 721.53 interrupts/sec Vector 0xde(IPI): 4743.47 interrupts/sec CPU 1: Vector 0xdd(TMR): 136.44 interrupts/sec Vector 0xde(IPI): 444.26 interrupts/sec CPU 2: Vector 0xdd(TMR): 718.93 interrupts/sec Vector 0xde(IPI): 3874.52 interrupts/sec CPU 3: Vector 0xdd(TMR): 136.24 interrupts/sec Vector 0xde(IPI): 170.79 interrupts/sec **** Processor usage **** Intel energy model derived package power (CPUs+GT+SA): 9.79W LLC flushed residency: 0% System Average frequency as fraction of nominal: 174.83% (2272.84 Mhz) Package 0 C-state residency: 0.00% (C2: 0.00% C3: 0.00% C6: 0.00% C7: 0.00% C8: 0.00% C9: 0.00% C10: 0.00% ) Core 0 C-state residency: 1.95% (C3: 0.00% C6: 0.00% C7: 1.95% ) CPU 0 duty cycles/s: active/idle [< 16 us: 48421.92/53709.13] [< 32 us: 1616.05/1451.05] [< 64 us: 3725.90/942.06] [< 128 us: 2403.50/430.28] [< 256 us: 328.60/15.78] [< 512 us: 29.56/0.00] [< 1024 us: 8.19/0.00] [< 2048 us: 7.79/0.00] [< 4096 us: 2.60/0.00] [< 8192 us: 2.40/0.00] [< 16384 us: 0.60/0.00] [< 32768 us: 1.20/0.00] CPU Average frequency as fraction of nominal: 174.99% (2274.86 Mhz) CPU 1 duty cycles/s: active/idle [< 16 us: 59157.15/44087.35] [< 32 us: 565.72/11942.98] [< 64 us: 233.72/2441.65] [< 128 us: 106.07/1535.94] [< 256 us: 33.96/91.49] [< 512 us: 6.79/4.19] [< 1024 us: 2.00/4.19] [< 2048 us: 1.20/1.20] [< 4096 us: 3.40/2.00] [< 8192 us: 1.20/1.20] [< 16384 us: 0.60/0.40] [< 32768 us: 0.80/0.20] CPU Average frequency as fraction of nominal: 174.88% (2273.49 Mhz) Core 1 C-state residency: 1.63% (C3: 0.00% C6: 0.00% C7: 1.63% ) CPU 2 duty cycles/s: active/idle [< 16 us: 52772.26/58135.39] [< 32 us: 2563.70/1559.12] [< 64 us: 2825.19/779.46] [< 128 us: 2252.48/405.91] [< 256 us: 427.08/17.78] [< 512 us: 32.36/0.00] [< 1024 us: 10.39/0.00] [< 2048 us: 7.39/0.00] [< 4096 us: 3.20/0.00] [< 8192 us: 1.40/0.00] [< 16384 us: 1.40/0.00] [< 32768 us: 0.80/0.00] CPU Average frequency as fraction of nominal: 174.76% (2271.94 Mhz) CPU 3 duty cycles/s: active/idle [< 16 us: 45270.92/29302.61] [< 32 us: 507.99/9788.78] [< 64 us: 320.81/5017.34] [< 128 us: 85.10/1920.88] [< 256 us: 31.76/192.17] [< 512 us: 6.39/2.20] [< 1024 us: 2.40/4.79] [< 2048 us: 2.60/3.00] [< 4096 us: 4.99/2.00] [< 8192 us: 0.80/1.00] [< 16384 us: 0.40/0.00] [< 32768 us: 0.60/0.00] CPU Average frequency as fraction of nominal: 174.46% (2267.93 Mhz) **** GPU usage **** GPU 0 name IntelIG GPU 0 C-state residency: 99.69% (0.02%, 99.67%) GPU 0 P-state residency: 1000MHz: 0.00%, 950MHz: 0.00%, 900MHz: 0.00%, 850MHz: 0.00%, 800MHz: 0.00%, 750MHz: 0.00%, 700MHz: 0.00%, 650MHz: 0.00%, 600MHz: 0.00%, 550MHz: 0.00%, 500MHz: 0.00%, 450MHz: 0.00%, 400MHz: 0.00%, 350MHz: 0.31%, 300MHz: 0.00%, 250MHz: 0.00%, 200MHz: 0.00% GPU 0 average frequency as fraction of nominal (200.00Mhz): 0.55% (1.09Mhz) GPU 0 GPU Busy 0.31% GPU 0 FB Test Case 0.00% *** Sampled system activity (Sun Dec 1 23:04:18 2013 -0600) (5004.69ms elapsed) ***
You can also hide CPU duty cycle data with the --hide-cpu-duty-cycle option, GPU duty cycle data with the --hide-gpu-duty-cycle option and show information about the entire uptime with --show-initial-usage. Overall, there's a good bit of information that can be obtained between powermetrics and systemstats, provided you have the time to parse the data and find the specific pieces of information that are pertinent to your debugging attempts.
krypted December 4th, 2013
“My computer sometimes just runs slow,” “the fan on my laptop won’t turn off sometimes,” and “my network connection keeps dropping.” These are amongst the most annoying off problems to solve for our users because they are intermittent. And to exacerbate things, many of these users have these problems at home or at remote locations, making it difficult for systems administrators to see them.
There is something I use in these cases, though, that has helped isolate these problems from time to time. Simply tell users to Control-Option-Command-Shift-Period when they have these problems. Doing so will run the sysdiagnose command and then open a Finder window with the output of the command. Sysdiagnose takes a quick snapshot of many common logs and performance data, zips it up and opens a Finder window, pointing to where it is (/var/tmp with the filename containing a date stamp of when the command was run). This file contains output from allmemory, lsof, top, netstat, sysctl, spindump, fs_usage, system_profiler, mount, airport, odutil and many others. Each is in its own log and easy to navigate.
When running /usr/sbin/sysdiagnose from the command line there are a couple of options. My favorite is -f (which I think must be short for favorite) which allows me to write to my file to a directory I specify rather than some random object in a tmp directory. You can also get even more output using -t. Verbose logging is obtained using -h and passing a pid will also provide information about the pid. So let’s say that process 10883 is giving me some problems. I could run the following to get some good output on my desktop:
sysdiagnose -h -t -f ~/Desktop
Anyway, hope you enjoy!
krypted May 13th, 2013
In Lion Server, Open Directory can be managed in one of three ways: using the Server application the Server Admin application or using the command line utilities. Configuring Open Directory has never been easier than it is in the Server application, though. As we looked at in a previous article, setting up an Open Directory master should be done using the Server application. But setting up an Open Directory replica should be done using the Server Admin application. The Server Admin application is not installed when you buy OS X Server on the App Store and so it can be obtained here.
But first (or while that’s downloading even), open the Server application. If this is the first time that you’ve opened the Server application then you’re in for a bit of a wait. This is a nice time to grab yourself the first shot of Jäger of the day. According to your internet speed, you could end up with 3 or 4 of these. That’s fine though, the new Open Directory makes much more sense afterwards.
When you first open and start using the Server application, you’re creating local users. The Server application automatically creates local users until you setup Open Directory. Before you set up Open Directory as a Replica on the system, it should have a static IP address and a name in the DNS servers that the server uses (forward and reverse lookups for said address). The Server application has a Next Steps drawer. Clicking on the drawer and then the Configure Network button brings up a screen that will complain if your DNS has any problems. If DNS is working great, then the Configure Network section of the Next Steps drawer will appear as follows:
Not to get off topic on the hostname/dns/etc thing, but when you click on Network, if you decide to change names before you promote to an Open Directory Master/Replica, clicking on Edit for the Host Name, you should almost always click on the third option, Host Name for Internet…
While the Server app is cool, it caches stuff and I’ve seen it let things go threat shouldn’t be let go. Therefore, in order to make sure that the server has such an address, I still recommend using changeip, but I also recommend using the Server application. In Lion, I’ve seen each find things that other misses. To use changeip:
sudo changeip -checkhostname
The address and host names should look correct and match what you see in the Server application’s Next Steps drawer.
Primary address = 10.0.0.1
Current HostName = mdm.krypted.com
DNS HostName = mdm.krypted.com
The names match. There is nothing to change.
dirserv:success = “success”
Provided everything is cool with the hostname, open the Server Admin application from /Applications/Server. Then click on Settings in the application’s toolbar. At the Settings screen, click on Services. Click on the checkbox for the Open Directory Service and click Save to see the Open Directory service appear in the Server Admin sidebar. Then, click on Open Directory in the Server Admin sidebar and then click on the Change… button to bring up the Open Directory Assistant.
At the Choose Directory Role screen, click on Set up an Open Directory replica and then click on the Continue button.
At the Replica and Certificate Authority screen, provide the name or IP address of the Open Directory master in the IP address or DNS name of master field. Actually, just use the name. If you can’t find the Open Directory Master by name, then you should really fix that before moving forward. Also provide the Open Directory administrative user name in the Domain administrator’s short name field and that account’s password in the Domain administrator’s password field. If you have any problems, make sure you can ssh into the Open Directory master using this account.
Also, new in Lion, there’s a CA administrator’s email address field. Put in here, what you put into the Organization Information field back when you promoted the master (screen shown for posterity).
If you’ve lost track of the email address you used, keep in mind that the SSL certificate can be used to grab that information. Open Keychain Access, click on Certificates, search for the host name of the Master (this is all from the master, btw) and then do a Get Info and you’ll see the Email Address used.
Anyway, back to the Open Directory Assistant on the new Replica. Click on the Continue button and finish the wizard to complete promoting the replica. That’s it. Don’t forget to check your logs when the promotion is complete.
I’ve been finding that there are a lot of issues with promoting Replicas in Lion so far. This has meant bad directory data (import + export), bad DNS, security policies, using a bad username and password combination (not the systems fault) and other issues. To fix the bad directory data, you have to import and export (in my experience not an archive and restore but an actual export and import, losing all passwords in the process). The Next Steps drawer can guide you through the host names/DNS issues. For security policies, I’ve found the following command to work for me (run on the master):
slapconfig -setmacosxodpolicy -binding enabled
For the username and password issues (the errors don’t always tell you what is or is not a password problem) I have found using dscl or even Workgroup Manager to test the login is an important step.
You can also still use slapconfig for Open Directory replicas, a great way to get a lot of detailed information. For example, one time, the replica promotion was failing because the server was a member server in a domain; however, using slapconfig -getstyle the server simply reported as Standalone. To promote a replica, you will define want to make sure to include the new –certAdminEmail option, followed by the email address on that certificate of the master. This is then followed with the address and the admin username of the master. For example:
slapconfig -createreplica --certAdminEmail email@example.com odm.pretendco.com diradmin
When slapconfig runs, it will give you a detailed account of where it failed and why.
Finally, I have noticed that some machines fail in the Server Admin GUI and Server Admin simply doesn’t show that the machine failed, but instead just makes the system a member to the server. When this happens, I have always had to clean install the system in order to get it to promote to a replica again, properly. To make sure a replica is indeed a replica, consult slapconfig:
Now is when you get to have a little more Jäger. This whole process hopefully only took about 5 to 10 minutes, so it’s about time anyways. If the process took longer, then I hope you didn’t wait until now for round 2. Later, we’ll discuss directory trees and using those as a means of building sites. For that, you might want to move onto something a bit stronger, like mescaline.
krypted March 1st, 2012
There are a number of ways to troubleshoot network connections on (or using) an iOS device. These can be common troubleshooting steps that you might run from the command line or a third party app on a desktop computer or they could be specific to testing the network environment for an iOS device. Some of these apps are even free.
One of the most common tasks that most administrators routinely do to test both DNS resolution and connectivity is pinging something. Ping Lite comes with a function to show your IP, a ping tool, a tool to ping the subnet, the ability to run trace routes and for good measure a little telnet love as well. Not bad for the fat price of nothing. Developed by MochaSoft, Ping Lite is a must for anyone who does any kind of network troubleshooting, unless you’re paying good money for a more robust tool!
Ping Lite is a great tool for isolating whether you’re having connectivity problems to an IP address. However, if Exchange’s auto discover isn’t working or some other
One of my favorite tools for finding things on the network, Bonjour is a multicast tool and what many of the features meant to be used in a home where zero configuration networking is important
I think that one of the more common tasks in troubleshooting network connections is to determine whether Internet speed is satisfactory. Satisfactory is a relative term. Both relative to the expected performance and relative to the perception of users. For example, the bandwidth that a user is getting on a device may exceed the expected performance based on the speed provided by the DSL, cable modem or other WAN connection provided. However, that speed may be less than what the user’s would like (one can never have enough bandwidth!).
ezShare is a nice little tool that lets administrators log into shares of various types. The cool thing about this little tool is that you can connect via SSH, FTP, WebDAV, S3, Google Docs, Box.net, SMB/CIFS, or NFS. This allows you to test WebDAV from a different tool if you’re having a problem opening WebDAV connections from within Pages, test the speed of downloading a document from a FTP site, check Google Docs or Box.net connectivity and even see if that file server is available when users call in with problems connecting to SMB/CIFS shares on Windows servers.
If you have an Apple AirPort acting as a WAP or the gateway to your office/home then this little app is awesome. Apple has eased the setup process for their Wireless Access Points to the point that you can set the entire thing up, change settings and even troubleshoot the odd connectivity issue without ever touching a desktop computer. AirPort Utility is also a great way to test whether you can connect to shares hosted by devices and update passwords on the fly.
krypted February 13th, 2012
Posted In: iPhone