macOS 14.4 and the Addigy MDM Watchdog

A new change in macOS has removed support for commands that are used to restart various system services in the OS. This will cause scripts that utilize these commands to fail on macOS devices that upgrade to macOS 14.4, including the Addigy MDM Watchdog.

What’s changed?

In the release of macOS 14.4, Apple has deprecated support for several commands, including  launchctl kickstart. This update will cause all existing scripts that utilize the launchctl kickstart to fail on machines running macOS 14.4, and Apple advised using the kill command instead when attempting to restart services on macOS devices. However, in testing, the kill command has not been effective and instead, Apple recommended restarting devices to force the services to restart in lieu of the kickstart command.

This directly impacts the Addigy MDM Watchdog, which relies on the launchctl kickstart command to restart the MDM and software update processes on stuck macOS devices. When these processes become stuck, devices cannot carry out MDM-based actions and perform System Updates. As a result, the MDM Watchdog will no longer be able to restart the essential services on devices running macOS 14.4 or later. However, it should continue to function on macOS devices running an older version of the OS.

How does this affect MDM Watchdog?

Available as part of the Addigy management Agent, as well as a free self-deployed agent, the MDM Watchdog was created to help ensure the reliability of macOS Updates via MDM. When we released macOS Updates via MDM last year, we found an abnormally high rate of failed updates. Upon digging into the root cause of these failures, we often found that the update command sent via MDM was getting stuck either in the process of reaching the macOS device and being acknowledged by the MDMClient process, or it was getting stuck during the download, preparation, and install stage run by the softwareupdated (Software Update Daemon).

As we wanted to make updates more reliable, we began creating the agent that would eventually become MDM Watchdog. This new agent did several things: first identify if the device was in fact stuck on an update, second to try and determine the root cause of that device sticking point, and then finally systematically remediate that stop point with a reboot of just the affected service(s).

The workflow that MDM Watchdog followed to remediate these stuck updates:

1. Read the Unified System Logs for the following:
   1. MDM Client Process for the last 24 hours in info and debug mode
      1. /usr/bin/log show –predicate “process=’mdmclient'” –last 24h –info –debug
         1. If for the last 24 hours, there has been any logged error about Unable to create MDM identity, we consider the MDM identity itself to have possible issues.
   2. Managed Client logs
      1. /usr/bin/log show –style syslog –predicate “process=’mdmclient’ or subsystem=’com.apple.ManagedClient'”
         1. If the very last response we have received from the device is a message containing “Processing server request” and “OSUpdate,” we assume that the OS Updates process is the last action that is stuck.
   3. MDM Client Process for the last 90 minutes
      1. /usr/bin/log show –predicate “process=’mdmclient'” –last 90m
         1. Here we are looking for a received response for an MDM command with a status 200 for the HTTP communication in the last 90 minutes. If we do not get that response, MDM is determined to be stuck and the remediation in number 2 of this workflow takes place.
            
2. If the MDM Client is determined to be stuck, MDM Watchdog then takes action:
   1. For MDM Client only issues:
      1. /bin/launchctl kickstart -k system/com.apple.mdmclient.daemon
   2. For OS Updates Issues:
      1. /usr/bin/dscacheutil -flushcache
      2. /usr/bin/killall -HUP mDNSResponder
      3. /bin/launchctl kickstart -k system/com.apple.softwareupdated
         

As a result of the changes with macOS 14.4, the critical steps in 2.1.1 and 2.2.3 where the MDM Client and softwareupdated are restarted can not take place.

How does this impact your end-users?

There will be little direct impact on your end-users, as the MDM Watchdog does not generate any form of error message or system dialogue triggered to the end-user if the Watchdog fails. 

It is more likely that you may notice machines are not receiving System Updates deployed via MDM, as the MDM Watchdog cannot forcefully restart the service when it detects that it is no longer communicating properly. A restart of the device should trigger a full restart of the associated MDM and software update processes, per Apple’s instruction.

What to do if you experience devices that are no longer updating.

It is highly recommended that devices running macOS 14.0 and newer should utilize the Apple DDM (Declarative Device Management) Updates workflow, as opposed to the MDM System Update method utilized previously. 

While DDM Updates still leverage the same software update process that Watchdog can no longer forcefully restart, DDM Updates have proved much more reliable than the previous MDM Update implementation and offer greater control over the behavior and end-user notifications for the updates. 

For more information on DDM-based updates, please see our Knowledge Base articles and previous Sudo Talks webinar:

• Addigy Sudo Talks: Introducing System Updates via DDM
• Knowledge Base: System Updates via MDM and DDM
• FAQ: System Update via MDM and DDM

Current Addigy customer? Keep up to date with the latest.

Not an Addigy customer but want to keep up on macOS 14.4 and the MDM Watchdog? Learn more.