Root Cause Analysis: Intermittent Job Creation Failures
Summary
Between June 29, 2026 at 01:00 UTC and July 1, 2026 at 10:20 UTC, some Datto RMM partners on the Syrah and Vidal platforms experienced failures when creating jobs for certain component configurations. Affected job creation attempts returned an error and did not complete successfully. Jobs scheduled before the 15.0.1 release continued to run as expected.
Root Cause
A software defect, introduced in the 15.0.1 job creation implementation, caused the service to incorrectly interpret certain existing component-variable indexes. The implementation assumed indexes started at zero, which was not true for all existing configurations. As a result, job creation failed for affected configurations.
The fix removed the framework's index-reading behavior and moved index creation and persistence into Datto RMM application code. The fix was validated in pre-production and subsequently deployed to restore affected job creation.
Identification of the issue was delayed because its symptoms closely matched those of a previously known defect, where component updates and edits were causing similar errors with jobs, making it difficult to initially distinguish as a separate underlying cause. This also contributed to the delayed status page notification. We acknowledge that public notification should have occurred sooner.
Incident Timeline
Identified: June 29, 2026 at 01:00 UTC
Public Notification: June 30th, 2026 13:26 UTC
Resolved: July 1, 2026 at 10:14 UTC
Preventative Measures
To reduce the likelihood and potential impact of similar incidents, we are taking the following steps:
Enhanced test coverage: Expanding the current automated regression testing to include a broader range of historical and nonstandard component variable configurations. This will help identify similar compatibility issues before changes are released to production.
Enhanced monitoring and alerting: Enhance existing targeted monitoring for elevated job creation failure rates so that similar conditions can be detected and investigated quicker.
Release and recovery improvements: Reviewing the deployment process for urgent corrective changes to help reduce the time between successful validation and production deployment.