Root Cause Analysis: 15.1 Vidal Maintenance and Activity Log Service Impact 

Summary 

On August 18, 2026, a scheduled Datto RMM 15.1 maintenance activity for the Vidal environment was not completed within the planned maintenance window of 08:00–12:00 UTC and remained in progress beyond the scheduled end time. Service stabilized at 13:32 UTC, and the extended maintenance window concluded at 14:00 UTC. Customers were unable to access the service during the extended maintenance period. 

Between August 18, 2026, at 14:15 UTC and August 21, 2026 13:17 UTC, after 15.1 was deployed and maintenance ended, some customers experienced intermittent Activity Log and Device Activity failures, elevated query times, and timeouts. The service was mitigated after workload rebalancing, increased cluster capacity, and an application hotfix that prevented reuse of connections already closed by OpenSearch. 

Root Cause 

The 15.1 release event involved two related but distinct technical conditions:  

• The maintenance overrun occurred when an automated database change required a lock on a heavily used device table while devices were reconnecting after the upgrade. This prevented the database change from completing within the planned maintenance window.  

• The subsequent Activity Log degradation resulted from a connection-management incompatibility between the existing Activity Log client and OpenSearch 3.0, combined with uneven shard distribution and shard allocation remaining disabled from earlier troubleshooting. These conditions increased search queueing and latency on heavily loaded nodes. 

Incident Timeline 

• August 18, 2026, 12:00 UTC – During a scheduled maintenance event, an extension to the maintenance window was announced to allow additional maintenance activities to be completed. 

• August 18, 2026, 14:00 UTC – The extended maintenance was completed, and services were returned to normal operation. 

• August 18, 2026, 14:15 UTC – Shortly after the maintenance window concluded, an issue was identified affecting Activity Log functionality. A public notification was issued, and investigation and remediation efforts began. 

• August 18, 2026, 14:15 UTC to August 19, 2026, 13:53 UTC – Engineering teams worked continuously to diagnose and remediate the issue. Regular public updates were provided throughout this period to communicate ongoing progress and mitigation efforts. 

• August 20, 2026, 12:38 UTC – After 24 hours of monitoring, engineering teams observed no additional stability concerns and communicated that the issue was considered resolved at that time. 

• August 21, 2026, 14:15 UTC – Additional remediation measures were deployed to address remaining Activity Log issues affecting a subset of customers. A public update was issued to communicate the continued engineering efforts. 

• August 21, 2026, 17:17 UTC – Monitoring confirmed stable operation following the additional fixes, and the incident was declared fully resolved. All related service interruptions were considered remediated. 

Preventative Measures 

To reduce the likelihood and potential impact of similar incidents, we are taking the following steps: 

Enhancements to transaction management: We are reviewing existing transaction handling within our services to ensure database transactions fully complete or roll back within defined time limits, including when unexpected errors occur, which will help prevent incomplete transactions from consuming database connections and affecting service availability. 

Enhancements to monitoring and alerting: We are expanding our current monitoring and alerting capabilities to provide earlier detection of long running database transactions and lock contention, helping prevent impacts to database connection availability and customer-facing services. 

Enhancements to automated safeguards: We are enhancing our safeguards that automatically identify and safely address inactive database transactions that remain open beyond established thresholds, which will reduce the risk of these transactions accumulating and affecting database performance or service stability. 

Enhancements to database observability: We are improving the diagnostic information retained for database sessions to support faster identification and analysis of incomplete or long-running transactions, which will help accelerate troubleshooting and reduce the time required to identify and address similar conditions.