On 8th November 2024 at 11:36AM EST (5:36PM UTC), partners on the Zinfandel and Concord Platforms experienced an issue where the Web Application (both Legacy and the New UI) exhibited slowness loading pages and processing login requests, as well as an issue with the Web Remote feature not loading successfully.
The root cause of this service interruption was identified to be a vendor issue that caused several components within the Datto RMM Infrastructure to become unhealthy, and experience timeouts when handling requests.
Automatic infrastructure scaling resolved the issue after the vendor outage has concluded.
In order to identify a similar issue faster in the future, alerting has been configured to monitor vendor service status, and alerting within our own infrastructure has been adjusted to be more sensitive to indicators of a vendor outage.