A preliminary hotfix to mitigate future occurrences of the backup performance issues experienced in this incident has been deployed and is being monitored in preparation for a full release.
Posted Dec 18, 2024 - 15:17 EST
Update
Backup success rates across all services fleet-wide have returned to normal levels and remain stable.
Our R&D team is continuing to test a release to mitigate future occurances of the backup performance issues experienced in this incident. The release is expected to be deployed next week.
Posted Dec 13, 2024 - 15:11 EST
Update
Our R&D team has completed development of a hotfix to address the issue and it is actively being tested internally in preperation for release.
Backup performance has improved and remained stable after the changes initiated last week and the team is continuing to monitor.
Posted Dec 09, 2024 - 16:08 EST
Update
The hotfix to address this issue is still in active development and our R&D team continues to monitor backup performance and apply configuration changes as needed to mitigate impact.
Posted Dec 06, 2024 - 14:03 EST
Update
The R&D team continues to see improvement to backup performance on the affected pods after additional configuration changes were implemented.
A hotfix to address the issue is still in active development.
Posted Dec 04, 2024 - 14:14 EST
Update
The R&D team has observed substantial improvement to backup performance on the affected pods after the changes implemented yesterday.
Another hotfix is in active development to address the issue further.
Posted Dec 03, 2024 - 14:37 EST
Update
Additional infrastructure changes to further mitigate the issue have been implemented on all affected pods.
We continue to see positive improvement in overall success rates for affected pods/services and our R&D team is continuing to monitor recovery progress.
Posted Dec 02, 2024 - 16:16 EST
Update
Our R&D team has identified additional pods in the GBE and AUE regions showing backup performance degradation for some M365 services.
Current affected pods: 'use1-saas-p0' 'use1-saas-p3' 'use1-saas-p4' 'use1-saas-p8' 'use1-saas-p10'
'gbe2-saas-p1' 'gbe2-saas-p0'
'aue1-saas-p0'
The R&D team is implementing additional changes to the affected pods to resolve the issue and will continue to monitor.
Posted Nov 29, 2024 - 13:32 EST
Update
Our R&D team has implemented some additional configuration changes alongside the hotfix that was deployed to address the issue further and will continue to monitor.
Posted Nov 28, 2024 - 11:50 EST
Monitoring
Our R&D team has finished deploying a hotfix to address the issue and are actively monitoring results.
Posted Nov 27, 2024 - 12:24 EST
Identified
Our R&D team has identified a problem causing backup performance degradation for some M365 services, which is affecting some Datto SaaS Protection customers hosted on certain pods in the US-East Region.
The R&D team is in the process of deploying a hotfix to address this issue.