Inky Public Root Cause Analysis
EU Central Email Processing Delay
Summary
On the date of the incident, Inky experienced a disruption to email processing within the EU Central region. Increased inbound email volume coincided with a failure of infrastructure autoscaling, preventing additional processing capacity from being automatically provisioned. As a result, email messages queued for processing and experienced delivery delays.
Service was restored after infrastructure capacity was manually increased, allowing queued messages to be processed and normal throughput to resume.
Customer Impact
Customers whose email traffic was processed through the EU Central region experienced delays in email analysis and delivery.
Impact Details
Service affected: Email Processing
Region affected: EU Central
Impact start: 08:15 UTC
Service restoration: 10:00 UTC
Total duration: Approximately 105 minutes
Maximum observed delay: Up to approximately 90 minutes for some messages
No customer data was lost during the incident. Queued messages were retained and processed once capacity was restored.
What Happened
During a normal increase in email traffic, the platform required additional processing capacity. The automatic scaling mechanism responsible for adding resources did not successfully provision additional servers due to a configuration issue within the scaling configuration.
Because additional capacity was not added as expected, processing resources became constrained and email messages began accumulating in processing queues, resulting in delivery delays.
Resolution
The operations team investigated the processing delays and determined that additional processing capacity was required. Infrastructure resources in the affected region were manually scaled to meet demand.
As the additional resources came online, queue depth decreased and processing throughput returned to normal levels. All queued messages were subsequently processed and delivered.
Root Cause
The incident was caused by a configuration condition within the autoscaling setup that prevented automatic expansion of processing capacity when demand increased.
A previous capacity adjustment increased the desired resource level but did not include a corresponding increase to the maximum scaling limit. When additional capacity was required, the autoscaling process was unable to complete the scale-up operation, preventing new resources from being added automatically.
Preventative Actions
To reduce the likelihood of similar incidents, the following corrective actions are being implemented:
Improve autoscaling configuration safeguards
Infrastructure templates are being updated so scaling limits are automatically aligned with required capacity settings, reducing the possibility of configuration mismatches.
Enhance monitoring and alerting
New monitoring will directly detect and alert on autoscaling failures, enabling faster identification and response when scaling operations do not complete successfully.
Accelerate operational response
Alerting and operational procedures will be updated to provide faster, more direct mitigation guidance when autoscaling failures occur.
Final Status
The issue has been fully resolved, and normal email processing operations have been restored. Additional monitoring and infrastructure improvements are being implemented to strengthen resiliency and improve detection of similar conditions in the future.
Incident Duration: 08:15 UTC – 10:00 UTC (105 minutes) Affected Service: Inky Email Processing (EU Central Region)