Summary
On 2026‑04‑09, DRMM customers on the Pinotage platform experienced intermittent UI login failures between 07:17–09:30 UTC. The impact occurred in brief spikes (~2 minutes each) during which login attempts could time out and, in some cases, users were subsequently presented with an “Access Denied” message in the UI. Service returned to normal by 09:30 UTC.
Customer Impact
Who: A subset of Pinotage DRMM customers attempting to log in via the UI
What: Intermittent inability to log in (timeouts; some attempts surfaced as “Access Denied”)
When: 07:17–09:30 UTC, peaking during brief bursts
Impact: Degraded authentication availability (intermittent)
Root Cause
The login path relies on a backend caching service for session/tenant/authentication-related lookups. During a burst of login traffic, the application’s available cache connection pool was exhausted, preventing some requests from acquiring a connection in time. This led to request timeouts and, depending on the failure path, some failures were mapped to an “Access Denied” UI message rather than a “temporarily unavailable/try again” error.
Contributing Factors
Burst traffic pattern (e.g., start‑of‑day surge) created short periods of elevated count of concurrent login attempts.
Connection pool default limits were not sized for this burst concurrency.
Error mapping in the UI/auth flow surfaced dependency timeouts as “Access Denied,” which can be misleading and increased user confusion.
Monitoring/alerting did not provide sufficiently early signal on pool saturation to reduce detection and resolution time.
Incident Timeline
Identified: 2026-04-09 08:06 UTC
Public Notification: 2026-04-09 09:00 UTC
Resolved: 2026-04-09 09:30 UTC
Preventive Measures (Planned/Committed)
Infrastructure resiliency (capacity): Increase default cache connection pool capacity and validate against expected burst concurrency.
Monitoring & alerting: Add dashboards and alerts for cache pool utilization, connection acquisition latency, and login error rate/SLO burn.
Faster detection: Route alerts to operational channels/on-call with clear runbooks and thresholds tuned to identify spikes early.
Scaling strategy: Ensure scaling policies consider backend constraints (e.g., pool sizing per instance, cache max connections) so that scaling the app layer does not inadvertently worsen pool exhaustion.
Customer-facing correctness: Update error handling so that dependency timeouts return a retriable “service busy/please retry” message rather than “Access Denied,” unless an authorization failure is confirmed.