Nightly refresh failed again, data still stale
Ticketclosed
Brimscombe Energy · Nightly refresh failed again, data still stale
Refresh failed twice within two days of 22 Jul 2026, leaving data stale and blocking customer teams. Siân Pritchard escalated to leadership. Two root causes identified (compute timeout and API rate limit); data current by 24 Jul 2026 and architectural review underway.
What happened
What this says
As at Fri 9 Oct 2026Refresh failed twice within two days of 22 Jul 2026, leaving data stale and blocking customer teams. Siân Pritchard escalated to leadership. Two root causes identified (compute timeout and API rate limit); data current by 24 Jul 2026 and architectural review underway.
Problem: Nightly refresh failed twice within two days starting 22 Jul 2026, leaving dashboards stale and blocking customer teams from finding information they needed. Siân Pritchard, Head of Customer Data and Insight, escalated to leadership.
Handling: Kate Ormerod responded within 12 minutes on 22 Jul 2026 and investigated logs. By 24 Jul 2026 morning she had identified two separate failure modes: compute timeout on the customer's Snowflake account and a Salesforce API rate limit issue. Jamie Frost was looped in to review refresh architecture for a permanent fix. Refresh completed and data was current by 24 Jul 2026.
Watch: Siân was frustrated by a lack of promised updates between responses, widening the escalation. The company identified root causes and fixed the issue, which is good recovery; however, communication gaps should not recur. Key-person risk if Siân is the sole champion for the brain at Brimscombe.
Details
Updated 9 Oct 26Trail
created · Siân Pritchard 22 Jul 2026 09:56The refresh failed again last night. This is the second time in two days. We were told yesterday the issue was fixed, but this morning the dashboards are still showing stale data. Our customer teams are unable to find the information they need and this has been escalated to our leadership. I need to understand what's actually happening and get a real fix in place, not another temporary patch. We can't continue operating with unreliable data.- Public reply · Kate Ormerod 22 Jul 2026 10:08Hi Siân, I'm very sorry to hear the refresh failed again. I know this is frustrating after we resolved it yesterday. Let me pull the logs from last night's run and work out what we're dealing with. I'll have an update for you within the hour.
- Public reply · Siân Pritchard 24 Jul 2026 09:30Kate, it's been nearly three days since the first refresh failure. We were told yesterday the issue was fixed, but our dashboards are still stale. I've heard nothing from you since yesterday afternoon despite you saying you'd update within the hour. This is critical to our business and I need proper escalation now. Can you get someone who can actually diagnose and fix the root cause on a call with us today?
- Internal note · Kate Ormerod 24 Jul 2026 11:03Logs show two different failure modes. First night: compute timeout on customer's Snowflake account. Second night: timeout connecting to Salesforce API (likely rate limit hit). Not the same issue. Architecture needs review for resilience. Loop in Jamie to assess load pattern and design a proper fix rather than a patch.
- Public reply · Siân Pritchard 24 Jul 2026 11:38Thank you for identifying the two separate issues. That's helpful. But I need to know: when will our data be current, and when can we expect a permanent fix? Our teams are unable to function without reliable data.
- Public reply · Kate Ormerod 24 Jul 2026 11:53Siân, the refresh completed this morning and your data is current as of now. Jamie is reviewing the refresh architecture to handle both compute spikes and API rate limits more robustly. We're going to ensure this is a proper fix, not a patch. I'll have a detailed plan to you by Friday. I understand how critical this is and we will get it right.
- Status change · 28 Jul 2026 23:59