What happened
What this says
Current to 10 Oct 26Data refresh failed for three business days because CRM connector token expired and permissions were revoked. Stale data blocked client calls and operations. Serious incident escalated to platform team.
Problem: Data refresh completely stopped overnight, leaving records stale for three business days. Customer needed fresh placement data for client calls and could not use the brain at all. Handling: Bethan asked clarifying questions and escalated to platform team led by Ewan Pryce. Root cause found: CRM connector token expired and Salesforce permissions revoked. Watch: Three-day data outage is a critical incident affecting operations; both SLA compliance and customer confidence are at serious risk.
Details
Updated 9 Oct 26Trail
- created · Jenna Price 15 Oct 2024 11:37The data refresh didn't complete last night. The brain still shows data from yesterday morning. We have client calls this afternoon and need the latest placement data. Can you look into this urgently?
- public_reply · Bethan Lister 15 Oct 2024 12:43Thanks, Jenna. Just so I've got this right – the refresh didn't complete last night? Can you tell me: what time does the refresh usually finish for you, and when did you first notice the data was stale? Thanks, Bethan
- public_reply · Jenna Price 16 Oct 2024 12:36Sorry for the slow reply – been in calls all morning. The refresh should finish by 07:00, but nothing came through. The data's still from yesterday. We'll need it updated for the afternoon calls.
- public_reply · Jenna Price 18 Oct 2024 10:20Still no refresh. This is now three business days with no updates. The team can't use the brain at all right now. This is really affecting our operations – what's the issue?
- public_reply · Bethan Lister 18 Oct 2024 12:20I'm sorry, Jenna. I've escalated this to our platform team and they're looking at it now. Can you tell me: have you made any recent changes to the Salesforce system on your end? Any permissions changes or data updates? That can sometimes affect the refresh. Thanks, Bethan
- internal_note · Bethan Lister 18 Oct 2024 13:59Platform team found it. CRM connector token is expired. Permissions got revoked on Salesforce. Need new OAuth token from customer.
- status_change · 22 Oct 2024 23:59