Nightly refresh stuck
Ticketclosed
Lornefield Pensions Administration: documents · Nightly refresh stuck
Craig Paterson reported the nightly refresh hung for over 24 hours starting 06:00 on 21 Aug 2023, leaving all team members unable to access current data. Callie Brennan identified Salesforce API rate limits as the cause and resolved it.
What happened
What this says
Current to 10 Oct 26Craig Paterson reported the nightly refresh hung for over 24 hours starting 06:00 on 21 Aug 2023, leaving all team members unable to access current data. Callie Brennan identified Salesforce API rate limits as the cause and resolved it.
Problem: Nightly refresh started 06:00 on 21 Aug 2023 but hung until 14:43 the next day with no error logs. Craig reported a second day of stale data, team unable to work. Handling: Callie traced the issue to Salesforce connector hitting API rate limits overnight, blocking the full refresh. She resolved it and Ewan (Head of Platform) reviewed refresh sequencing to prevent recurrence. First response was 85 minutes. Watch: This was a high-impact incident affecting all users for 24 hours, exposing a platform-level rate-limiting design flaw.
Details
Updated 9 Oct 26Trail
created · Craig Paterson 21 Aug 2023 14:43Hi, the nightly refresh didn't complete last night. The job started at 06:00 as expected but it's now 14:43 and the data is still marked stale. No obvious error in the logs, just stuck. This is affecting all the team – they're querying old data and can't do their work. Regards, Craig- public_reply · Callie Brennan 21 Aug 2023 16:08Hi Craig, thanks for flagging. I can see the refresh is hung on the Salesforce connector. I'm looking into the cause now and will update you within the hour. Thanks, Callie
- public_reply · Craig Paterson 22 Aug 2023 16:14Callie, we're now into a second day of stale data. The team can't get their work done properly. Any progress on your end? Regards, Craig
- internal_note · Callie Brennan 24 Aug 2023 10:05Root cause: Salesforce API hit rate limits on the contact sync overnight, blocking the full refresh. Platform config issue, not customer-side. Ewan to review refresh sequencing.
- public_reply · Callie Brennan 24 Aug 2023 10:15Craig, I've found the cause – the Salesforce sync hit API rate limits overnight and blocked the full refresh. That's resolved now and the refresh completed this morning. We're also making a change to the refresh sequencing to prevent this happening again. Thanks, Callie
- status_change · 29 Aug 2023 23:59