Feed refresh delayed

Incident Report for The Old Reader

Postmortem

Timeline:
March 7, 2024 - Our hosting provider did maintenance that ended up bringing down the site for almost 8 hours. This caused our feed refresh system to get backed up.

April 1, 2024 - A build up of feeds needing refreshed slowly grew due to the previous backup until it overwhelmed the server controlling the updates. The server was restarted causing many feeds to not get scheduled for updates.

April 2, 2024 - We were notified of the feeds not being updated and began working to resolve the issue. Ran the scheduler at reduced capacity to keep as many feeds as up to date as possible.

April 5, 2024 - Began rollout of a new scheduling system to that will handle backups and downtime better. We also started reprocessing feeds that had not been updated recently.

April 6, 2024 - Released additional components of the new scheduling system.

April 7, 2024 - All feeds had been updated within the scheduling window they are expected to update in.

Upcoming work:

Additional updates to the new scheduling system to make it more resilient
Implementing alerts so that we will be notified of queue backups before they start affecting most users
Continuing to investigate feeds that are not updating for various reasons unrelated to the scheduling

Posted Apr 08, 2024 - 12:21 UTC

Resolved

Most feeds are now updating regularly. There are a small number of feeds that have specific issues that we are continuing to evaluate. Please see post mortem for more details.

Posted Apr 08, 2024 - 12:20 UTC

Update

We have slowed down the refresh frequency in an effort to get all feeds to be able to stay more up to date. Our goal is to refresh all feeds today and keep them refreshing at least once per day while we implement a new system that can better handle the number of feeds we are dealing with (which is currently a little over 8.5 million!)

We thank you again for your patience on this.

Posted Apr 05, 2024 - 12:38 UTC

Update

We are still catching up on the feeds that were backed up. We will hopefully be caught up in the next few hours. Thank you for your patience.

Posted Apr 04, 2024 - 03:08 UTC

Update

We are still catching up on the feeds that were backed up. Thank you for your patience.

Posted Apr 03, 2024 - 22:51 UTC

Monitoring

We are working through the backlog of feeds that have not been processed recently and will continue to monitor.

Posted Apr 03, 2024 - 05:06 UTC

Identified

We are currently experiencing a backup with feed refreshes, we are working to correct the situation.

Posted Apr 02, 2024 - 22:28 UTC

This incident affected: API and Website.