Timeline:
March 7, 2024 - Our hosting provider did maintenance that ended up bringing down the site for almost 8 hours. This caused our feed refresh system to get backed up.
April 1, 2024 - A build up of feeds needing refreshed slowly grew due to the previous backup until it overwhelmed the server controlling the updates. The server was restarted causing many feeds to not get scheduled for updates.
April 2, 2024 - We were notified of the feeds not being updated and began working to resolve the issue. Ran the scheduler at reduced capacity to keep as many feeds as up to date as possible.
April 5, 2024 - Began rollout of a new scheduling system to that will handle backups and downtime better. We also started reprocessing feeds that had not been updated recently.
April 6, 2024 - Released additional components of the new scheduling system.
April 7, 2024 - All feeds had been updated within the scheduling window they are expected to update in.
Upcoming work: