On Tuesday morning we migrated databases for 6 of our services to a new provider. One of those service was that allowing stores to stipulate how events are handled - the Integrations service. This service allows the set up of web-hooks so other systems can be informed of events happening within a service.
The migration was a success and data sync between old and new providers was monitored and verified.
All events were processing successfully with no issues reported.
On Thursday evening, it was reported that a single store's web-hooks were failing to process. After a brief investigation we determined that while the all the services had successfully transitioned to the new providers, the same could not be said for the workers processing the events.
This was rectified quickly.
The store had created some new web-hooks - these had been written to the databases on the new providers platform, the workers however, were still reading from the 'old' database provider and hence could not see the newly created web-hooks.
No previously existing integrations were affected.
We have tasked guarding against a recurrence of this issue in the future.