Post mortem

On the night between Friday September 2, 2022 and Saturday September 3, 2022, we experienced a system outage. The outage affected all payment methods from 00.15 > 01.15 CEST. From 01.15 > 11.30 CEST we still experienced downtime for payments made via Vipps, Swish & Klarna.

The outage was caused by a hardware failure on our database.

Our systems are designed with failover in mind, with a backup database always available and up to date with the primary. The system is designed to automatically switch to the backup database, and use this as the primary in case of hardware failures.

Unfortunately this did not happen friday. As a result, all transactions were refused.

Our operation team was quickly alerted, but the error that caused the systems to not automatically switch, also made the manual fix more difficult . At around 01.15, the team were able to restart all payment services running on the new database, and transactions were again processed.

How to make sure this does not happen again

Hardware failures are part of the game of self-hosted systems. Quickpay are in the process of migrating our systems to being cloud hosted, which will make sure that such hardware failures do not happen again.

Until this migration is completed, we cannot avoid hardware failures, but we can make sure that the services will automatically switch to the backup database.

We are already working on fixing the problems, and making sure that if such a problem should happen again, the systems are able to correct themselves quickly, without anyone noticing.

Why was nothing announced on status.quickpay.net

We are sorry that nothing was announced at the time of the outage. When we have an outage that affects all payments our first priority is to fix the issue at hand. We have a status-page service provider that allows us to make default announcements which did not work for various reasons, this will also be fixed to make sure that we communicate as quickly as possible in the future."

Additionally payments with some payment methods have been affected longer.

This problem is also fixed now, and we are investigating and monitoring the situation