All systems operational

Show current status

Resolved
Problems with our online systems

Started
September 02, 2022 at 10:51 PM
Status
Resolved after less than a minute

Impact

Operational
Affected
API
Payment
  • Resolved
    September 06, 2022 at 7:38 AM

    Post mortem

    On the night between Friday September 2, 2022 and Saturday September 3, 2022, we experienced a system outage. The outage affected all payment methods from 00.15 > 01.15 CEST. From 01.15 > 11.30 CEST we still experienced downtime for payments made via Vipps, Swish & Klarna.

    The outage was caused by a hardware failure on our database.

    Our systems are designed with failover in mind, with a backup database always available and up to date with the primary. The system is designed to automatically switch to the backup database, and use this as the primary in case of hardware failures.

    Unfortunately this did not happen friday. As a result, all transactions were refused.

    Our operation team was quickly alerted, but the error that caused the systems to not automatically switch, also made the manual fix more difficult . At around 01.15, the team were able to restart all payment services running on the new database, and transactions were again processed.

    How to make sure this does not happen again

    Hardware failures are part of the game of self-hosted systems. Quickpay are in the process of migrating our systems to being cloud hosted, which will make sure that such hardware failures do not happen again.

    Until this migration is completed, we cannot avoid hardware failures, but we can make sure that the services will automatically switch to the backup database.

    We are already working on fixing the problems, and making sure that if such a problem should happen again, the systems are able to correct themselves quickly, without anyone noticing.

    Why was nothing announced on status.quickpay.net

    We are sorry that nothing was announced at the time of the outage. When we have an outage that affects all payments our first priority is to fix the issue at hand. We have a status-page service provider that allows us to make default announcements which did not work for various reasons, this will also be fixed to make sure that we communicate as quickly as possible in the future."

  • Resolved
    September 03, 2022 at 9:55 AM

    Additionally payments with some payment methods have been affected longer.

    This problem is also fixed now, and we are investigating and monitoring the situation

  • Resolved
    September 02, 2022 at 10:51 PM

    Sep 3 from 00:51 to 01:15 CEST, we experienced problems with the payment systems.

    All transactions were refused.

    The problem was fixed by our operations team. We will provide a more detailed update later this week.

    Best regards, Quickpay