API

99.97% uptime
Sep 2023 · 100.0%Oct · 100.0%Nov · 99.92%
Sep 2023100.0% uptime
Oct 2023100.0% uptime
Nov 202399.92% uptime
Payment
99.97% uptime
Sep 2023 · 100.0%Oct · 100.0%Nov · 99.92%
Sep 2023100.0% uptime
Oct 2023100.0% uptime
Nov 202399.92% uptime
manage.quickpay.net
99.99% uptime
Sep 2023 · 100.0%Oct · 100.0%Nov · 99.97%
Sep 2023100.0% uptime
Oct 2023100.0% uptime
Nov 202399.97% uptime
3rd party services
100.0% uptime
Sep 2023 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2023100.0% uptime
Oct 2023100.0% uptime
Nov 2023100.0% uptime

Quickpay Support Mail

100.0% uptime
Sep 2023 · 100.0%Oct · 100.0%Nov · 100.0%
Sep 2023100.0% uptime
Oct 2023100.0% uptime
Nov 2023100.0% uptime

Notice history

Nov 2023

MobilePay transactions are processing again.
  • Resolved
    Update

    Post-Mortem Analysis

    Incident Overview
    On Friday, November 17th, Quickpay encountered significant operational challenges, affecting the delivery of callbacks and the processing of MobilePay Online and Vipps payments.

    Root Cause Analysis

    *Background: *About three months prior, Quickpay transitioned from an on-premise queue system to a cloud-based alternative. This change was not due to performance issues with the on-premise queue but was required for planned future updates to our platform.

    Incident Details: At approximately 11:20 CET on Friday, Quickpay began a routine import of payments from another PSP into our system, a process typically conducted 3-4 times weekly. However, each imported payment inadvertently triggered a callback to merchants, overwhelming their systems and leading to the rejection of these callbacks.

    This surge in rejected callbacks accumulated in our system, eventually reaching a threshold we had not previously encountered. This bottleneck also impacted MobilePay Online and Vipps transactions, which utilize the same queue system. The MobilePay Online and Vipps events are prioritized above callbacks to ensure new transactions are processed even if there is a delay in callbacks. However, as the platform reached the threshold, almost no events were processed.

    Regrettably, our team was unable to manually intervene in the queue without deleting all items, which would result in irreversible data loss.

    *Resolution: *By 13:45, we successfully reinstated our on-premise queue system. Subsequently, MobilePay Online, Vipps, and callbacks resumed normal processing. By 10:00 on Saturday, a portion of the failed events were processed in the queue, enabling us to process all remaining items, including pending MobilePay and Vipps payments.

    Next Steps

    Immediate Actions: We reverted to our on-premise queue system last Friday. This system, used up to six months ago, has been rigorously tested over several years, including during peak periods like previous Black Fridays, without issues related to the queue.

    The incident was not due to overall platform load. We are confident in the system's stability in the upcoming months and, with the on-premise queue, we are better equipped to swiftly address similar issues if they arise.

    *Long-Term Strategy: *In the future, we will transition back to the cloud-queue system. We are already sketching out enhancements that will enable us to process significantly more rejected callbacks than during the incident. We will also develop tools for more effective manual intervention. We will not transition to the cloud-system again before next year.

    Communication Shortcomings

    We acknowledge our failure to meet our communication standards. Despite promptly identifying the problem, internal miscommunication led to delayed updates to our customer support team and on our status page (https://status.quickpay.net). Post-resolution, our communication regarding the processing of queued events was also insufficient.

    As part of a larger corporate group, we are exploring ways to leverage group resources to enhance our external communication capabilities.

    Conclusion

    We recognize the significant inconvenience caused to our merchants by this incident. We sincerely apologize for these disruptions. While we cannot guarantee the absence of future issues, we are committed to continually improving our platform and communication strategies.

  • Resolved
    Resolved

    We are again processing MobilePay transactions.
    VippsPSP transactions have also been affected in the time frame.

    We are still investigating the pending MobilePay transactions and delayed callbacks, and monitoring.

    Best regards,
    QuickPay

Problems with MobilePay Online transactions
  • Resolved
    Resolved

    We are still experiencing problems.

    To avoid having pending MPO transactions, the service is shut down. Customers will receive an error instantly as they select MobilePay in the payment window.

    You can disable MobilePay Online in the Quickpay Manager, in Settings > Acquirer > Mobilepay Online, to make sure that customers do not experience the error in the payment window.

    Contrary to previous update, some callbacks are still getting delayed.

    We continue to work on the situation-

  • Identified
    Identified

    We are still experiencing problems.

    To avoid having pending MPO transactions, the service is shut down. Customers will receive an error instantly as they select MobilePay in the payment window.

    Contrary to previous update, some callbacks are still getting delayed.

    We continue to work on the situation-

  • Investigating
    Update

    We are still experiencing problems.

    The MobilePay transactions are authorized towards the acquirer, but are currently listed as Pending in Quickpay.

    Callbacks are processed as normal.

    We continue to watch the situation.

  • Investigating
    Investigating

    We are experiencing problems with MobilePay Online transactions.

    We are investigating and working on fixing the issue.

    Additionally, some callbacks are currently getting delayed.

    Best regards,
    QuickPay

Oct 2023

Apple are experiencing problems with Apple Pay.
  • Resolved
    Resolved

    Apple is currently experiencing issues with Apple Pay.

    You can check the status at https://www.apple.com/support/systemstatus/.

    We do not anticipate providing further updates on this incident.

Sep 2023

Sep 2023 to Nov 2023