Post Mortem

English version below

Tidslinje over certifikatproblemer natten mellem 26. oktober og 27. oktober.

kl. 23.30 d. 26. oktober

• Et af QuickPays centrale certifikater revokes af vores certifikatudbyder.
• Dette betyder, at betalingsvinduet ikke vil have været tilgængeligt gennem enkelte browsere (bl.a. Firefox), samt at https://api.quickpay.net ikke vil have været tilgængeligt gennem visse systemer.
• Problemet opfanges desværre ikke i vores overvågning af certifikater.
• Antallet af gennemførte betalinger i løbet af natten har været tilsvarende en gennemsnitlig nat. Og da problemet kun har berørt ganske få transaktioner, er det heller ikke opfanget i vores generelle overvågning.

kl. 06.10 d. 27. oktober

• Problemet opdages manuelt af en af vores ansatte.

kl. 06.18

• Det centrale certifikat bliver udskiftet til et nyt certifikat fra en anden certifikatudbyder på alle services.

kl. 08.22

• Det nye certifikat er ikke "trusted" af MobilePay Online, hvilket har betydet at betalinger gennem MobilePay Online ikke har kunnet gennemføres. Det er et ganske normalt forbrugsmønster, at der ikke er mange MobilePay Online transaktioner tidlig morgenstund.
• Visse andre direkte integrationer direkte mod QuickPay API, vil ligeledes have oplevet problemer hvis de ikke har "trusted" det nye certifikat.

kl. 08.50

• Da vi stadigt ikke er i stand til at få fat i vores gamle certifikatudsteder af certifikat rulles det gamle "revoked" certifikat på https://api.quickpay.net, hvilket løser problemer med MobilePay Online og andre integrationer, der ikke har understøttet det nye.

kl. 09.00

• En ny version af det oprindelige certifikat modtages og rulles på.
• Alt er herefter "back to normal".

—

Det er stadigt uklart hvorfor det oprindelige certifikat i første omgang er revoked. Det nye certifikat er med samme key, og det gamle var ikke udløbet.

Udsteder har annonceret at de ville revoke certifikat d. 22/10, men desværre til en ikke-eksisterende mailadresse hos QuickPay, og udsteder har herefter ikke reageret på at mail er bouncet.

—

Vi er i dialog med vores certifikatudsteder, for at finde ud af hvorfor det præcist er sket, samt sikre os at lignende brist i kommunikationen ikke kan ske igen.

Vi er samtidigt i dialog med MobilePay for at sikre at de får opdateret til at "truste" nye certifikater fra vores alternative udbyder til fremtiden.

Vi kigger ligeledes på vores overvågning for at sikre, at vi fanger et evt. lignende scenarie. Vi beklager meget det hændte.

English version

Timeline of problems with certificates around the 27th of October.

At 23.30 - October 26.

• One of the central certificates in QuickPay is revoked by the certificate provider.
• This means that it will not have been possible to access the QuickPay payment window in some browsers (e.g. Firefox), and https://api.quickpay.net for some systems.
• Unfortunately this was not detected by our automatic monitoring of certificates.
• The number of completed payments during the night was close to a regular night. And as the problem has only impacted few transactions, this was also not detected by our general monitoring.

At 06.10 - October 27

• The problem is manually detected by an employee.

At 06.18

• The certificate is replaced with a new certificate from another provider on all services.

At 08.22

• The new certificate is not trusted by MobilePay Online, and it was therefore not possible to complete payments through MobilePay Online.
• Some other integrations to QuickPay API will also have been impacted, if they did not trust the new certificate.

At 08.50

• We have still not been able to contact our old certificate provider, and the old revoked certificate is again active on https://api.quickpay.net which solves the problem with MobilePay Online and other integrations.

At 09.00

• We receive a new version of the original certificate, and it is deployed.
• All systems are now back to normal.

It is still unclear why the original certificate was revoked in the first place. The new certificate used the same key and the old one did not expire.

The provider has announced that they would revoke the certificate on October 22, but send the mail to a non-existing mail-address at QuickPay, and have not followed up as the mail bounced.

We are talking with the certificate provider to understand why the certificate was revoked, and to ensure that communications will not break in the future.

We are talking with MobilePay Online to ensure that they in the future trust certificates from the alternative provider.

We are also looking at our automatic monitoring systems to make sure that we in the future will detect this kind of issue. We are sorry for any inconvenience this has caused.

All systems and operations are back to normal and all operations are working again.

Best regards, QuickPay