Technical difficulties
Incident Report for Proton Services
Postmortem

Shortly before 11:40 UTC we began a server-side data migration to support a new Proton feature. Unfortunately, a bug in this migration caused our servers to issue a signal to some users' mail clients that instructed them to reset their local cache and redownload all data from the server. At 11:40 UTC this mass resynchronization reached critical load on one of our storage layers, which then began having difficulty serving data to all requests. In order to reduce the load and protect other services such as Proton Drive and VPN, we took Mail and Calendar offline. We then restored the offline services gradually, with all services except for Proton Mail Bridge restored by approximately 12:44 UTC.

We have reverted the change, so any clients which were offline at the time of the signal will not reset their cache, and most mail clients will have resynchronized very quickly. Proton Mail Bridge, however, is a special case. While it was not the root cause of the issue, it maintains a complete offline copy of all mails and attachments, so Bridge resynchronizations caused the most additional load. As a result, the best way to manage the recovery of Mail services was to slow down Bridge resynchronization, which took some time to put in place. Bridge service is now restored as well, though resynchronizations will take longer than usual. We will lift these limits once load on our infrastructure returns to normal levels. As always, we will do a thorough post-mortem to identify exactly what happened and how, and study how we can both avoid recurrence and improve our response to issues in the future. We deeply appreciate your patience and apologize for the inconvenience.

Posted Jul 25, 2023 - 17:36 CEST

Resolved
Proton Mail Bridge is now back online, but still recovering. The email delivery and sync will be slower than usual for the next few hours. A more detailed account of today's incident will be published soon.
We apologize for the inconvenience.
Posted Jul 25, 2023 - 16:22 CEST
Monitoring
We are now gradually bringing Proton Mail Bridge back online.
Posted Jul 25, 2023 - 15:42 CEST
Update
The issue has now been isolated, and all of our services are back online as of 2:44 pm CEST, except for the Proton Mail Bridge. We are still fixing Proton Mail Bridge, so if you use it, we recommend using the web app or mobile apps to access your Proton Mail until then.
We confirm that no data has been lost, and we apologize for the inconvenience.
Posted Jul 25, 2023 - 14:52 CEST
Update
We are continuing to work on a fix for this issue.
Posted Jul 25, 2023 - 14:35 CEST
Identified
Thank you for your patience. Proton Drive and Proton VPN are now back online, and we are working on a fix for other services.
Posted Jul 25, 2023 - 14:29 CEST
Update
We apologize for the current situation. No data is lost, but access, email delivery, and notifications will be delayed. We are working on the issue and hope to fully restore our services shortly. We'll continue sharing updates here as they happen.
Posted Jul 25, 2023 - 14:12 CEST
Update
We are continuing to investigate this issue.
Posted Jul 25, 2023 - 14:04 CEST
Update
We are continuing to investigate this issue.
Posted Jul 25, 2023 - 13:47 CEST
Investigating
We are currently experiencing technical difficulties affecting some of our users. We are working to fully restore services as soon as possible. We apologize for the inconvenience.
Posted Jul 25, 2023 - 13:40 CEST
This incident affected: Proton Mail (Web Application, Bridge, Mobile Apps), Proton VPN (Free servers, Regular Servers, Secure Core Servers, Streaming, Mobile and Desktop Apps), Proton Calendar (Mobile Apps, Web Application), Proton Drive (Web Application, Mobile and Desktop Apps), Proton Pass (Mobile Apps, Browser Extensions), and SimpleLogin, proton.me Website.