We have identified the issue and have already taken action to ensure that similar issues do not happen in the future. Below is a summary of our findings.

We assume that different types of errors may happen, and have taken measures to recover safely from these errors. In this case, the real issue that led to the downtime was that all of these measures failed simultaneously:

1. Automatic recovery of the webserver service on our Helsinki failed after the service unexpectedly failed with an error. We have identified the issue and have applied the necessary corrections.

2. Our status monitors failed to notice that the shared webserver was unresponsive. This not only prevented manual intervention by the person on call, but also automatic failovers that depend on the status monitor. We are working with our status monitor vendor to make sure this does not happen again, and will take the necessary actions based on the outcome of this work.

(The actual root cause was an error when renewing HTTPS certificates. The error happened in the Caddy webserver software, which is what we primarily use. We will file an issue report as well as continue our own investigation into why the error happened.)

We have identified the issue and all services are running normally. We are still looking into the root cause. We will publish updates on our findings here.

We have received reports that some sites are unresponsive. We are working on the issue.

Our main server for shared hosting customers in Helsinki is down.

Note: this status update was created after investigating the issue to reflect the actual start of the downtime.