Update - The situation has stabilised, but is not solved yet.
Beyond the one server still not having power restored, allmost all databases were affected in some manner (though no data has been lost) and replication has to be restored immediately to prevent further issues should another powerloss event ocurr.
This puts additional load on the databases wich leads to some sites being very slow - we are sorry, but we can not influence that, the restore has to complete for speeds to normalize.
Should you still experience _full_ outage issues please contact us.
This is the worst outage of our datacenter provider that has happened in years, with some impact in almost all parts of our infrastructure - we are working hard to solve the general issues and your specific website problems, please bear with us if ticket replys take longer than you would like them too.
May 24, 13:59 UTC
Update - We have recovered a good portion (about 3/4) of the affected systems, things are looking up.
Databases have been restored, Shared storage has been restored, so far we can see no data loss.
Currently only one Virtualization server is contistenly down and we are waiting on our datacenter provider to restore operation, which leaves 20 customers still affected.
We will update you with information as soon as we have it.
May 24, 12:19 UTC
Update - Since our datacenter provider has tentatively restored operation, we are in the process of cleaning up the effects of the outage. Operation to most customers will be re restored "soon".
We will give another update in the next 30 minutes as to the state of things.
We already know that a few systems are still not powered up again and our datacenter provider is working on these systems, more details will follow in the next update.
May 24, 11:12 UTC
Update - Our datacenter provider has come forward with another update:
Power has been restored (without backup-redundancy) to all affected datacenters.
Their technicians are working on restoring said redundancy and manualy ensuring that all servers that have not automatically been started are being started/repaired if physical damage was taken.
Identified cause was a power spike in their local power grid.
We will keep you updated.
May 24, 10:35 UTC
Update - Our datacenter provide has come forward with a more detailed update:
3 data center rooms experienced a power outage, which led to a failiure of core networking equipment, which in turn led to an outage accross the whole datacenter.
They are currently in the process of recovery, we will keep you updated.
May 24, 10:27 UTC
Update - Our datacenter provider has come forward with the information that the cause is a power outage and not just a network outage, they are working to resolve the issue.
May 24, 09:54 UTC
Identified - Our Datacenter provider has experienced a major network outage.
The situation is currently being resolved, services are recovering.
We'll update you here when we have news.
May 24, 09:37 UTC
Investigating - We are currently experiencing a network-related outage that is affecting almost all services. It is our highest priority right now and we hope to have it resolved as soon as possible.
We will update this post as soon as we have more information.
May 24, 09:12 UTC