Email from Caro.
Hello,
At approximately 3:15am EST (GMT-5) Caronet Managed Hosting DC3 experienced a large power surge. Since we have multiple power feeds in to the facility, our incoming power is governed by a Critical Buss Synchronizer. Some event, as of yet undetermined, during the surge damaged the device that governs the redundant power to our facility and, because of the danger of mixing feeds from two different AC power sources, the device activated a Emergency Power Off (EPO).
Within 8 minutes, our onsite electrician was diagnosing the issue. Once Safety was checked, the damaged regulator was bypassed and power restored via a single source backed up be generator.
At this time, we have few details to work with but are continuing to gather information. The decision was made, in the field, and per our operating procedure, to continue the EPO as the extent of damage and potential for catastrophic failure was too high for us to restore single source with generator backup power while still on battery.
We realize for the data center of our stature, any outage is inexcusable. We accept full responsibility for the performance of our system.
At this point, service should be restored as we continue to diagnose the problem. As more information becomes available, we will distribute an update.
Again, please accept our apologies for this inexcusable error.
Regards,
Caronet Managed Hosting Support
Noah Cain
Caronet Managed Hosting
Technical Support