AIS Power Outage
Thursday, April 23rd, 2009
Once power was restored after the brief loss, we had to manually start turning on all servers in our cage. This is a tedious task as it involves hundreds and hundreds of servers. In cases like this, there are always a few that do not come back online either. Most likely either a power supply failure, hard drive failure or worse - a fried RAID card or motherboard. With this outage we had a little bit of everything. We had a couple managed dedicated servers that needed power supplies replaced. We had one of our VPS servers sustain a corrupt RAID array resulting in the loss of all data on the server. We also lost one of our R1Soft backup server’s arrays, so weeks worth of backups were lost. This sort of loss takes weeks to recover, as we have to rebuild the server and its RAID array and then start recreating all the backups.
Times like these are particularly tough for the clients of WebIntellects as well as the staff. Nobody wants their websites and email to be down and we as a staff do not want this either. Unfortunately some of these things are out of our control. We really do appreciate the kind words from customers who understand when these things happen. We also understand the unhappy clients who have their websites down or even lost data because of the outage. Believe me there is nobody more upset about these outages than the staff at WebIntellects.