AIS Power Outage
Thursday, April 23rd, 2009
Things have finally settled back down here at WebIntellects after the power outage we experienced last month at the AIS data center. I finally have a little bit of free time to sit down and blog about it. It is crazy how the loss of power for a few seconds could cause almost a month of work to get things back to normal. My name is Josh and I am a Systems Administrator for WebIntellects. I have been working here for almost 8 years and have been involved in a few of different outages during my tenure here. As a Systems Administrator when these things happen my life goes from a normal eight-hour workday to anywhere from twelve to sixteen hour work days. If I am not at the data center fixing various server issues, then I am back at the office working remotely or even from home working to fix important issues to get things back to normal in a timely manner.
Once power was restored after the brief loss, we had to manually start turning on all servers in our cage. This is a tedious task as it involves hundreds and hundreds of servers. In cases like this, there are always a few that do not come back online either. Most likely either a power supply failure, hard drive failure or worse - a fried RAID card or motherboard. With this outage we had a little bit of everything. We had a couple managed dedicated servers that needed power supplies replaced. We had one of our VPS servers sustain a corrupt RAID array resulting in the loss of all data on the server. We also lost one of our R1Soft backup server’s arrays, so weeks worth of backups were lost. This sort of loss takes weeks to recover, as we have to rebuild the server and its RAID array and then start recreating all the backups.
Times like these are particularly tough for the clients of WebIntellects as well as the staff. Nobody wants their websites and email to be down and we as a staff do not want this either. Unfortunately some of these things are out of our control. We really do appreciate the kind words from customers who understand when these things happen. We also understand the unhappy clients who have their websites down or even lost data because of the outage. Believe me there is nobody more upset about these outages than the staff at WebIntellects.
I will be posting here from time to time just to say hello and let everybody know what is going on in the world of System Administration at WebIntellects. Now I am off to start a server migration to migrate some of our clients off an old hardware/software solution to a new one. This is just another day in the life of a Systems Administrator.