SOAR

Sunday January 4th, 2009

SOAR Customers,

It is our policy to provide you with notification of any issue that has interrupted service to your website.

We do this to ensure good communication with our customers and to make you aware of the issue so you can answer questions raised by your unit members.

Service Outage – 1/4/09 4-10 AM EST

At 4:00 AM EST on Sunday January 4th, 2009 one of our customer servers went offline. The server in question hosts approximately 40% of our customers, so a large number of customer sites became unavailable.

We put a call into our hosting provider at 4:30 AM EST to log the issue and have them start working on it. After diagnostics by our hosting provider it was clear there was a hardware problem – the RAID (disk drive) controller had failed. The state of the disk drives and their data integrity was unknown.

We arrived at the office at 6:00 AM EST with warm caffeine (coffee), cold caffeine (aka soda), and bagels in preparation for a long morning :0

By 7:00 AM EST our hosting provider had completed all hardware fixes. The state of the data on the disk drives was in question. We have a good backup from earlier in the morning, just prior to the hardware failure. We decided a full fresh server build and data restoration was in order to ensure the continued health of the server and customers sites. By 8:00 AM EST the server was rebuilt and customer data was restored by 10:00 AM EST.

At 10:00 AM EST on Sunday January 4th, 2009 service was restored to the customer sites affected. Given we used the backups from earlier this morning, no data was lost.

The server was down during the scheduled EBlast for Sunday, so we manually initiated EBlasts at 10AM EST.

Service History

We thought it would be a good idea on each Service Outage announcement to include a record of service interruptions during the last year.

While we strive to provide 100% service availability, unscheduled service outages will occur from time to time. When these do occur our upfront planning will help to minimize the duration of these service outages. Our service outage history for the last 12 months will better demonstrate that with facts.

01/04/09 - 40% of customer website unavailable from 4-10AM EST, due to hardware failure
12/29/08 - EBlasts for Sunday (12/28) sent out on Monday (12/29)
08/20/08 - Boy Scout Troop websites unavailable from 3-6PM EST, due to software configuration issue
08/04/08 - Customer websites unavailable from 4-8PM EST, due to software configuration issue
04/08/08 - Customer websites unavailable from 6-8AM EST, due to database failure
02/02/08 - Customer websites unavailable from 6-11AM EST, due to customer email loop overloading a server

Server Uptime for the last 12 months: 99.9977%

Do I need to do anything?

You do not need to do anything. All customer data was restored.

All EBlasts scheduled for Sunday (1/4) were manually sent at 10AM EST.

The SOAR Support Team
support@soarol.com
585-388-0211
9AM - 4PM EST, Mon-Fri