SOAR

We apologize for the delay in getting back to you with a final report on the service outage last week. We received a final report from our service provider last Wednesday on the incident and spent most of yesterday working through next steps.

On Wednesday April 18th at 2:15 PM EST service to all SOAR websites was interrupted - access to customer websites and email functionality was no longer available.

SOAR has used Cogeco Peer 1 (www.cogecopeer1.com) for hosting for 11 years. The majority of our dedicated servers are in their San Antonio, TX data center. In theory, each of their facilities has redundant connections to the Internet. Should one of the connections fail, service to the data center should continue through the other connection.

A 3rd party fiber line between San Antonio and Dallas, TX was damaged and was the root cause of the outage. The damage severed both redundant connections to the Cogeco Peer 1 San Antonio data center as well as other companies besides Cogeco Peer 1. A more detailed report from Cogeco Peer 1 is dowloadable here.

The fiber line was fixed at 7:00 PM EST and SOAR websites became available again.

The likelihood of such an event occurring again is very small. At the same time, the experience has shown us we need to be better prepared for any type of interruption of service. We are taking the following actions to ensure better service to our customers.

1. Our two NameServers will be moved to venders other than Cogeco Peer 1
   • Each NameServer will have a separate vendor, and a different city location
   • This will enable us to route traffic in any emergency
   • This action was completed this week
2. We will keep hosting with another vendor, other than those used for the NameServers or Cogeco Peer 1
   • This will give us a place to redirect traffic in case of emergency
   • This will allow each SOAR customer website to serve up a message with details on what is going on
   • This action will be completed in early May
3. We will store a secure, updated copy of customer information in our offices
   • This will allow us to communicate with proper customer contacts during emergencies
   • This action was completed this week
4. We are going to keep our primary hosting with Cogeco Peer 1
   • The redundant internet connections to the San Antonio data center are through different companies
   • Through mergers/acquisitions of external companies they ended up in the same fiber line without Cogeco Peer 1 noticing
   • Since the unthinkable did happen, we are confident Cogeco Peer 1 will not only fix the issue, but make sure it doesn’t happen again
   • In our 11 years of business with them, this is really the first time there has been an outage of such duration

We sincerely apologize for your SOAR website being unable for 5 hours last week. We were in the dark just like you were. We believe the above actions will make sure we can react better during any future service interruptions.

If you have any questions about any of this, please feel free to contact us directly.

SOAR Support