Cruzio DSL Outage Details

[consultants@mycruzio.com]

Hello Cruzio Consultants,

Late Wednesday night, up to 130 Cruzio customers suffered a DSL outage
due to a problem in the ATT network. This email is meant to provide
information about what happened, what caused it, and what we are
doing to prevent it from happening again.

What happened
--------------
According to Cruzio's DSL monitoring equipment, service loss occurred  
around 11:30 PM.
Chris Neklason (an owner and CTO) quickly mobilized the team to diagnose
the issue. It  became clear that the affected customers
were the same group of people who had suffered the 2 previous outages
in the past 4 weeks.

What caused this outage
-----------------------
The previous 2 outages were due to network congestion on the
ATT ATM network which provides the transit from the various ATT
central offices to the aggregation center and then to our routers
in San Francisco.

This particular outage was caused by ATT routing some of our customer
traffic to their routers and some of ATT's traffic to our routers.

What we are doing to prevent this from happening again
-------------------------------------------------------
We are setting up a meeting between the ATT reps, Cruzio, and
Sonic.net, with whom we are sharing the transit circuit.

Sonic and Cruzio represent about 40,000 customers, and we hope
to apply some political pressure to ATT to improve the quality
of the team working on our circuits.

In the course of coordinating our response with Sonic, we might
decide to escalate this to the California PUC and engage in the
formal complaint process as a means of applying more policial
pressure on ATT.

In addition, the Cruzio Support and Engineering teams are
continually working on improved contingencies and procedures to
rapidly respond to any future such outages.

Cruzio customer service is contacting each affected customer to  
credit their account
for a month of service and to find out from the customer if there
are any other ways we can make this right by them.

In conclusion
--------------
Cruzio has been working with Pacific Bell, then SBC, now ATT to
resell and support DSL since 1999 and it has proved to be a
remarkably stable access product.

Interruptions of service are rare, usually affecting a single  
customer or
a very small group. Multiple outages affecting the same group of  
customers
are rarer still. We do not expect another occurrence.

The Folks at Cruzio
831.459.6301
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.mycruzio.com/pipermail/consultants/attachments/20070615/424d948f/attachment.html