Dear mole,
As you might know, some of our users have been experiencing slow performance with the TypePad service over the past few weeks. Ben and Mena, our founders, posted about this earlier this week. I want to provide more details and give you additional insight into how we got here, and what we're doing to fix the current problems.
How did we get here?
For over a year we have been growing the TypePad service and adding capacity as needed. This growth has been extraordinary, and we are now serving ten times the traffic we were serving a year ago. About three months ago we learned that the data center we were in had no more available power, with no option to obtain more. After some evaluation of options we decided that we needed to go through the process of physically moving out of that data center. And while we would need to grow capacity, we had to stay within the limits of the power available in our current facility.
At this point a race began to procure a new data center, buy more equipment and move data, and then servers, to the new facility.
We did a search for a facility that had the capacity for us to avoid getting into this situation again. We then leased space, numerous new servers and other equipment, added ten times the bandwidth of the old facility, got a dedicated data connection between the old and new facility and began to build out new cabinets to house the servers. Two weeks ago we began the move. Due to even more rapid growth than expected in the use of TypePad, we needed the capacity a week earlier than that. So basically we lost the race to move, and the excess load on the servers in the old data center started to cause failures in systems that have created poor service for you and, unfortunately, further slowed the move.
The engineering and operations team have been working around the clock to fix problems as they arise, while continuing the move to the new data center and get the new equipment in place and your data over to the new machines.
So where are we now?
Over the last two days we have made significant progress in relieving the stress on the servers in the old data center and completing the move. Below is a chart presenting specific improvements scheduled over the next few days and over the next few weeks. Many of these items were already mentioned in the post yesterday but I want to be sure you have all the data you need to understand the process.
Over the Next Few Days Over the Next Few Weeks
Storage Add 2 Terabytes of additional data storage Add new high-performance network storage system with over 10TB of capacity
Database Implement code improvements making the database more efficient on its current hardware Move to new high end database servers
Bandwidth 5X more available than last week Double the bandwidth again (10X over today!)
Application Introduce less database intensive logic to calculate stats Continue to make the application faster and more efficient
Power Enough power for the next 6 months of growth Power for the next year
Space Space to accommodate TypePad for the next year Options on another year's worth of space
Load balancing Ten times the bandwidth of today Double again
Backups 2 tier backups today 3 tier backup solution
As you can see a great deal of activity will be happening in the next few days. This Saturday night between the hours of 9PM β midnight PDT, we have scheduled downtime to complete some of these activities. Specific details are posted on our status page.
By the end of next week, we hope to have all this behind us and to be back to providing you the superior service you have come to expect from TypePad. We ask for a little bit more patience until that time. We all canβt wait to get back to the situation where our customers are delighted with our service and happy to recommend it to their friends. Until that time please accept our sincere apologies and thanks for your patience.
Sincerely,
Barak Berkowitz
Chairman and CEO
Six Apart, Ltd.
[email protected]
The makers of TypePad, LiveJournal and Movable Type
As you might know, some of our users have been experiencing slow performance with the TypePad service over the past few weeks. Ben and Mena, our founders, posted about this earlier this week. I want to provide more details and give you additional insight into how we got here, and what we're doing to fix the current problems.
How did we get here?
For over a year we have been growing the TypePad service and adding capacity as needed. This growth has been extraordinary, and we are now serving ten times the traffic we were serving a year ago. About three months ago we learned that the data center we were in had no more available power, with no option to obtain more. After some evaluation of options we decided that we needed to go through the process of physically moving out of that data center. And while we would need to grow capacity, we had to stay within the limits of the power available in our current facility.
At this point a race began to procure a new data center, buy more equipment and move data, and then servers, to the new facility.
We did a search for a facility that had the capacity for us to avoid getting into this situation again. We then leased space, numerous new servers and other equipment, added ten times the bandwidth of the old facility, got a dedicated data connection between the old and new facility and began to build out new cabinets to house the servers. Two weeks ago we began the move. Due to even more rapid growth than expected in the use of TypePad, we needed the capacity a week earlier than that. So basically we lost the race to move, and the excess load on the servers in the old data center started to cause failures in systems that have created poor service for you and, unfortunately, further slowed the move.
The engineering and operations team have been working around the clock to fix problems as they arise, while continuing the move to the new data center and get the new equipment in place and your data over to the new machines.
So where are we now?
Over the last two days we have made significant progress in relieving the stress on the servers in the old data center and completing the move. Below is a chart presenting specific improvements scheduled over the next few days and over the next few weeks. Many of these items were already mentioned in the post yesterday but I want to be sure you have all the data you need to understand the process.
Over the Next Few Days Over the Next Few Weeks
Storage Add 2 Terabytes of additional data storage Add new high-performance network storage system with over 10TB of capacity
Database Implement code improvements making the database more efficient on its current hardware Move to new high end database servers
Bandwidth 5X more available than last week Double the bandwidth again (10X over today!)
Application Introduce less database intensive logic to calculate stats Continue to make the application faster and more efficient
Power Enough power for the next 6 months of growth Power for the next year
Space Space to accommodate TypePad for the next year Options on another year's worth of space
Load balancing Ten times the bandwidth of today Double again
Backups 2 tier backups today 3 tier backup solution
As you can see a great deal of activity will be happening in the next few days. This Saturday night between the hours of 9PM β midnight PDT, we have scheduled downtime to complete some of these activities. Specific details are posted on our status page.
By the end of next week, we hope to have all this behind us and to be back to providing you the superior service you have come to expect from TypePad. We ask for a little bit more patience until that time. We all canβt wait to get back to the situation where our customers are delighted with our service and happy to recommend it to their friends. Until that time please accept our sincere apologies and thanks for your patience.
Sincerely,
Barak Berkowitz
Chairman and CEO
Six Apart, Ltd.
[email protected]
The makers of TypePad, LiveJournal and Movable Type
