On Monday, with the end of November behind us, we were gearing up for our semi-regular update on availability for the month. Then, of course, we had a completely coincidental, prolonged and unplanned downtime and have been focused there ever since.
We take performance and availability extremely seriously. Since the incident, we’ve been working around the clock on addressing the underlying issues, monitoring the Ning Platform and identifying the plan from here.
But first, the uptime stats for November. For the month of November, we had uptime of 99.68%. You can see the monthly graph here:
So, that’s the good news.
On Monday, as we were rolling out a new release to the Ning Platform, we hit upon a connection limit issue that made inaccessible one of the key databases in the system. Given the data security we apply to all our databases, no data was lost.
However, because of the time it took to unequivocally identify the problem and ensure the that the entire platform was in a coherent state, along with testing, we needed three hours to bring the platform back online.
The Plan From Here
We’ve been stable since Monday night and are in the process now of addressing the underlying issues that triggered this problem. Specifically, we are working on a set of optimizations around one of the mechanisms we use to distribute information regarding a user’s friends across different clusters that will ensure the system continues to perform well and remains highly available. Having doubled in traffic since June, we hit this particular pressure point faster than we had anticipated.
While we are currently working on the solution, it is possible that we may have to quickly roll out some temporary changes intended to relieve the systems that are currently under stress, and these temporary changes may affect how quickly information related to your friends on Ning is available, and whether it is displayed exactly as it is today. The important thing to keep in mind about these changes is that they will be temporary, lasting for a couple of weeks at most, and they will have no impact to functionality of your social networks aside from temporarily making it more difficult to access information or content related only to your friends on a network.
We’ll do our best to keep everyone posted on our progress here. In the meantime, please don’t hesitate to let us know if you have any questions about any of this work in the Ning Help Center.
Thanks for your patience, flexibility and understanding as we continue to drink out of the firehose of amazing social networks and phenomenal growth thanks to your creativity and passion. That’s what makes weeks like this worth it.
No related posts.