PLEASE NOTE:
The following information was conveyed to our customers during a major Internet outage in early November 2007. This shows how EcTownUSA used its "disaster recovery plan" to get all of its customers backup and processing in a short amount of time... long before the Navisite system was back up to 100%. Once the recovery was done our customers immediately started processing member information and providing important data from many very busy Chamber websites.
Information presented below is to prove that EcTownUSA is capable of recovering from an Internet disaster quickly with NO LOSS OF CUSTOMER DATA.
Notifications below are the latest communications to our customers (top) to earliest notifications that we are having problems with the network from Navisite (bottom).
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------
NETWORK UPDATE (1:11PM 11/6/2007)
Systems at 100% - Please Report Any Issues - Video Training Coming this Month
Good News,
The full recovery has been completed and we've been testing for the past three hours and can report that all service is fully restored. You should notice a much faster system response as we've upgraded to a much faster server array during this process. Please let us know if you run into any issues by REPLYING TO THIS EMAIL so that our engineering team and support can get to it immediately.
We appreciate all of the kind words we have received from many of you during this bad situation - that was not expected - Thank you.
On a Better Note...
- We will be releasing a new video training program on every module within the system by the end of this month... once the videos are completed (90 of them) you will be able to view a 2-5 minute video on each module so you will be able to view all of the new services and know how to use them. Some of the modules you may want to release to the public via your website to help add additional functionality. We will notify you once this area of the system training is released.
- Also a new program for automating the sales online and tracking of sponsor advertising will be added as well (Sponsor Tracker).
We understand that NaviSite is now operational, but the new system will be kept for obvious reasons and then we'll create another disaster recovery server array incase this was ever to happen again.
BACKUP NOTE:
Within all systems we offer you the ability to BACK UP ALL OF YOUR DATA on your local hard drive by going to MMS Configuration > Data Base Backup Info. As you can see we keep your data VERY SAFE, but if you like this service is there for you at any time.
--
Sincerely,
Richard Scully, Founder
EcTownUSA, LLC.
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------
(10:45AM PST 11/6/2007)
Dear Customers:
This has been a terrible ordeal for us too, but we have tried to make the very best decisions that we possibly could based on information provided by Navisite to restore full service in the fastest possible way. Navisite kept telling us in 2-3 hours and again in 2-3 hours and again and again (all night long)... obviously they have made some very bad decisions and caused us to delay our disaster recovery plan... we are sorry for that, but are pleased that the our disaster recovery host http://www.liquidweb.com/about/ and http://www.liquidweb.com/facilities/ is now fully online and we have all Chambers back up processing NOW and website automation is also restored through the letter "P" as of this very moment.
We will be working on removing all associations with Navisite for the Chamber Organizer and then re-establish our disaster recovery program.
I know we have lost some respect and hope you can forgive us for this problem that was out of our control. If one good thing comes from all of this is our ability to prove our ability to restore from a major outage with NO LOSS of data and restore full service at another host quickly.
We have moved the entire Chamber Organizer system over to a very reliable hosting facility and should experience no more interruption of service (I know!). You should experience faster response times soon as a new server array is being used to service all of you.
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Dear Customers:
We are moving from the Navisite Network to another hosting system in order to get our customers up and running this morning. Please use this login http://www.chamberorganizerbackup.com/members/mlogin.php to begin full processing. We will also be re-routing our DNS so that all website functionality will come up very soon too.
Please call if you have any questions 530-283-5553.
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------
PLEASE NOTE THAT ECTOWNUSA IS NOW RUNNING ON ITS BACKUP SERVER ARRAY WITH THE LATEST DATA (UP TO THE MINUTE BACKUP DATA USED FOR RESTORE). ALL CUSTOMERS SHOULD HAVE RECEIVED AN EMAIL EARLY THIS MORNING WITH LOGIN INSTRUCTIONS. PLEASE CALL US AT 530-283-5553 IF YOU HAVE ANY QUESTIONS. THE NAVISITE NETWORKING CENTER INFORMATION IS BELOW FOR LATEST INFORMATION. RESOURCES WE HAVE SPOKEN TO at 8:30 AM TELL US THAT ALL SERVERS ARE LIVE AND NOW WAITING FOR DOMAIN NAME RESOLUTION. WE'LL NOTIFY ALL OF YOU ONCE WE GO 100% LIVE AND SEE THAT THE NETWORK IS STABLE. BACKUP SERVER LOCATED AT:
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Navisite Clients - 11/06/2007 1:00 p.m. EST
Our two primary focuses currently are to bring all of the name servers back online as well as collaborating with Cisco to correct high levels of ARPs on the network. As more hosts were being brought on line, a significant number of ARP requests were overloading the network, so we were tasked with resolving stability issues.
Overnight, we were troubleshooting and resolved name server issues many more hosts were being brought on line. This required a network configuration change in the midst of resolving specific DNS issues.
We have continued to hand off completed environments to clients – at a much faster pace. We are doing this across multiple environments so that we are satisfied that standards for Quality Assurance (QA) have been met. Routing issues were resolved by assigning dedicated engineers on a client-by-client basis to support the process.
A full complement of engineering resources have been committed to completing the overall migration and have been working around the clock to resolve overall service issues and to resolve specific problems. We did have some improved service-level commitments in mind that have included:
Improve the level of service you had been receiving by leveraging a significantly larger and deeper technical organization
Move you to a data center that will provide a higher level of scale, redundancy and availability
Migrate you from servers that 2, 3 and 4 years old to brand new equipment – both servers and storage that will allow you to scale your business
Again, we regret the circumstances and apologize for significant inconvenience and remain positive that the long-term decision to migrate will ultimately be in our collective best interests.
And, we will continue to update our progress.
Sincerely:
Mark Clayman
Senior Vice President of Hosting Services
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Navisite Clients - 11/06/2007 9 a.m. EST
We just resolved the network routing issue that was referenced in the last update. We will now start working on bringing all of the name servers back on line as well as working with Cisco to correct the high levels of ARP'ng on the network. These two items are the primary success criteria in bringing the remaining hosts on line.
Navisite is going down in flames
Submitted by Wayne on Mon, 2007-11-05 13:30. Business | Technology
Navisite must be one of the worst web hosting companies alive today. Besides their fiasco of a migration of Alabanza, going on 4 days of down time for thousands of sites (up to 200,000 websites), they repeatedly lied to their customers about the recovery process. They missed every deadline they set for themselves--repeatedly. They failed to provide even basic protection for DDoS attacks even though their NOC was supposedly more advanced than Alabanza's. (There are doubts the reported DDoS attack was even real or if it was just another lie of theirs.) Communication was embarrassingly absent and when present, it was either a falsehood or merely vacuous. As of this writing, no attempt to remedy the enormous losses to those hundreds of thousands of customers has even been hinted.
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Navisite Clients - 11/06/2007 8 a.m. EST
As we were bringing more and more hosts on last night, we were creating a significant number of ARP requests on the network. The number of ARP requests is primarily due to the fact that we were still resolving name servers issues in part of the environment. The number of ARP requests ultimately overloaded the network which is why you were seeing instability.
In the process of troubleshooting the over utilization issue with Cisco, a routing error was made which has caused most of the environment to be inaccessible. We are working to reverse this change out with Cisco and will then work both the name server and ARP'ng issue.
Sincerely:
Mark Clayman
Senior Vice President of Hosting Services
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------
11/05/2007 10:00 p.m. EST
We are making continued progress in troubleshooting specific issues and bringing up individual hosts - as we fix them. The root cause of the DNS issue that we have been experiencing - and which has prevented us from bringing up the additional hosts - has been identified.
We will be making a network configuration change at 11 p.m. EST that will take approximately 15-30 minutes to resolve the issue. We expect this will solve both email and name resolution issues that have been identified, and we will continue our troubleshooting efforts
Sincerely:
Mark Clayman
Senior Vice President of Hosting Services
NaviSite
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Posted information on the Navisite move:
News
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Dear Customer,
As you know we are fully committed to providing our customers with excellent service and feel terrible about what is going on today. We work with three major hosting facilities all providing excellent services and high uptime guarantees.
Currently a major hosting firm called Alabanza which hosts our database components, was bought by a significant international hosting company called Navisite. Upon completion of their purchase they decided to combine data centers. This was supposed to have taken just a day over the weekend to complete and as we all know it is still not back up. We keep informing all of you that it will take another hour or two, as this is told to us. After deciding to call in on the BRIDGE conference call I found many large hosting company CEO/CTO’S present on the line and are understandably frustrated.
I have placed links below to the resources we are using to monitor the situation and are very sorry.
We decided not to re-start via a backup system for disaster recovery because of the many promises given to us by the Navisite personnel always indicating a fix was just around the corner… we are still waiting and it still seems best to wait in this case.
Richard Scully
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------
As of 2:35am PST on 11/5/07 we have been informed by the Navisite Networking Center that 90% of systems should be active within the next 1-2 hours. 1/4 million or more sites have been affected and are truly sorry for this system interruption. We are monitoring the situation and have been calling customers about updates as we know them. This have been caused by the moving of a HUGE datacenter that is affecting many major hosting facilities all over the World.
