Department Issues Report on E-9-1-1 Outage

Errors Contributing to 2 1/2 hour Outage across Northern Vermont on September 3 Identified
Montpelier, Vermont - The Department of Public Service today released a report detailing the results of a thorough investigation into the causes of an outage that left Vermonters across northern Vermont unable to call 9-1-1 on the morning of September 3, earlier this year. The report concludes that the outage was the result of a series of preventable errors or omissions by FairPoint and Verizon, operating under contract to FairPoint, in implementing system changes requested by the E-9-1-1 Board, which operates the E-9-1-1 system in Vermont.
The report also concludes that the E-9-1-1 Board and its contractor, microData, missed opportunities to detect some of these errors before the outage occurred. FairPoint provides key elements of the E-9-1-1 network, and Verizon continues to provide network monitoring and other services to FairPoint under contract until FairPoint completes a cutover to new operations support systems after the end of January 2009. The report's conclusions come after the Department interviewed and sent data requests to the E-9-1-1 Board, FairPoint and Verizon. "An outage of the E-9-1-1 system is absolutely unacceptable," said Public Service Commissioner David O'Brien. "We conducted a detailed investigation of this incident so that Vermonters and their first responders can have confidence that when they call 9-1-1, their call will get through."
The Department's report identifies nine errors that contributed to the approximately 2 1/2 hour outage on September 3, 2009. It describes the types of redundancy built into the E-9-1-1 network that should prevent outages and concludes that no single error that occurred would have by itself caused the outage on September 3. Instead, a series of errors occurring between August and September, when FairPoint implemented the network changes requested by the E-9-1-1 Board, led to a situation where FairPoint's network was attempting to direct E-9-1-1 calls down lines that had been disconnected, instead of down lines that continued to be available to carry calls to 9-1-1 answering points. The report concludes that these errors happened in part due to failures to use or follow standardized procedures. While the report does not conclude that the outage was caused by flaws in the network design, it offers recommendations to strengthen procedures at FairPoint and the E-9-1-1 Board, as well as recommendations to further strengthen the E-9-1-1 system itself.
The Department is considering additional action as a result of its investigation. "Our first priority has been to make sure that everyone who has a part of making the E-9-1-1 network work in Vermont had a detailed understanding of what went wrong, so that the problems that led to this event can be corrected and events like it prevented in the future," said O'Brien, "but we also have a responsibility as a regulator to see if any of the companies we regulate violated any regulation or requirement, and hold them accountable. We are very close to reaching a conclusion on that point."
The Department's investigation was conducted with the cooperation of the E-9-1-1 Board, FairPoint, and Verizon. "The Department's investigation has built on work that the E-9-1-1 Board has done to conduct its own investigation of this incident," said O'Brien. "We also appreciate that FairPoint and Verizon have made available their own technical staffs to answer questions in order to help us understand how this incident happened."
O'Brien also praised the staff of the Department assigned to the investigation. "It has been important that we uncover exactly what went wrong, and developing a full and correct understanding of the causes contributing to this outage took persistence and hard work by the Department's technical staff over a number of months. I want to recognize that good work."