Verizon Report Acknowledges 911 Failures During June 29 Derecho

"The recent Verizon report on the 9-1-1 service failure after the derecho storm is a good sign that Verizon is acknowledging and correcting what went wrong on its part,” Fairfax County Chairman Sharon Bulova told us in a written statement. “I'm encouraged that Verizon is prepared to address the technical inadequacies of the system. This is, however, just a first step toward preventing another interruption of our critical 9-1-1 services.” The member of COG’s board first suggested the council investigate and form a task force in July.

"It sounds like Verizon is being pretty open and frank about what happened,” said Virginia State Corporation Commission Telecom Director William Irby. He’s “encouraged” after scanning the report and said it will be incorporated into the Virginia commission’s investigation of the outages. That commission sent Verizon questions and expected responses Tuesday, Irby said. Commission staff will file an interim report Sept. 14, he said. Virginia’s Secure Commonwealth Panel is also “discussing” 911 outages, not directly inspired by the derecho but with “added concerns” because of it, a spokesman for Gov. Bob McDonnell (R) told us. The Maryland Public Service Commission is investigating the derecho outages, a spokeswoman said.

Backup power “is one the things that we'll look up in the course of this investigation and the inquiry,” FCC Chairman Julius Genachowski said in July (CD July 20 p1). The FCC adopted a backup power requirement for wireless facilities under former Chairman Kevin Martin following Hurricane Katrina. The requirement was strongly opposed by carriers at the time and eventually withdrawn, in the face of disapproval by the Office of Management and Budget (CD Dec 2/08 p1) and a challenge by carriers in the U.S. Court of Appeals for the D.C. Circuit. After the derecho, it seemed “backup power was really the culprit” in the 911 outages, “confirmed” in Verizon’s report, Irby said.

"I don’t know if the [FCC investigation] will lead to raising that issue again, but we'll see,” a public safety official said of backup power rules. “But as Verizon indicated, backup power wasn’t the only reason why there were disruptions of service. I think that public safety would be interested in whatever it is that the network providers are doing to ensure their network remains up and running."

"External power failures affected more than 100 Verizon locations,” the telco said. “At each of these locations, batteries and nearly all the back-up generators worked as designed, allowing us to continue service.” But two generators failed, “disabling hundreds of network transport systems,” the report said. The failures led to a temporary loss of 911 service to four of more than 200 centers Verizon serves “across the storm’s path,” the report said: Fairfax County, Prince William County and Manassas centers “did not receive 911 calls for several hours Saturday, June 30, and another (Manassas Park) did not receive 911 calls for much of that weekend.” Problems were not limited to these four centers, Verizon said. It described “other 911-related problems, consisting primarily of a lack of delivery of location information on 911 calls and the loss of administrative and back-up phone” that affected “a number” of the many locations. Also failing were “systems that enable us to monitor the condition of our network facilities in Northern Virginia,” Verizon said, factors it said “hindered” assessment and repairs.

911 directors from Alexandria, Arlington, Fairfax, Loudoun, Prince William and Stafford recommended five “constructive” steps Verizon should take, the report said. The company outlined ways it'll approach all of them. Verizon said it has “strengthened” thresholds for emergency communications and “will make every effort to share actionable information” during disasters and alert customers of 911 failures via its current broadcast email procedures as well as soon by texting. The company has plans “to develop and exercise procedures for drills that model potential or actual 911 outages with any of the jurisdictions that request such a joint exercise,” it said. The jurisdictions asked for a contact list of Verizon officials tasked with the jurisdictions, updated monthly, and Verizon said it would provide a draft Friday.

The report underscored the importance of communication when emergency operations centers (EOC) are activated. A big post-derecho problem “was the lack of communication” between Verizon and the 911 centers, Irby told us. “We have discussed options for virtual participation in any EOC via an ‘instant messaging-like’ application with the Virginia Commonwealth emergency management leaders,” Verizon said. “We have discussed joint training with Fairfax Emergency Management personnel and would welcome the opportunity to participate in that activity.” Verizon pledged to “share additional information about our restoration efforts more quickly to provide greater insight” on outage duration and impact and that it’s “mobilizing a more robust emergency response communications process” for media and other channels. The 911 centers have discussed “a joint regional 911 EOC” strategy, according to Verizon, which said it would make “an excellent vehicle for Verizon to be present with multiple jurisdictions.”

Verizon “designs its network to provide 911 services even during disasters,” the telco said. “Our 911 network designs include multiple levels of diversity and redundancy, as well as back-up power in critical facilities, to optimize resiliency during a crisis.” At all “critical facilities,” Verizon installs both batteries and generators, the report said. “The batteries provide an immediate source of power following the loss of commercial power until the generators go online (which is designed to occur automatically), and then the batteries act as the back-up power source should the generators fail."

The report offers more details than were previously available on what went wrong at the Verizon facilities in the hours after the massive storm. Generators failed to start at each of the Fairfax and Arlington facilities, the report said. Technicians tried unsuccessfully to restart them. While power was restored using battery power until early the next morning, eventually the batteries were drained of power, the report said. Verizon later called in third-party experts to examine each of the two generators that didn’t start. At the Arlington facility, “We determined that air had entered the fuel system, resulting in a lack of fuel in the lines. We have since replaced the fuel lines for both of the back-up generators at the Arlington facility (even though no leaks were found in the generator that started),” the report said. In Fairfax an investigation found the generator did not start “because the auto-start mechanisms failed,” the report said. “Those mechanisms are designed to automatically start the generator once commercial power is lost, but they did not operate correctly and have since been replaced."

"Verizon understands the fear and frustration of people who were unable to reach 911 after the June 29 devastating storm that hit the region,” a spokesman said. “We're disappointed with our performance, and our comprehensive investigation has helped us understand what happened, why it happened and what needs to be done to prevent it from happening again. The corrective actions in this report will help us improve our performance moving forward.”