FCC Investigating Widespread AT&T 911 Outage

Industry officials said there was apparently a failure in links between AT&T’s lines and TeleCommunication Systems (TCS), a company that routes calls to many public safety answering points. The problems involved voice over LTE calls in particular, though public safety answering points able to receive emergency text were able to do so, and thousands of 911 calls likely were affected, industry officials said. AT&T attempted to route some calls to West, which is part of Intrado. The problem was broadly felt, but apparently not nationwide, industry officials said.

“Every call to 911 must go through,” FCC Chairman Ajit Pai said in a statement. “So when I first learned of yesterday’s outage, I immediately directed FCC staff to contact AT&T about it and the company’s efforts to restore access to emergency services to the American public.” Pai said he spoke with AT&T CEO Randall Stephenson, “and stressed the urgent need to restore service and to communicate with first responders, as well as AT&T customers, about the status of operations. Additionally, I announced last night that I have directed Commission staff to track down the root cause of this outage.”

“We take our 911 obligations to our customers very seriously and will be sharing additional information with the FCC," an AT&T spokesman said.

A former spectrum official said the FCC’s approach is likely to reflect the differences between Pai and former Chairman Tom Wheeler and their regulatory philosophies. Wheeler was likely to move more quickly to propose rules when he thought they were necessary, while Pai brings a more anti-regulatory mindset, the former official said.

AT&T contacted the National Emergency Number Association and 911 officials Wednesday to say there was an outage and it was being addressed, consistent with FCC rules, said Brian Fontes, NENA CEO.

“AT&T has always had a very good record on public safety and emergency communications,” said Jamie Barnett, former chief of the FCC Public Safety Bureau, now at Venable. “They have a staff that pays attention to it, and in this case, the FCC will want to know what happened and what they’re going to do to prevent this” from happening again, he said. “The carriers are greatly incentivized to keep their networks operating,” Barnett said. Otherwise “they hear from their customers who have other options most of the time.”

The National Association of State 911 Administrators is waiting to learn more about what happened through the FCC investigation, said NASNA Executive Director Evelyn Bailey. “All I have heard from my members is that several of them were affected by the outage and that some of their PSAPs had received notices from AT&T or one of its vendors to the effect that there was an outage impacting wireless call delivery and phase I and II location information,” she emailed. “No other details were revealed in those notices.”

Local Impact

Local officials said effects were widespread, in interviews and statements.

“This is a serious matter,” emailed Texas 9-1-1 Alliance CEO Jim Goerke, a working group chairman last year on the FCC Task Force on Optimal Public Safety Answering Point Architecture. “We are still assessing what kind of impact this had on wireless 9-1-1 call volume from AT&T Mobility. … If this was truly system wide, then that is really concerning, and speaks to the potential for single points of failure. But, not drawing any conclusions until we get more detail.” Alliance members “received the same AT&T Mobility E911 Outage notice as everyone else across the country,” he said. “We know that the Commission is following up on it, and will closely monitor the cause and subsequent corrective action.”

Washington, D.C., responded to the “potentially serious” outage by providing the public with direct numbers to the 911 center, said an Office of Unified Communications spokesman. The outage affected only AT&T wireless customers, he said. “This is why we develop contingency plans and are prepared to move quickly in the event of an outage like this,” he emailed. “There will be ongoing conversations with the carrier as we maintain relationships with every agency and vendor that interacts with the 911/311 center.” Historically, 911 outages tend to happen regionally, the spokesman said. “The issue for us is that the carrier let us know of a service disruption as soon as possible so we can get our backup numbers out. We don’t know who’s not getting through to us -- so we need awareness as quickly as possible. And they’re generally good about that.”

Minnesota had “intermittent 911 calls not able to reach a PSAP,” requiring public safety to communicate alternative contact methods to the public, said Department of Public Safety Emergency Communication Networks Director Jackie Mines in a statement. Carrier 911 outage notifications to PSAPs aren’t always useful, she said. “The FCC has mandated that carriers provide immediate notification to PSAPs when they are experiencing an outage. However, Minnesota’s experience with the notifications is they are not timely, nor do they contain enough information about the source and scope of the issue for our PSAPs to offer alternatives to our citizens.” The Minnesota department received notification from AT&T at 8:13 a.m., but the email said the incident start time was 3:52 a.m., Mines said. “The email stated they identified a potential impairment affecting more than one location.” Migration to Next-Generation 911 will require discussions among the FCC, carriers and PSAPs to mitigate future occurrences, she said.

PUC Jurisdiction

State utility commissions may lack jurisdiction to probe the outage because it was wireless, depending on the state. That’s true under D.C. code, said D.C. Public Service Commission Chairman Betty Ann Kane. News reports indicate the outage was limited to AT&T wireless customers and didn’t affect wireline providers that could be regulated, she emailed: “Thus, at this point the Commission does not plan to initiate an investigation into this unfortunate and troubling service outage due to a lack of jurisdiction.”

Some observers expressed concerns the FCC won’t take a tough enough stance. “How fortunate for AT&T Wireless that the new FCC chair doesn't believe in that whole ‘enforcement’ or accountability thing for 911,” Public Knowledge Senior Vice President Harold Feld said in a Facebook post. “Think I'm kidding? Go read Pai's dissents on the FCC's previous ‘sunny day’ outages.” In May, Pai partially dissented when the FCC approved revised network outage reporting rules and a Further NPRM that would extend reporting requirements to broadband providers (see 1605250061). But in October 2014, Pai expressed concerns (see 1410170057) over a widespread 911 outage, saying then: “911 call centers must remain vigilant and hold their service providers accountable.”

"Problems like these show that no matter how attentive carriers may claim to be and actually are, the set it and forget it attitude of” FCC Republicans “has real-world consequences,” said Matt Wood, Free Press policy director. “People don't just want promises from the provider that these life-saving systems should work, or after-the-fact clucking from the FCC about investigations,” Wood said. “They want assurances, before a widespread problem like this occurs, from the government officials who are supposed to be safeguarding the public interest."

“I expect Chairman Pai’s first impulse would not necessarily be the headline-grabbing fine or new regulations, but I’m sure he intends to get to the bottom of what happened and assess whether there is anything the commission can do to help prevent similar incidents,” said Doug Brake, telecom policy analyst at the Information Technology and Innovation Foundation. “LTE networks are more complicated than the PSTN [public switched telephone network], and have more potential points of failure. Five nines was always an ambitious target and has to be re-evaluated with more complex services. But a nationwide, 90-minute outage is a pretty big deal, and availability of IP systems increasingly important as the IP transition and FirstNet deployment moves forward. It’s definitely worth the commission’s attention.” Five nines refers to 99.999 percent uptime.

“Even when people are doing their best and behaving as they should, things happen,” said Richard Bennett, network architect and free market blogger. “‘Shoot first and ask questions later’ is no way to regulate an industry, so I look forward to the analysis of the 911 outage.”