Data Collection Errors Compromise FCC’s Broadband Speed Test Efforts
An interconnection problem in Mountain View, Calif., caused six days’ worth of broadband data to be compromised, leading the FCC to throw out data for the second month in a row, the commission reported Thursday. In April the commission restarted a month of data collection because of “anomalies” in the network affecting some of the measurement locations in March (CD May 7 p8). Stakeholders we contacted aren’t worried about the accuracy of the data for the remainder of April, but some are concerned about the reliability of the collection platform, given the repeated performance issues.
Sign up for a free preview to unlock the rest of this article
If your job depends on informed compliance, you need International Trade Today. Delivered every business day and available any time online, only International Trade Today helps you stay current on the increasingly complex international trade regulatory environment.
As a result of the diagnostics put in place after the March anomalies, “performance irregularities” were observed in April at the Mountain View M-Lab node, according to an ex parte filing written by Walter Johnston, chief of the FCC Electromagnetic Compatibility Division, describing a June 6 meeting (http://xrl.us/bndbif). Initial analysis suggested the performance degradations occurred for less than 24 hours, but “further analysis” showed that “the duration of the event could be first observed in the morning of April 8, persisting until being resolved in the evening of April 13,” Johnston wrote. The filing was dated June 12 and posted in docket 09-158 on Thursday.
The issue was “the fault of an ISP peering/interconnection problem,” said New America Foundation’s Benjamin Lennett, a representative of M-Lab. That “not only affected M-Lab’s server in Mountain View, but any other ISP customer traffic that passed through that interconnection point,” he told us. M-Lab worked with the interconnection facility and affected providers to fix the issue, he said. Steve Bauer, an MIT consultant advising the FCC, told us the incident was an issue with an upstream provider of network connectivity.
FCC contractor SamKnows said the performance drops “demonstrated an easily recognized traffic profile,” and could be removed from the reported results “without statistical impact,” Johnston wrote. It said commission staff suggested that although only providers with clients tested against the Mountain View server were impacted, “excluding data for the period for all providers would preserve the fairest baseline for comparisons of all providers in the upcoming report.” An FCC official told us the commission has “complete confidence in the data as we prepare the Measuring Broadband America Report. No compromised data about which any concerns have been raised will be used.” The Measuring Broadband America Report 2012 is scheduled to be presented to commissioners at the July 19 meeting.
"The FCC is actually being extremely cautious in their approach,” said David Young, a Verizon vice president. The commission has been very open to input, and the company has “a lot of confidence in the process,” Young said. “This is something that they themselves identified and brought forward and made the decision to exclude all of that data. So they're certainly being proactive in trying to find any anomalies.” Verizon FiOS fared quite well in the 2011 test, and its promotional material often touts a chart showing FiOS to be the fastest and most consistent high-speed Internet service.
Charter Communications, also involved in the discussions on M-Lab data, agrees with the FCC’s approach to exclude all data from all providers and locations for that six-day period, said Vice President Christin McMeley. Charter has also engaged an independent third party to review its data, and has “been assured that there were no material performance issues for the remainder of April,” she said. The cable company does have concerns about the collection platform “because of the repeated performance issues,” and is working with the FCC to “find solutions on a going-forward basis,” she said.
Mediacom is “unsure about this round of testing because we haven’t seen the results yet,” said Vice President Tom Larsen. Mediacom didn’t score as highly as some other ISPs in last year’s broadband test, and has criticized the small size of the test sample for overemphasizing the “anomalous results of just a few geographically remote users” (CD Feb 21 p11). The cable operator has expressed in previous filings that the testing nodes are “constructed around [NFL] football cities, and that may not be the way the Internet is constructed,” he told us. “Unfortunately, not every provider provides service in football cities, and we just happen to be one of those."
Meeting participants expressed concerns regarding the “instabilities observed on the M-Lab platform,” and asked the commission what its plans were for addressing potential future anomalies, Johnston wrote. He said the new finer-tuned diagnostic tools are “capable of identifying and resolving issues much more quickly,” and participants discussed the potential use of donated test nodes for “increasing the diversity of measurement sources with new submissions.” Young thinks the various anomalies from March and April were just bad luck. Computers are “not perfect, and things can and do go wrong, so you just have to anticipate that and build that into the processes,” he said. The FCC and participating companies will look at what can be done to include more redundancies “so if there is a problem with one of the test servers, there is an alternative source,” he said. “It’s all learning as we go."
Measurement methodologies, testing infrastructures, analysis and reporting procedures, and broadband networks “can always be improved,” said Bauer. But he has no particular concerns about the testing process. SamKnows, M-Labs and the FCC “should be commended for having set a high bar of quality for the data used in producing the FCC’s summary reports,” he said. “Anything less would not be fair to ISPs.”