BTOP Grantees Embrace Richer Outcomes-Based Project Evaluation
It’s time to better evaluate how broadband stimulus has worked, the government and grant managers say. As federal Broadband Technology Opportunities Program (BTOP) grants continue to fund expansion of broadband throughout the states, the questions arise -- how effective are these grants? And how are they truly helping people? In the last year there has been “a tremendous shifting in the public-computing arena into collecting data and reporting and thinking about outcomes,” said Samantha Becker, a University of Washington professor responsible for assessing BTOP programs in her state who helps oversee the education of grantees across the country in 2012. BTOP grantees are “trying very hard” to properly evaluate how well their programs are doing, Becker said, describing as a barrier “the huge range in sizes and sophistication” of the grantee programs.
Sign up for a free preview to unlock the rest of this article
If your job depends on informed compliance, you need International Trade Today. Delivered every business day and available any time online, only International Trade Today helps you stay current on the increasingly complex international trade regulatory environment.
Details and history differ for every grantee. NTIA has pushed BTOP grantees to develop these mechanisms for evaluation throughout 2012 and held its first Evaluation 101 webinar in January and several others through the spring. Becker has played an active role in these webinars and described a broader goal of developing “common indicators” to allow researchers, program managers and government officials “aggregate results” and understand the impact of BTOP on a state and national level rather than on a program-by-program basis. “We're looking to develop kind of a playbook,” Becker said, explaining she hopes to have a tentative draft by the end of 2012. “The purpose of outcomes-based evaluation extends beyond compliance to include methodologies for evaluating programs and complying with grant requirements,” NTIA said in its summary of the first webinar (http://xrl.us/bnc7im). “It is an orientation towards performance management and is part of a cycle that helps orient an organization toward achieving results from programs and measuring impacts in society.”
Standardized evaluation instruments emerged from Philadelphia recently. The New America Foundation’s Open Technology Institute (OTI) has sought to develop the proper assessment tools throughout recent months, and their Philadelphia team has produced a workstation user survey, standardized exit survey, interview guides and scripts, tools which have received approval from two independent institutional review boards (IRB) and the NTIA. OTI released these evaluation tools publicly on their website on Monday to help other BTOP grantees (http://xrl.us/bnc7ey). The broader goal, according to the researchers, is “to assess the impact of the BTOP program on three major types of stakeholders: community members who use the BTOP programs, PCC facility assistants and SBA course instructors hired through the program, and the Philadelphia Freedom Rings Partnership staff.” The team emphasized the importance of “exploring the impact of the program on participants’ adoption, employment, education and community engagement.”
"Broadband adoption doesn’t happen overnight,” said OTI Policy Analyst Greta Byrum. She called the lack of broadband access in Philadelphia “a crisis” and “even worse than what we thought” initially, based on the statistics her team has begun reviewing. The federal government awarded Philadelphia $6,362,129 in BTOP money to fund the city’s Freedom Rings initiative, which was launched in 2010 and is focused on setting up 77 public computer centers and every week granting an additional 13,000 Philadelphians broadband access. It is expected to expire at the end of 2013. Many BTOP grants expand broadband infrastructure and fiber, but in Philadelphia, researchers are concerned about how people are using the new technology.
Philadelphia has opened 70 of the 77 planned centers and said the grant has served nearly 100,700 people so far, according to the latest NTIA report (http://xrl.us/bnc7cq). The program also is striving to educate 100,000 Philadelphians about broadband and encourage technology use within the home. “You can’t throw technology at people,” Byrum said. “You have to understand what their needs are.” Byrum and her colleagues have attempted to understand how these computer centers are working by looking at the broader communications habits of the city’s residents, in particular at the “very important” libraries and community centers that offer access as well as “intermediaries, support staff” that help people come online and communicate in different ways. Access is a far more complicated concept than what’s available in a person’s home, Byrum said. OTI researchers held an April workshop to discuss these evaluative methods with city residents, NTIA representatives and other BTOP grantees and had finished developing most of the research tools by then, Byrum said. Their surveys focus on both demographic information as well as, significantly, the purpose of those who come to the centers. Are they there to visit government websites, for email, for social networking? The researchers want to know. The next step is all about data, both quantitative and qualitative and intended for new monthly and quarterly reports to NTIA that elaborate on the major themes of what they're finding. “The plan is to move into data collection mode,” said OTI Senior Research Fellow Seeta Gangadharan. “Ultimately we want the programs to run effectively.” Their team began collecting this data in the second-to-last week of June.
The same trends are emerging elsewhere. California-based nonprofit ZeroDivide, which oversees two BTOP grants amounting to more than $2 million total, anticipated a need for evaluation early on. “Increasing capacity to do evaluation was built in,” said Senior Program Manager McCrae Parker. ZeroDivide has helped many of the west coast’s organizations, from Access Humboldt to the Boys and Girls Clubs of Santa Fe to Portland Community Media’s Youth Media Program, marshal BTOP funds and tighten evaluation techniques. Different program managers began coming together in a conference call about once a month this year to discuss methods and “I've noticed a growth in the confidence of the program staff to take on evaluation,” Parker said. “The commonalities [in metrics] have surfaced.” ZeroDivide hopes to unify all this evaluation work in a new and tighter model that will kick off in September, he explained. Their data collection is already being rolled into quarterly BTOP reports as is. Parker is aware of the new research instruments that Philadelphia has produced and said ZeroDivide “will be taking a look at them” to see if their organizations can “capture some of that information as well.” As much as he supports standardization of data, he said he wants to “keep our eye on the idea of qualitative, too” to capture “things that are regionally and culturally specific.”
The real value in this data collection is how “recyclable” the information will be, with uses in advocacy, in programming decisions, in future endeavors and as a guide to federal priorities, and in, quite simply, the reports to the NTIA that grantees have already been submitting over the course of the last two years, Becker said. One difficulty is the lack of money devoted to evaluation nationally, and as a consequence, the coordination efforts and education have relied on the goodwill of many volunteers, she said.