NSF strives to get its ‘Broader Impacts’ criterion taken more seriously by researchers.
With $4.5 million riding on their proposal, Jeffrey Forbes and his colleagues left little to chance when they sought National Science Foundation support in 2011 for “a realistic global three-dimensional model of Earth’s Electrical System from the surface into the ionosphere.” Their idea was intriguing, even exciting. The model would let scientists around the world try to solve the mystery of how electricity flows between Earth and the atmosphere, probing a complex system that encompasses lightning, electrified clouds, cosmic rays, ions, magnetic fields, and radon from beneath Earth’s surface. But even if peer reviewers agreed the proposal had intellectual merit, how could they be persuaded of its “Broader Impact,” the second criterion that NSF requires proposals to meet?
“The stakes were very high,” recalls Forbes, a professor of aerospace engineering at the University of Colorado, Boulder. “When you’re dealing at that level, you have to check every box and do it very well.” His team addressed not one but several of the five “broader impacts” categories. Societal benefit? Check: The team said its research would improve predictions of electrical storms, which not only disrupt communication and navigation systems but threaten security infrastructure, pipelines, and the overall economy. Education and dissemination? Check. The project would “educate wide audiences through engaging and informative education videos on YouTube, YouTube EDU, and YouTube University, through an educational web site, and through a new Sun-Earth system exhibit.” All this would be documented, assessed, and disseminated. They would also engage the scientific community through a website, international partnerships, and a major conference. Broadening participation? They had it covered. The project would “train and educate undergraduate and graduate students and postdoctoral researchers [and] attract talented students from underrepresented groups.”
Too Often an Afterthought
Forbes says his team “took [the broader impacts requirement] pretty seriously.” But such isn’t routinely the case, NSF has found. Too often, researchers try to get by with what Pramod Khargonekar, NSF’s former assistant director for engineering, has called perfunctory “drive-by broader impacts.” An example might be a program with a nearby junior high school. That kind of proposal, he says, presents broader impacts “as an afterthought, instead of thinking up front.” Nor are researchers the only ones who can be faulted. According to a presentation earlier this year by Suzi Iacono, head of NSF’s Office of Integrative Activities, outside panels of experts have found a lack of in-depth comments from reviewers on “broader impacts.” The relative weighting of “intellectual merit” and “broader impacts” is inconsistent, and reviews, panel summaries, and review analyses reveal variations in expectations, she reported.
Judging from their comments, reviewers of the Forbes proposal gave more thought and attention to the project’s technical and scientific value. One wrote simply: “The Broader Impacts seem fine.” Another stated that the broader impacts “are typical of others I have seen and more than adequate.”
Introduced in 1997 and mandated for all proposals in 2002, the “Broader Impacts” criterion aimed to show Congress and the public that taxpayer-funded research satisfies more than ivory-tower curiosity and will benefit society. But almost from the time it was introduced, “Broader Impacts” has produced confusion and inconsistent responses from researchers and reviewers alike. Interpretations also varied within NSF itself. Some scientists considered the requirement a distraction that could slow down the research itself. Khargonekar attributes the confusion to a “lack of an in-depth, shared understanding” of the criterion as well as to NSF’s reluctance to be prescriptive. “It does not want to tell the PI community what to do,” he says.
One problem is defining “broader impacts.” If it means the importance of a project to society, that’s hard to ascertain with basic research. It can take years for the significance of a particular discovery to become obvious. Who would have imagined, for instance, that a grad student working on Hector Garcia-Molina’s “digital library project” at Stanford in the 1990s would go on to co-found Google, or that AT&T’s need to replace vacuum tubes would inspire integrated circuits, Intel, and the beginning of Silicon Valley?
NSF, however, lists “benefit to society” as just one of five “broader impacts” categories. If researchers can’t predict how important their work will turn out to be, they can fulfill the requirement by: Advancing discovery and understanding while promoting teaching, training, and learning; broadening participation of underrepresented groups; enhancing infrastructure for research and education; and broaden dissemination to enhance scientific and technological understanding. “Pathways to impact,” the British equivalent of broader impacts, is more explicit. It means “fostering global economic performance, and specifically the economic competitiveness of the United Kingdom, increasing the effectiveness of public services and policy, and enhancing quality of life, health and creative output.”
Kat Steele chose a hybrid approach when she applied to NSF for an early career grant. An assistant professor of mechanical engineering at the University of Washington, she looks for ways to improve exoskeletons and other assistive devices for people with disabilities, such as stroke patients or those with cerebral palsy. Her proposal called for modeling experiments aimed at predicting nerve and muscle responses to ankle orthoses. If successful, the results would lead to greater mobility for some of the 1 billion people worldwide with neurological disorders.
“I’m pretty far on the ‘applied’ side,” Steele says, so anticipating the impact of her research is “pretty straightforward.” It can be summed up in two words: walking ability. But Steele also knew that NSF’s coveted CAREER award program demanded more. “They’re not only reviewing research, but education and outreach,” she says, and was happy to comply. “I do really enjoy teaching a lot.” Her proposal included “a multidisciplinary education program focused on human engineering” for both engineering and medical students. “Students will participate in journal clubs, coursework, and design projects where they will create open-source orthoses for individuals with disabilities. We will also develop open-source outreach modules focused on human engineering in partnership with local programs that encourage under-represented groups to pursue careers in engineering, including women, minorities, and individuals with disabilities.” The combination won over reviewers. “Most of my comments said it was a good fit between research and education.” She’s now in the second year of a $518,000 grant.
The University of Utah’s Rajesh Menon, Jason Shepherd, and Erik Jorgensen likewise incorporated a varied educational component in their project to obtain high-resolution images from deep inside the brain—where memory resides—of freely moving animals using relatively inexpensive using computational cannula microscopy. Their project, which has been awarded $925,000 to date, aims to get tangible evidence of the theory that changes in the strength of individual synapses underlie learning and memory. Their “broader impacts” included using research outcomes in their teaching, links with summer laboratory internships offered to underrepresented minorities, TEDx and social media platforms to reach undergraduates, high school students, and the public, and a promise of broad research dissemination.
Successful proposals often include partnerships with education and outreach experts to spare principal investigators from spreading themselves too thin. “Partnering is absolutely the right way to go,” says Khargonekar, now the vice chancellor for research at the University of California, Irvine.
The Earth’s Electrical System project led by CU’s Forbes, for instance, engaged an assessment expert and partnered with the National Center for Atmospheric Research and the University Corporation for Atmospheric Research, an NSF-sponsored academic consortium. “They have professionals in public outreach,” Forbes says. Among other activities, UCAR’s Center for Science Education produced a lively Earth’s Global Electric System cartoon video and teacher’s guide. Steele collaborated with both her university’s Prosthetics and Orthotics program and a staffer from the Center for Teaching and Learning, who helps plan classes and outreach and conduct evaluations. Appealing to NSF’s increasing interest in entrepreneurship, Utah’s Menon team included a plan to market its brain imaging tools, tapping the university’s Technology Venture Commercialization Office and the state’s Science and Technology Research initiative. “[W]e expect this project to lead to intellectual property, commercialization, and high-value job creation,” the proposal said.
Researchers’ quest for worthwhile ways to expand a project’s significance gave rise to the National Alliance for Broader Impacts, an NSF-funded organization that holds annual summits and workshops. “NABI makes it easy for researchers to connect with program partners who can deliver their Broader Impacts,” a promotional video says. “I think that’s a way to foster a community of broader impacts experts,” says Britt Holbrooke, an ethics specialist at the New Jersey Institute of Technology who has studied the criterion and how it is applied. While NABI’s effort is legitimate, “I don’t think it should be the only option available,” he says, since it could let scientists and engineers hand off this work to others. “Part of the point [of the criterion] was to show that researchers are thinking about broader impacts.” Khargonekar has urged colleges and universities to come up with institutional goals that each proposal could serve.
After finding that broader impacts “is not generally well understood” and its execution flawed, the National Science Board in 2011 attempted to clarify them with new guidelines and a set of principles. But four years later, there were still “[p]ersistent questions from everyone, really, about how BI is applied at NSF,” Iacono told the 2016 NABI summit. These prompted directorate discussions followed by an agency-wide review. “Recommended actions for discussion” included more data analysis and giving additional guidance to researchers, reviewers, and committees of visitors.
Entering the final year of his Global Electric Circuit, CU’s Forbes takes pride in having led a multi-institution team of more than three dozen engineers, scientists, undergraduates and graduate students in producing “a lot of great stuff,” including “a huge number of publications.” Broader impacts “is the hardest part sometimes of doing these proposals,” he concludes, but it’s also “a necessary and important part of what we do.”
By Mark Matthews
Mark Matthews is editor of Prism.
Design by Francis Igot.
Image courtesy of Thinkstock.