Experts share strategies for crafting insightful rather than insulting critiques.
By Mary Lord
In academia’s publish-or-perish culture, peer reviews are the coin of the realm. For many authors, however, the system can seem flush with debased currency. Who hasn’t encountered put-downs masquerading as constructive criticism like those satirized in Reviewer #2 memes or hunted for a kernel of actionable guidance amid a dazzling display of 14-carat jargon? Even editors cringe at the perennial pan: “Not the article I would have written.”
Then there are the scholarly conventions and societal norms that work to stifle diverse voices. Clues that the publication playing field remains far from level include the documented paucity of citations by female scholars of color. One Black engineering education graduate student, stunned by what she perceived as biased language and racist ideas in a leading journal article, asked her adviser: “How does something like this survive the review process?”
Good question—and one that has provoked a flurry of reflection and responses. Over the past two years, journal and conference editors have overhauled policies, improved transparency, and increased supports for reviewers as well as writers. The goal, explains Journal of Engineering Education editor Lisa Benson, a professor of engineering and science education at Clemson University, not only is to “reframe the peer review process and make it more equitable and inclusive” but also “change the sense of reviewing as gatekeeping.” JEE now makes that commitment explicit in updated author guidelines and review criteria. Those criteria are aligned with a scoring rubric to rate manuscripts—including noting such items as the gender balance of the population studied—and are available on the publication’s website, along with such useful information as how the editorial board makes decisions and guidelines on conflicts of interest for reviewers. “People’s careers are on the line,” Benson underscores. “This is really important work. Let’s try to change the conversation.”
Julie Martin, an associate professor of engineering education at Ohio State University, is already rolling up her sleeves. Her aim as editor-in-chief of the Journal of Women and Minorities in Science and Engineering (JWM) is “to raise the standard of peer review for academic publishing.” For starters, the journal’s guidelines explicitly reject submissions from authors unwilling or unable to participate as referees on other manuscripts while theirs is being evaluated. Martin, who presented at the 2021 Collaborative Network for Engineering and Computing Diversity (CoNECD) conference on how to write effective journal or conference paper reviews “without being a jerk,” sees “constructive reviewing as a form of mentoring and a way to move our field forward.” Done well, the process enables “good research to be published (and sooner), leads to better publications, and helps authors be more productive and perform at their highest level.” That, she adds, boosts their odds of promotion while raising the quality of research in the field.
Former JEE editor Michael Loui, who used referees’ comments to broaden his understanding and inform his independent assessment of a manuscript’s strengths or flaws, likens the process to Winston Churchill’s take on democracy as a form of government: “a flawed method of evaluating quality, but all others are much worse.” IEEE Spectrum columnist Robert Lucky reached a similar conclusion in 1993. After musing on the merits of creating a “Consumer Reports” for engineering studies that would rate papers from “unacceptable” to “best buy,” he decided that a committee of colleagues was as good as it gets for now. “No one else can understand this stuff,” Lucky quipped. “Let’s face facts—we have enough difficulty ourselves.”
There’s no need to compound confusion or inflict pain, however. As Benson puts it, “We have to stop tearing each other down to show how much we know.” Turn crabby—or flabby—critiques into enriching feedback with these tips from the pros:
Critique the article, not the author. It’s “really easy” to dismantle a piece and “take potshots,” says Benson. “The harder task is saying how to make it better.” Her advice: Focus comments on content. Stephanie Cutler, who is studying the characteristics of productive peer reviews as co-principal investigator on a National Science Foundation grant, concurs. “We cringed the most when the reviewer used the word ‘you,’” says the associate research professor at Penn State’s Leonhard Center for Enhancement of Engineering Education. Constructive reviews, she and co-Principal Investigator Kacey Beddoes of San Jose State University have found, highlight why something is a problem and recommend ways to fix it rather than applying a “corrective lens” to pinpoint typos, citation omissions, and other minutia and perceived failings.
Be specific. “Constructive feedback uses examples from the manuscript wherever possible to help the author and editor understand what you mean,” explains JWM editor Martin. Other hallmarks of effective reviews—she has identified seven—include actionable suggestions for improvement and priorities for revising. Distinguishing global concerns about the whole article from those that apply to a specific section, such as in the research design, can help editors prioritize concerns and keep tabs on revisions, Martin adds. Flag no more than five major items that must be corrected “and spend lots of time digging into the pieces” to focus on the largest concerns, recommends Cutler. If the major items get fixed, “then some of the minor points go away.” Bonus advice: No copyediting in the first review round.
Identify strengths. Authors are more open to criticism “if they know that the reviewer has made a good faith effort to write a balanced review that first acknowledges the manuscript’s strengths,” argues Loui. Praise lucid writing and convincing arguments drawn from data. Does the research advance the field’s collective knowledge? Say so!
Strive for speed. “We all need feedback; our careers are dependent on it,” observes Martin. Research can’t be shared and used by others if it isn’t published in a timely manner. Prompt reviewing facilitates follow-up and enables authors to respond on deadline. It’s a long slog: Loui estimates that about 75 percent of JEE manuscripts were rejected after the first round of peer reviews, with the remaining authors invited to submit revised manuscripts that “invariably went out for another round of peer review.”
Check your tongue. “Be mindful of your tone,” cautions Cutler, who presented on crafting meaningful versus mean critiques at ASEE’s virtual conference in July. Positive, tactful language can boost an author’s receptiveness to feedback, but referees can sometimes come across as “judgy” or condescending. That doesn’t mean sugarcoating criticism or cloaking concerns in vague, tentative observations, though. “Unless ‘I thinks’ or ‘I feels’ are meant to soften the review, then they’re not needed,” contends Cutler. Kristen Koopman, a doctoral student in science and technology studies at Virginia Tech who participated in the virtual conference session, seconds that notion. Instead of a “snake fight,” she’d prefer pointers on what works or doesn’t in her manuscript.
Share your perspective. Authors don’t know who’s reviewing their work and vice versa. Conveying where you’re coming from as an expert in the field not only demonstrates you know what you’re talking about but also “helps the author and editor place the reviewer’s comments in context” and prioritize which critiques require a response, says Martin. Using field examples to highlight why you think a manuscript’s data analysis or conclusions are problematic can yield valuable insights. But avoid criticizing the author for failing to use the methods or conduct the study that you would have.
Backchannel concerns. Sensitive issue? Take advantage of notes to the editor, urges Cutler. Since authors don’t see these comments, “you don’t have to worry as much about tone and can be a little more blunt.” Editors also want to know about any uncertainties, such as limitations on expertise that prevent a reviewer from competently evaluating a particular aspect of a manuscript. But use this confidential channel sparingly, cautions Loui.
Invest in training. Though rarely taught, constructive reviewing is a critical professional skill that can improve with guided practice. JEE’s six-month Mentored Reviewer program pairs new referees with experienced hands to instill best practices from the outset. Workshops and conference sessions, such as the interactive presentation at ASEE’s 2021 Annual Conference on evaluating tone in peer reviews, are another way to glean tips of the trade.
Embrace reform. Efforts to make the peer review process more inclusive, equitable, and productive are starting to shift expectations. Since updating its author guidelines in July 2020, for example, JWM has required statements on positionality and the protection of vulnerable populations for all manuscripts—quantitative as well as qualitative. The latter, the author guidelines state, “should include measures undertaken as part of the research/writing process to protect those with marginalized or otherwise vulnerable social identities; if no measures were taken to protect these populations, this should be listed as a limitation of the work.” Transparency is the watchword at JEE, which now posts its updated review criteria online; previously they were sent via email and seen only by reviewers. Other scholarly publications are taking heed, and program chairs for engineering education conferences such as Frontiers in Education have expressed interest in adopting JEE’s revamped policies, reports Benson. Individual scholars, she suggests, can “take a simple piece” of improving the system, such as mentoring graduate students so constructive reviewing is part of their training and professional development.
Do unto others. “Peer review is a collegial process,” stresses former JEE editor Loui, noting that authors can choose to not adopt every suggested fix. Editors render their own, independent judgment on whether a submission deserves publication and sometimes will use multiple rounds of reviews to negotiate changes with authors. Above all, he advises, remember that “reviewing is not only a privilege; it is an obligation.” Those who submit manuscripts should reciprocate by reviewing other authors’ submissions. Martin concurs, citing participatory benefits that range from sneak peeks at new research to improved writing skills. “There are no immediate rewards,” concedes Loui, “but editors can recommend outstanding reviewers for promotions, awards, and membership on the editorial boards of journals.” In his “Perverse Law of the Academy,” the reward for good work is more work!
Mary Lord is Prism’s deputy editor.
Design by Toni Rigolosi