Links to Other ASM Pages:

• Table of Contents
• ASM News Issues

Last March, members of the Committee on Science, Engineering, and Public Policy (COSEPUP), a component of the National Academy of Sciences (NAS) in Washington, D.C., announced a successful first test of benchmarking--a relatively quick and efficient approach to comparing specific disciplines or broader fields within the U.S. sciences to their foreign counterparts. Moreover, because one part of the test entailed evaluating immunology, a COSEPUP panel concluded that the ``United States is the world leader in immunology'' and that ``dominance is also evident in the major subfields'' of this discipline.

International benchmarking provides a way of comparing the quality and impact of research in one country with world standards of excellence for that discipline, according to Marye Anne Fox, chancellor of North Carolina State University in Raleigh, who chaired the NRC committee that conducted these tests. The underlying goal is to develop a ``reliable tool that allows us to evaluate the quality of our research programs and to provide us with a basis for funding allocations,'' she says.

Fox and other COSEPUP members deem this initial test of benchmarking ``an encouraging first step'' toward developing better assessment methods for scientific disciplines that also could lead to better decision making by those who fund science and technology. ``The benchmarking mechanism can work,'' she says. Not only does this approach provide a snapshot describing the relative position of U.S. researchers in a particular discipline, it does so ``rapidly and relatively inexpensively,'' she says. ``It's possible to take a snapshot and the results correspond to quantitative indicators.'' The results also can be used to ``define fields where international cooperation should be pursued.''

To do benchmarking, ``You've got to have honest world leaders in a field, who are willing to put aside their egos and say who made the most significant advances in a field during the past five years,'' says Irving Weissman of Stanford University, Stanford, Calif., who led the benchmarking test involving the field of immunology. ``Because we spent a lot of time figuring how to do this in an efficient manner, I feel very confident we can do the same exercise for another field.'' Moreover, side-by-side analyses of such fields might be used to point to explanations as to why one field may be lagging and what might be done to bring it up to speed.

This same kind of evaluative approach also works within the chosen discipline for looking at subfields. Although the COSEPUP benchmarking review assessed U.S. immunology research as overall ``the world leader,'' it also identified ``problems with clinical studies,'' Weissman says. For instance, because health care is more centralized in many European countries, clinical researchers can move more quickly to begin clinical trials in areas such as immunology. ``In the U.K.,'' he notes, ``there is no problem with HMOs competing with academic centers for patients. That's one reason why the U.S. is behind there.''

The current effort to develop and test benchmarking traces to the past decade. During the early and middle 1990s, COSEPUP developed several reports recommending that the U.S. government continue vigorous funding of research across the board, and that U.S. researchers maintain leadership in many areas of science and remain ``among the leaders'' in other areas. In one of those reports, a COSEPUP-appointed panel concluded that ``field-by-field peer assessments'' would be useful, and it recommended establishing independent panels for that purpose. Such panels were to include researchers who work in the field being evaluated, individuals from closely related fields, research ``users'' who closely follow the field, and foreign scientists.

More recently, several COSEPUP groups were formed to test the applicability of these principles to three scientific fields--immunology, mathematics, and materials science and engineering. In each case, a panel with about 12 members met once, subdivided the field, and subsequently conferred by e-mail and telephone while using several methods to evaluate it, according to COSEPUP member David Challoner of the University of Florida in Gainesville. Those methods include identifying which researchers they would choose to invite to imaginary ``virtual congresses,'' conducting citation and publication analyses, counting numbers of graduate students and looking at who is receiving prizes within the discipline, and determining which scientists in a particular field are being chosen to participate in actual international congresses.

Among these exercises, COSEPUP members seemed to take special delight in their discussions of virtual congresses, a concept and exercise that proved useful for all three panels, according to Challoner. ``Each panel asked leading experts in the field to identify the `best of the best' researchers for particular subfields anywhere in the world,'' he says. Then they polled leaders throughout the world and asked them to ``imagine themselves as organizers of a session on their particular subfield and to furnish a list of 5 to 20 current desired speakers.'' Such lists, indicating who among researchers would be chosen to speak at these imaginary conferences, thus serve as an index of how the United States is doing in a particular field.

Fox says that benchmarking might also prove useful to federal research agencies in efforts to comply with the Government Performance and Results Act (GPRA), legislation that entails a full-scale performance review as part of a periodic report to Congress. Benchmarking also might complicate GPRA evaluations because its measure of scientific disciplines or fields is not conducted within typical federal agency boundary lines, according to Challoner. Moreover, Weissman says, one way to augment these current approaches to benchmarking--and as a way of applying this evaluative mechanism to GPRA reviews--might be to assemble review panels that include consumers as well as experts.

Jeffrey L. Fox