Abstract: Panel peer review enables scientists to pursue new and exciting research projects with the allocation of funds and fellowship positions. However, the panel peer review process is a group decision making process and social psychology states that people make decisions differently when in groups. This paper applies this aspect of group dynamics to the panel peer review process for grant and fellowship applications, seeking to find if specific phenomena impact decision making within these panels. Four phenomena – group polarization, groupthink, common-knowledge effect, and conformity - are discussed and applied to the process. Numerous risk factors for each phenomenon were found, including similarity of panel members, lack of discussion time, prior discussion of applicants, failure to share opinions and knowledge, differing levels of preparedness, and persisting voting norms. Social psychological phenomena therefore do impact the process. Assigning panel members different sections of the applications, as well as using a “devil’s advocate” or group moderator are suggested ways to reduce the occurrence of these phenomena and help improve decision making. It is ultimately concluded that understanding social psychological phenomena will help panel members take action to reduce the flaws in the review process, ensuring the best applicant is chosen and the most powerful decision is made.
Introduction
Research is the backbone of science. Ask any scientist, and they will surely explain the importance of testing ideas and hypotheses with well thought out experiments. After all, the greatest breakthroughs in scientific history, from the discovery of DNA’s double helix, to new cancer treatments, all started with ideas that were tested, analyzed, and repeated. But how do scientists fund these experiments? How do they obtain opportunities to further their growth as researchers? They do so by applying for grants and fellowship positions. For example, suppose a scientist is applying for a grant that will help them conduct groundbreaking research. They have pieced together an extremely thorough and informative proposal and have submitted it for consideration. The scientist does not simply receive the funding or fellowship just because they asked for it. They must be chosen through panel peer review, a process in which panel members discuss the pros and cons of numerous applications, voice their opinions, and decide on a recipient (Olbrecht & Bornmann, 2010). These panel members, otherwise known as the “gatekeepers of science,” ultimately decide the fate of grant and fellowship applicants, playing a significant role in the scientific community (Daniel et al., 2007, p.71). While this process appears to be productive on the surface, there are significant risk factors that weaken the outcome of the final decision. Because this type of review is a group decision making process, and decisions are made differently in groups, the risk of social psychological factors negatively impacting the process is relatively high. The panel peer review process is ultimately flawed, and in order to improve the quality of decisions and ensure the best applicant is chosen, panel members need to understand these phenomena and take action to limit their effects.
If you ask a teenager that fell victim to peer pressure why they made an unfortunate decision, they will likely list a reason similar to “Everyone else was doing it! I didn’t want to feel left out!” Parents and teens alike know that the choices we make are influenced by those around us, and research has proven that decisions are made differently in groups than when made individually (Obrecht et al., 2007). However, when engaged in a group discussion like panel peer review, factors such as group polarization, groupthink, common-knowledge effect, and conformity do not normally cross one’s mind. These social psychological phenomena need to be acknowledged, and in order to apply these phenomena to panel peer review, it is necessary to understand each one. To do this, imagine a group of friends deciding where to eat lunch together. Two of the group members, Friend A and Friend B, joined the group late because they were discussing how much they were craving pizza. They entered the group conversation and immediately voiced their intense desire to go to the local pizza parlor. Their extreme craving was clear to the other group members, Friends C and D, and they went along with their opinion. This is group polarization at play. Group polarization is when people discuss shared opinions with each other and increase the extremeness of those opinions (Olbrecht & Bornmann, 2010). Because friends A and B fed off each other’s opinions before joining the group, their desire for pizza increased to an extreme level.
Friend C secretly wanted Chinese food, but because she wanted to please her friends, she went along with the group decision to buy pizza for lunch. This is groupthink, a phenomenon that occurs when group members strive for harmony in opinion rather than considering the best possible decision (Olbrecht & Bornmann, 2010). Because Friend C did not want to upset anyone in her social circle, she did not share her opinion and fell into conformity, which occurs when group members go along with the opinions of others even if they do not agree (Olbrecht & Bornmann, 2010).
Additionally, Friend D knew that a Chinese restaurant nearby was running an incentive program, giving a ten percent discount to college students. She did not inform her friends of this information and instead went along with the consensus to go to the pizza parlor. This is common- knowledge effect: When unshared information, or information only select people know, is not voiced in discussion, causing common knowledge to influence decisions the most (Olbrecht & Bornmann, 2010). In this case, knowledge of the discount became unshared information and the common knowledge of Friends A and B’s desire to eat pizza most heavily influenced the decision. Had Friend D informed the group of the discount, the outcome of their decision may have been entirely different, and the friends could have saved money while still enjoying each other’s company. Eating lunch at the Chinese restaurant may have been a better decision in this case, but because of social psychology and group dynamics, it was not chosen.
The same concepts can be applied to the panel peer review process. In the previous example, the friends unintentionally excluded the factor of price when deciding where to eat lunch. Consequently, the final decision to eat at the pizza parlor may not have benefited the friends financially. Similarly, the best applicant for grants and fellowships may not be chosen if panel members fall prey to these social psychological phenomena and miss considering a unique aspect of an applicant. However, instead of missing out on a discount and losing a few dollars, the negative consequences of this in panel peer review may result in an applicant missing out on an incredible opportunity to further their research and education, potentially depriving the scientific community of significant discoveries. Therefore, the risk factors for the occurrence each phenomenon need to be understood so panels can decrease the risk of these phenomena occurring and negatively impacting the final decision. For example, the risk of group polarization increases when panel members meet beforehand and discuss their opinions of applicants (Olbrecht & Bornmann, 2010). Just as the friends in the previous scenario discussed their desire for pizza with each other and ended up with extreme cravings, panel members may discuss applicants, realize they share similar opinions, and come into the group discussion with an extreme desire for a specific applicant to be chosen. If this occurs, the extreme opinions may influence the voting behavior of other panel members, especially if the strong opinions are paired with strong public speaking and persuasion skills (Obrecht et al., 2007).
Similarly, the risk of groupthink occurring increases considerably when members have previously worked together, when they are all experts in the same scientific specialty, and when decisions must be made under a time crunch (Olbrecht & Bornmann, 2010). Similar to friends that do not want to upset each other over lunch plans, the more similar and friendly group members are with each other, the more likely they want to get along and come to a unanimous decision. This decreases their consideration of alternative candidates and may lead to a chosen recipient that is not the best choice for the grant or fellowship (Olbrecht & Bornmann, 2010).
Common-knowledge effect also increases in panel peer review when the information being discussed by the group is already known by the majority (Olbrecht & Bornmann, 2010). This can occur for numerous reasons. Group members may fear that others will not care or agree with their unshared information (Olbrecht & Bornmann, 2010). If “information is found credible by several persons,” as in the case of common knowledge, then there is also an unconscious notion that that information is more reliable and safer to voice (Olbrecht & Bornmann, 2010, p. 300). However, this leads to fewer perspectives being heard and decreases the probability that the best decision will be made in the long run, like how the friends were unable to save money because the knowledge of the discount was not shared with the group. For example, if a panel member notices that an applicant’s research relates to a new and promising field but does not speak up, that knowledge becomes unshared information only they know about. Science is constantly changing as new breakthroughs and fields emerge, and it is possible that certain members are more up to speed on the newest discoveries than others. If the panel member does not voice the unshared information, other panel members may not realize the promising potential of a research proposal, decreasing the chances of the best applicant being chosen.
Lastly, the risk of conformity occurring in the review process increases when decisions are difficult and must be made soon, as well as when some members have completed reading all of the applications, while others have not (Olbrecht & Bornmann, 2010). This is similar to the risk factors for groupthink, since the more stressed members are about making a difficult decision, the more likely they are to go along with the opinions of others to get the decision over with (Olbrecht & Bornmann, 2010). Additionally, if some members read the entirety of each application, while others do not, then the members with less information will more likely be swayed by the decisions of the more informed (Obrecht et al., 2007). This suggests group members may conform to other’s opinions, so they do not seem unprepared. However, if these members had read each application thoroughly, they may have formed different opinions and voted for a different applicant, changing the outcome of the decision.
Conformity’s impact on the review process can also be seen by Janis (1982)’s observation that “members tend to evolve informal norms to preserve friendly intragroup relations and these become part of the hidden agenda at their meetings” (p.7). This finding is consistent with Obrecht et al. (2007), which states that “new committee members quickly acquire the culture of their committee as they begin attending meetings and… they become increasingly embedded in that culture over time” (p.85). This was discovered by a Standing Committee of the Canadian Institutes of Health Research (CIHR), which conducted a study to analyze new reviewers. The reviewers scored applicants using specific guidelines and no significant variation was found in how the applications were scored (Obrecht et al., 2007). These reviewers were then placed in different panels in order to experience committee meetings and were asked to score more applications at home. It was found that
…reviewers with no prior experience on a fellowship committee exhibited no inter-committee differences in their at-home scores. For the 435 at-home scores by reviewers who had experienced one committee meeting, there was some inter-committee variation in their at-home scores…For the 677 scores by reviewers who had attended two or more meetings of their committees, there was statistically significant inter-committee variation in their at-home scores. (Obrecht et al., 2007, p.85)
This data shows that there is variability between different committees’ voting habits. Before immersing themselves in a committee, the reviewers’ at home scores were very similar. After the immersion, their voting habits differed from each other, but matched the committee they were a part of. This suggests conformity within individual committees, since new members tend to change their voting habits to fit with the culture of their committee (Obrecht et al., 2007). Thus, conformity appears to increase when panels develop specific norms of thinking. Similar to a group of friends that consistently agree to order pizza because that is what they always do, a panel may continually vote in certain ways for recipients because that is the culture of their group.
The finding that conformity and voting norms are connected suggests that conformity impacts the panel peer review process by contributing to the continuance of deep-rooted biases. Bias related to gender and scientific field has been proven in the process and is described by Bornmann and Daniel (2005) as well as Daniel et al. (2007). Bornmann and Daniel (2005) found that chemistry applicants had about half the chance of receiving a doctoral fellowship position than biology applicants from Boehringer Ingelheim Fonds (B.I.F.), a biomedical foundation in Germany. Females were also found to receive fifty percent fewer fellowships than men (Bornmann & Daniel, 2005). Similarly, a simulation conducted by Bornmann and Daniel (2004) found that if the “applicant is not male, but female, the predicted probability of [being chosen to conduct research at a German university] decreased from 50% to 33%” and “if the applicant is not a biologist, but a chemist, the probability of approval declined from 50% to 25%” (p.11). These biases likely turn into voting norms and contribute to specific traditions and cultures of review groups. Based on Janis (1982)’s and Obrecht et al. (2007)’s observations that group members tend to follow the overall group’s voting norms due to conformity, it can be concluded that conformity contributes heavily to these persisting biases. It appears to act in a cyclical fashion. Bias occurs and is reinforced by conformity. Conformity then leads to persistent voting norms rooted in bias. This shows just how deeply social psychological phenomena impact the panel peer review process. Reviewers need to understand and recognize these phenomena, so biases can be disrupted and the flaws in the system can be combatted.
It is clear that social psychological phenomena such as group polarization, groupthink, common-knowledge effect, and conformity impact the panel peer review process, as shown by the numerous risk factors and negative consequences associated with each. In order for panel members to correct these flaws, ways to combat each phenomenon need to be addressed. Incorporating a “devil’s advocate” from outside of the group, assigning the role of group moderator to someone inside the group, as well as having each member assess different sections of the applications are the best ways to reduce the risks of these phenomena and improve the quality of decisions. Olbrecht and Bornmann (2010) argue that having each group member assess different sections of the applications increases the discussion of different perspectives. This relates to the argument of Obrecht et al. (2007), which claims it is difficult for panel members to read the entirety of each application before the group convenes, causing the less informed members to go along with the opinions of those that read more. If each panel member was assigned a different section, members would be more likely to complete their specific part of the application than read the whole application, and would come into the discussion feeling prepared, lowering their chances of being swayed by someone else’s strong opinion of an applicant.
Not only would assigning reviewers different application sections decrease conformity, but it would also help decrease common-knowledge effect and group polarization. If each person knows something different because they read different parts of the applications, then unshared information would be voiced and foster well informed decisions, weakening common knowledge effect (Olbrecht & Bornmann, 2010). This simple change to the process would force panel members to participate and share what they have to say, almost as if each member is contributing a necessary puzzle piece that comes together to form the overall picture of an applicant. Group polarization would similarly decrease because members would all know different information, making it hard for them to form and discuss similar opinions before the meeting took place (Olbrecht & Bornmann, 2010). Polarized opinions would be difficult to form without all the information, forcing panel members to work together, communicate thoroughly, and think about applicants in different ways in order to reach a final decision.
Using a “devil’s advocate” would also improve the outcome of decisions. Herbert and Estes (1997) state that hiring a devil’s advocate from a consulting firm in the business setting can help “underlying assumptions [be] exposed, new alternative plans [emerge], and creativity [become] enhanced” (p.664). They explain that “an official dissenter can heighten the probability that decisions will be thoroughly researched and [propose] solutions soundly based in reality” (Herbert & Estes, 1997, p. 667). Similar to consultants offering advice to make businesses more efficient, the dissenting ideas of devil’s advocates’ in the context of a panel peer review can encourage new opinions to be voiced on each applicant and urge group members to think about applicants differently. Additionally, Brodbeck et al. (2002) found that sharing dissenting opinions “induces divergent thinking, that in turn, [leads] to increased consideration of unshared information” (p.49). Since voicing unshared information improves the quality of decisions, divergent thinking that is fostered by the devil’s advocate can help decrease common knowledge effect, heightening the consideration of alternative choices when it comes to picking a grant or fellowship recipient. Using a devil’s advocate also applies to group polarization, since the greater variety of opinions there are from divergent thinking, the less likely group members would discuss similar opinions, increase the extremeness of those opinions, and cause biased or unfair decisions to occur during the review process.
It could be argued that hiring a devil’s advocate trained in social psychology to sit in on panel discussions and give feedback would be too expensive for panel groups. However, if the group cannot afford to hire a trained devil’s advocate, someone in the group can be assigned the role of “group moderator.” By acting as a group moderator, the assigned member should encourage participation and critical thinking by asking members why they are voting the way they are and why they believe one applicant is better than the other, essentially performing the role of a devil’s advocate for their own group (Olbrecht & Bornmann, 2010). This may increase the amount of time it takes to review applicants. While a longer process may appear cumbersome, lack of discussion time is a risk factor for both groupthink and conformity, so it is important to allot adequate time for this to occur. It is also the devil’s advocate or group moderator’s job to “present counterarguments and offer constructive criticism [to create] an atmosphere in which others are also willing to put forward their objections” (Olbrecht & Bornmann, 2010, p. 301). In this case, assigning the role of group moderator to a group member would actually benefit the review panel, since group members would be more knowledgeable of scientific terms and concepts and could more thoroughly provide constructive criticism. This is all the more reason for panel groups to be informed of the social psychological phenomena and understand their risks. By being aware of the consequences and understanding how to combat them, review groups can take action into their own hands to stop the phenomena from impacting their decisions.
It could also be argued that a simple way to combat group dynamics is to eliminate the group altogether. The research of Obrecht et al. (2007) suggests that having individual reviewers analyze applications in what is known as “at home review,” is a more effective decision-making strategy for awarding grants and fellowships. This solution appears to make sense, as individuals would not be susceptible to social psychological phenomena. However, the benefits of group decisions outweigh the benefits of individual decisions if group members act to prevent the negative consequences of group dynamics. Divergent thinking is fueled by the sharing of different viewpoints. A single viewpoint does not measure up. For example, Langfelt (2001) examined a situation where the chair of a panel group and a panel member disagreed about an applicant. The chair did not believe the applicant should receive a grant, but the panel member did. Because the panel member voiced their dissenting opinion, other panel members spoke up and voiced their support for the applicant, ultimately awarding the applicant a grant. This would not have occurred had the chair of the group been the only reviewer. This group defeated social psychological phenomena because of its multiple viewpoints, refusing to fall into groupthink and conformity over the superior’s opinion. When panel members know how to take action to combat group dynamics, the group’s decision ultimately benefits and is the most effective.
Additionally, one may think the panel peer review process does not need to be changed at all because it’s predictive validity in choosing the next greatest scientists is normally high. Bornmann and Daniel (2005) did find that “the selection procedure is…highly valid,” relating predictive validity to the amount of times a fellowship recipient had their work cited (p. 297). The more a work is cited, the higher its credibility and prestige, thus showing the greatness of the scientist (Bornmann & Daniel, 2005). However, the article also states that the opportunities associated with the fellowship may contribute the most to the scientists’ success, not the fact that they were the best to begin with (Bornmann & Daniel, 2005). They state,
fellowships…give the fellows such an advantage in training, opportunities, prestige, self-confidence, and so on that they later become superior scientists because of the fellowship, not because they were particularly promising at the point of application. Rather than picking the best scientists, the selection committee…[creates] them. (Bornmann & Daniel, 2005, p. 309)
For example, a fellowship at a prestigious institute would place a fellow in the company of extremely well-known scientists. They would have the opportunity to publish research and get their name affiliated with one of these scientists, potentially being one of the reasons their research is heavily cited in the first place. This fellow could theoretically be any of the applicants, but whoever the committee chooses is handed this career advancing opportunity. Panel members need to be aware that every applicant has the potential to be successful and they need to be cautious not to hop on the bandwagon of the majority’s decision because they may miss an application that is a hidden gem. Making changes to the review process is important because it will help reduce the impact of social psychological phenomena and give every applicant their best chance of becoming the next greatest scientist.
It is extremely important that scientists’ applications are given the fair consideration they deserve. If a scientist takes the time to prepare a grant or fellowship application, it should be judged fairly. If this does not occur due to social psychological phenomena, it may hinder these scientists from applying for other opportunities. Similarly, because scientists stress the importance of validity and accuracy in their data and research, it is assumed they want validity and accuracy in the decisions made in a panel peer review, especially since these decisions determine who will propel the scientific field forward. If social psychological phenomena occur, decisions may negatively be affected, and the best applicant may not be chosen. This may mean the difference between a groundbreaking research experiment being conducted or being tossed aside and forgotten.
Group discussion should ultimately foster different perspectives and help decisions be made more effectively than by one single person (Olbrecht & Bornmann, 2010). However, as shown by the flaws in the panel peer review process, group decisions have serious drawbacks. These drawbacks do not solely apply to peer review, since social psychological phenomena can negatively impact decisions in any group. Friends deciding where to eat lunch, company employees deliberating over budget cuts, juries deciding the verdict of a defendant, and even government officials considering taking military action can all be affected. Understanding that groups make decisions differently is essential to combatting these phenomena in any group decision making process. Just as research is the backbone of science, having all the facts is the backbone of making decisions. Not only do panel peer review members need to understand all the facts about their applicants, but they also need to understand all the facts about how group dynamics influence their consideration of those applicants. By being aware of social psychological phenomena and their risk factors, as well as making changes to the process such as assigning each member different sections of the applications and incorporating a devil’s advocate or group moderator, panel members can correct flaws and significantly improve their decision making. Powerful and well-made decisions will ultimately help powerful and well thought out experiments be conducted and enable scientists to develop powerful potential.
References
Bornmann, L., & Daniel, H. (2004). Reliability, fairness, and predictive validity of committee peer review: Evaluation of the selection of post-graduate fellowship holders by the Boehringer Ingelheim Fonds. B.I.F. Futura, 19, 7-19.
Bornmann, L., & Daniel, H. (2005). Selection of research fellowship recipients by committee peer review. Reliability, fairness and predictive validity of board of trustees' decisions. Scientometrics, 63(2), 297-320. doi:10.1007/s11192-005-0214-2
Brodbeck, F. C., Kerschreiter, R., Mojzisch, A., Frey, D., & Schulz-Hardt, S. (2002). The dissemination of critical, unshared information in decision-making groups: The effects of pre-discussion dissent. European Journal of Social Psychology, 32(1), 35-56. doi:10.1002/ejsp.74
Daniel, H., Mittag, S., & Bornmann, L. (2007). The potential and problems of peer evaluation in higher education and research. In Quality Assessment for Higher Education in Europe, ed. A Cavalli, pp. 71-82. London: UK Portland Press Ltd.
Herbert, T. T., & Estes, R. W. (1977). Improving executive decisions by formalizing dissent: The corporate devil's advocate. The Academy of Management Review, 2(4), 662-667. doi:10.5465/AMR.1977.4406749
Janis, I. L. (1982). Groupthink (2nd ed.). Boston: Houghton Mifflin Company.
Langfeldt, L. (2001). The decision-making constraints and processes of grant peer review, and their effects on the review outcome. Social Studies of Science, 31(6), 820-841. doi:10.1177/030631201031006002
Obrecht, M., Tibelius, K., & D'Aloisio, G. (2007). Examining the value added by committee discussion in the review of applications for research awards. Research Evaluation, 16(2), 79-91. doi:10.3152/095820207X223785
Olbrecht, M., & Bornmann, L. (2010). Panel peer review of grant applications: What do we know from research in social psychology on judgment and decision-making in groups? Research Evaluation, 19(4), 293-304. doi:10.3152/095820210X12809191250762
コメント