Judging Research Paper Competitions Fairly
Suggestions from
The AEJMC Standing Committee On Research
I. INTRODUCTION
The AEJMC Standing Committee on Research often gets requests from divisions for recommendations regarding research paper judging. The Committee makes several assumptions about the desired outcomes of research paper presentations at conventions. These include:
* Research papers should represent good scholarship; "bad" papers should not be accepted
* Research paper programs should be a forum for new work, new ideas, and new approaches. Divisions should welcome controversial ideas and approaches.
* Young scholars and students should be encouraged to present their original work.
* Research paper judging and programs should reflect the diversity of the Association.
The Standing Committee on Research believes that division leadership and research paper competition chairs should run paper competitions that not only are fair, but are perceived to be fair. To that end the Committee has several suggestions that we believe will increase perceptions of the fairness of the process. AEJMC divisions run their own competitions in accordance with the Council of Divisions' "Uniform Paper Call" guidelines. Divisions are free to accept or reject any of the suggestions offered here, but division research chairs should recognize that these suggestions are the result of many collective years of experience in paper judging across AEJMC divisions and are endorsed by the Standing Committee on Research.
II. FAIRNESS IN THE CALL FOR PAPERS
The Council of Divisions' "Uniform Paper Call" (UPC) provides several points about research paper competition procedures and deadlines. The following points are additional suggestions from the Standing Committee on Research:
1. Multiple submissions. The call guidelines require that papers (a) are not already under review for other national conventions; (b) can only be submitted to one AEJMC group for evaluation, and (c) should not have been presented to other national conventions or published in scholarly or trade journals prior to presentation at the AEJMC convention. In addition, some divisions limit the number of submissions to their division by an author and some divisions allow submission of papers presented at smaller regional conferences. Any limits of these types should be made clear to presenters.
2. Student papers. The uniform call suggests that student papers should compete on an equal footing in an open paper competition. Thus, several divisions do not hold separate student competitions but instead draw their top student papers from the open paper competition. Divisions taking this approach should remind student submittors to add "Graduate Student" to their paper's title page unless the paper is co-authored with a faculty member.
3. Presentation requirement. At least one author of a faculty paper must attend the convention and present the paper. Authors should be reminded of this requirement in their acceptance letter and be asked to immediately notify the research chair if no faculty author can present so that other quality papers can be programmed.
4. Recognition. The Standing Committee on Research recommends that divisions develop some means of recognizing top papers for both faculty and students. Often this recognition represents the top three faculty and student papers from the open competition. If awards are given only through a special competition research chairs should clearly state if papers are eligible for presentation in the open competition if they do not win an award.
III. SELECTING PAPER COMPETITION JUDGES FAIRLY
Selecting paper competition judges is one of the most important tasks of a paper chair. As with the paper call, divisions vary a great deal in how they select judges, but in the interest of fairness and perceptions of fairness, the following guidelines should be considered by the division and the competition chair.
1. Ideally, paper judges should have expertise in the method/theory of the papers they are judging. Paper competition chairs should determine judges' areas of expertise and match judges with papers as much as possible. Keep files for subsequent paper competition chairs so that those judges who are particularly effective can be used again.
2. Have both senior and junior faculty as judges. Generally don't use graduate students as judges, unless the graduate student has a unique area of expertise necessary to judge a research paper.
3. Put a reasonable limit on the number of papers each judge must read. Generally this should be about 3 to 5 papers per judge.
4. Judges should represent different perspectives in the division, including different universities. Avoid using judges primarily from the competition chair's home school. Make sure the process includes judges from diverse backgrounds.
5. Avoid real (and appearances of) conflicts of interest when selecting judges. Judges should be people who have not submitted a paper to the division competition, if at all possible. Judges should be asked to report any conflict of interest such as papers from those with whom they have recently co-authored, mentioned, or worked. Judges should not be assigned to review papers from others in their own institution. A paper competition chair should not submit a paper to his/her own division. If the paper competition chair has a conflict of interest (real or one that others might perceive) for the top paper(s) award competition, other members of the division would make the top paper(s) selection.
IV. RUNNING A FAIR COMPETITION
1. Remove any references to the author(s) from the paper before it is sent to the judges. This should include obvious citations of the author's previous work and affiliation (for example, articles in press or papers not yet presented). Use identification numbers on the evaluation form to maintain confidentiality.
2. Each paper should be reviewed by at least 2 judges, and preferably 3. If there is a great deal of variance in the evaluations, another judge should be sought. If a judge is not performing responsibly or does not understand a manuscript, then the judge should be replaced.
3. Maintain anonymity of the judges. Judges should be referred to by number on the report sent to the author.
4. Require reviewers to provide written comments on papers that they judge to be inadequate for presentation. Evaluation forms should have room for comments, or ask for comments on the back of the form or on a separate sheet.
5. A paper competition chair serves as the final arbiter. It's particularly important not to reject controversial work automatically. The chair is in a unique position to observe judges' lack of consensus on controversial scholarship. Thus, the chair should read all papers where there is variance in the judges' evaluations, and use his/ her own judgment as to whether to accept or reject the papers.
6. If numerical scoring is used, be sure it is fair. Most divisions use some form of numerical scoring for paper judges. This is not a problem as long as research paper chairs recognize that judges naturally will use different scoring "systems" for good and bad papers and that the quality of the particular set of papers received by a judge may bias the numerical evaluations. Thus, you cannot simply sum the numerical scores and use the summed numbers to make judgments of quality. The best solution is to use standardized scoring. This is not hard to do, and it helps to compensate for a judge's idiosyncratic scoring tendencies. (Instructions for standardizing scores are at the end of this document.) if standard scores are not used, then judge's evaluations should be simple rank orders 1,2,3, etc.) rather than ratings. If you use rankings, then each judge must evaluate the same number of papers, and papers should be randomly assigned to judges. The rankings then can be summed over all three judges (and a mean ranking created, if desired).
7. If the division has a theme, selection should not be biased by the theme. All paper selections should adhere to the same standard of quality.
V. DEVISING A FAIR PAPER JUDGING EVALUATION FORM
The Standing Committee on Research provides a sample evaluation form (contact AEJMC for a copy), but the actual forms used by divisions vary considerably. Criteria used by divisions include: clear purpose, originality of concept/topic, adequate literature review, grasps literature, complete literature review, method clearly described, appropriate method, effective method, good evidence, use of original/ primary sources, evidence related to purpose, evidence presented clearly, conclusions related to purpose, conclusions clear, conclusions supported, well organized, carried out well, good quality writing, questions handled creatively, shows theoretical strength, relevant to division, of significant interest to division, significant direction for research, significant contribution, important topic, quality and depth or research, conceptually rigorous, overall quality, worth presenting, and following the competition's rules.
1. A group leadership should agree on the criteria to be used to judge papers. When was the last time your group reviewed the criteria you are using? Do they need to be revised? Make sure the criteria used are appropriate to the division/interest group and the method(s) used.
2. Judging forms should protect the identity of the author and the judge. Use numbers to identify papers and judges.
3. The paper judging form should be consistent with the type of scoring to be used. If the division/interest group is not going to use standard scores, then the form should allow for rankings rather than ratings. It can be demoralizing to both judges and authors if the division gathers ratings data and then the paper competition chair ignores the data and uses only the overall ranking to choose papers for presentation.
4. The judging form should have adequate room for comments. The form should ask the judge to comment on the paper, particularly when the judge recommends that the paper not be accepted for presentation. These comments sometimes appear either on the back of the form or on a separate sheet.
VI. JUDGING STUDENT PAPER COMPETITIONS FAIRLY
1. Groups should decide whether to ignore the student/faculty status of authors for purposes of accepting papers. Some keep all the papers together and only make distinctions for awards, while others judge student papers as a separate competition.
2. Competition for student papers should be rigorous. In some groups where student papers are judged without reference to status, student papers fare as well or better than papers from regular faculty.
3. Student members deserve a fair opportunity to present scholarly work. Groups should not unfairly restrict the number of student papers accepted. Student papers should not be relegated to an "all-student" session but should be integrated into the division's sessions. Although AEJMC provides complimentary convention registration to three student authors in each group, a group need not limit student paper acceptances to three.
4. If a group limits the number of student authors on a paper, then the same rules should apply to the number of faculty authors. Some groups limit the number of student authors to three; this is unfair unless the same limits apply to the number of faculty authors.
VII. COMMUNICATING WITH AUTHORS
1. Authors should receive timely notification of acceptance or rejection of their papers. The uniform call specifies the date by which authors should be notified. Many schools provide little or nothing in travel funding, so faculty and students have to do considerable budget planning to attend a conference.
2. Acceptance notices should be specific about the nature of the session. Authors should be told how long they have to present their work, how to order audio or video equipment, when to send their paper to the discussant, and who the other presenters will be in the session. An example handout of information for presenters, moderators and discussants is available from AEJMC.
3. All feedback should be sent to the authors. Some chairs edit nasty comments before sending back the evaluation form, but this is up to the chair to monitor.
4. The author acceptance letter should include a summary of papers submitted and accepted in the division's competition. This information helps the author see where his/her paper stands with the others submitted.
VIII. THE PAPER SESSION
1. Paper sessions should allow enough time for presenters and discussants. Generally three or four papers per session is the limit if there is a discussant. A sample handout of information for presenters, moderators and discussants is available from AEJMC. Groups should tailor such a handout to their own needs.
2. Discussants should be eminently familiar with the work they are discussing. Choose discussants who are known to be effective oral communicators.
3. Diversity should be reflected by the choice of moderators and discussants. Also, make sure that the choice of discussants is not biased toward one perspective or one school.
4. Paper chairs should make sure that promised audio/video equipment is available.
5. Moderators should treat all presenters fairly, including equivalent time limits. It is not fair for those who present at the end of the session to have to give up part of their time because early presenters went over the time limit. Again, information about time limits should be provided to presenters, moderators and discussants in advance.
IX. THE POSTER SESSION
1. Groups may employ different criteria in assigning papers to a poster session (e.g. amenability to display, author request). However, papers presented in a poster session should conform to the same standards as papers accepted for other sessions (e.g. clearly stated goals, important to field, relevant topic, well-researched and written, good methodology, etc.). Perceived or evaluated quality of a paper should NOT be a criterion, nor should poster sessions become dumping grounds for the lowest ranked papers of those accepted.
2. Wherever possible, poster boards should be arranged to diminish noise and facilitate interaction. Sponsors of poster sessions also are encouraged to arrange them thematically, and groups are encouraged to co-sponsor poster sessions.
3. Authors of poster sessions are responsible for being present at the session to discuss their papers. For effective poster display, authors of poster papers should consult Welch, A., & Waehler, C. (1996). Preferences about APA poster presentations. Teaching of Psychology, 23(1), 42-44.
4. To encourage creativity, groups should, wherever feasible, create faculty and student awards for best visual display. This award should supplement, not replace, other awards based on the quality of the paper's content.
5. Groups may vary the format of poster sessions and are encouraged to be creative in the use of formats and displays (e.g. poster sessions could begin with 2-4 minute presentations by individual authors, or with round-table discussion, and they might end with discussion and critiques, either informally or formally). Groups should obtain feedback from poster presenters and attendees to improve future poster sessions.
Guidelines revised April 1999
Return to AEJMC Officer Resources