Comparison of Jury Ranking System Professional Guidelines

Comparison of Jury Ranking Systems

The Professional Guidelines recommends using a numerical ranking system of 1-7. This is the best system both from a mathematical point of view and for practicality. In fact, some commonly used jury ranking systems actually skew the results unfavorably and produce unreliable results.

If you don't have an in depth understanding of mathematics and don't want to read this document use every number between 1-7 for jury ranking. Yes, a juror should use every number including 1 and 7.

Jurors tend to use the middle range of numbers when scoring work because each piece viewed is rarely the best or worst that the juror has ever seen. Reality is that most work submitted to a jury is "average" (i.e. 2,3,4,5,6.)

Using every number between 1-7 will produce the most reliable results for jury ranking. The additional benefit is that there will be more subtle gradations rating the work, thus you can add or delete work depending on the amount of space or entries permitted.No matter what ranking system is used, the jurors should be encouraged to use the full range of numbers to rank the images.

PDF Comparison of Jury Ranking Systems

To understand the math behind several different jury ranking systems that are commonly used.

Below is an excerpt:

Juries should avoid using only the 1 – 4 range (i.e. 1, 2, 3, 4) because fewer choices usually result in too many ties.
On the other hand, a ten number range (i.e. 1 – 10) offers too many choices, which tends to slow down decision-making.
Sometimes it is suggested to use a system that removes the middle number from a 1 – 5 system (e.g. eliminating the “3”) and to use only 1, 2, 4, 5 to “force” a selection outside of average. However, using fewer score choices increases the number of possible ties (regardless of the number of jurors). Mathematically, there is no difference between 1, 2, 3, 4 and 1, 2, 4, 5 since the number of sums (outcomes) is identical.
There are two advantages of using the 1 - 7 system over the 1 – 5 system: With just three jurors and scoring choices from 1 – 7, there are nineteen possible sums. The 1 – 5 system produces thirteen possible sums. With nineteen possible sums, it is easier to find the dividing point between accepted work (the top sums) and rejected work (the lower sums).
Secondly, difficult negotiations among jurors can be kept to a minimum when designating awards because with fewer ties, it is more evident which pieces actually received the highest scores.

Read the entire document to learn more about PDF Comparison of Jury Ranking Systems.

I. Description of Numerical Ranking Systems

With a numerical system, each juror is asked to mark down a score for each slide shown. The jurors should agree ahead of time on what score choices will be used (such as 1, 2, 3, 4, 5, 6, 7,) with “1” as the lowest (worst) score and the highest number as the best score. This process would usually be initiated during the second or third viewing.

Jurors tend to use the middle range of numbers when scoring work because each piece viewed is rarely the best or worst that the juror has ever seen. No matter what ranking system is used, the jurors should be encouraged to use the full range of numbers to rank the images.

Juries seldom use only the 1 – 4 range (i.e. 1, 2, 3, 4) because fewer choices usually result in too many ties. On the other hand, a ten number range (i.e. 1 – 10) offers too many choices, which tends to slow down decision-making.

II. Summary of Alternate Numerical Ranking Systems

A. 1, 2, 3, 4, 5, 6, 7 [RECOMMENDED by the Professional Guidelines]
The Professional Guidelines recommend a ranking system of 1 – 7 (i.e. 1, 2, 3, 4, 5, 6, 7), with “7” being the score for the best work, as the most useful ranking system for the judging process and readily leads to effective discriminating results.

B. 1, 2, 4, 5 Sometimes it is suggested to use a system that removes the middle number from a 1 – 5 system (e.g. eliminating the “3”) and to use only 1, 2, 4, 5 to “force” a selection outside of average. However, using fewer score choices increases the number of possible ties (regardless of the number of jurors). Mathematically, there is no difference between 1, 2, 3, 4 and 1, 2, 4, 5 since the number of sums (outcomes) is identical. (see Section V. The problem with the 1, 2, 4, 5 ranking system for a detailed explanation)

C. 1, 3, 5, 7 Sometimes it is suggested to use only the odd numbers from 1 – 7 “to reduce ties.” This is a fallacy -- regardless of the number of jurors. Using only the odd numbers actually increases the ties because the number of score choices has been reduced to only four scoring choices (1, 3, 5, 7). There is no mathematical difference between 1, 2, 3, 4 or 1, 3, 5, 7 or 1, 2, 4, 5 since the number of sums (outcomes) is identical for any system with four score choices.

III. Merits of the 1, 2, 3, 4, 5, 7, Numerical Ranking System (Recommended by Professional Guidelines)

A. There are two advantages of using the 1-7 system over the1–5 system. With just three jurors and scoring choices from 1 – 7, there are nineteen possible sums. The 1 – 5 system produces thirteen possible sums. With nineteen possible sums, it is easier to find the dividing point between accepted work (the top sums) and rejected work (the lower sums).

B. Secondly, difficult negotiations among jurors can be kept to a minimum when designating awards because with fewer ties, it is more evident which pieces actually receive the highest scores

IV. Limited Choices Cause Fewer Sums

A. Jurors using only the odd numbers 1, 3, 5, 7 for their ranking system mistakenly think that there are more combinations because the numbers are higher. In reality, they have no more combinations than using 1, 2, 3, 4.

B. A ranking system using 1, 3, 5, 7 eliminates nearly half the possible sums or combinations because:

1. If there are an odd number of jurors, the sums will ALWAYS BE AN ODD NUMBER.

2. If there is an even number of jurors, the sum will ALWAYS BE AN EVEN NUMBER.

C. A ranking system using 2, 4, 6, 8, will always result in an even numbered score, whether there is an odd or even number of jurors.

D. A system using 1, 2, 3, 4 will not rank work as finely as a system using 1, 2, 3, 4, 5. A broader range of score choices will always help to separate the results more finely. This is an advantage. A system using 1, 2, 3, 4, 5, 6, 7 provides the broadest practical range for jury selection. Reserving numbers 1 and 7 for the lowest and highest scores means that the five numbers 2, 3, 4, 5, 6 can be used to score the more average work.

V. The Problem with the 1, 2, 4, 5 Ranking System

A. When using the 1, 2, 4, 5 ranking system (leaving out the score choice of “3”) the range appears to be larger because the highest value is “5”. However, as a juror, you still only have four score choices. As a result, the number of possible outcomes is identical to any other system with only four score choices. This is a definite disadvantage.

B. Another problem with 1, 2, 4, 5, is excessive weighting to a dissenting juror. If one juror gives an unusual score using the 1, 2, 4, 5 system, it outweighs the judgment of the other jurors. An example is illustrated below.

1. Example 1) ranking example: 4 + 4 + 4 = 12
In the first example, (4 + 4 + 4 = 12), these uniform scores demonstrate unanimous opinion that the quality of the work is “medium high.”

2. Example 2) ranking example: 5 + 5 + 2 = 12
The second example, (5 + 5 + 2 = 12), demonstrates that if two jurors think the entry is of the highest rank and one juror gives it a “medium low” rank, then the work appears to be ranked equally to the (4 + 4 + 4 = 12) example.

3. Example 3) ranking example: 5 + 5 + 1 = 11
In the third and final example, two jurors judged the entry at the highest rank but one juror ranked it the lowest. As a result, one juror’s vote skewed the score. This gives one juror’s judgment too much weight, since two jurors cannot overrule one juror using the 1, 2, 4, 5 system. It is doubtful that this inequity was the intention of any jury.

VI. The Problem with the 1, 2, 3, 4 Ranking System

A. Using a 1, 2, 3, 4 system is not an ideal system either because it only gives four choices but it is better than the 1, 2, 4, 5 system. Why? Because every juror’s opinion is equally weighted.

B. Using 1, 2, 3, 4, 5 is better than a system with only four ranking choices because it uses finer increments for scoring. The jurors must make an effort to use the entire range of the scale.

C. The 1, 2, 3, 4, 5, 6, 7 system is the recommended ranking system since it uses even finer increments without being unnecessarily burdensome.

VII. The Problem with Even or Odd Number Only Ranking Systems

Using only odd numbers or only even numbers reduces the number of score choices. The consequences are a reduced number of possible sums by nearly half, which increases the likelihood of ties.

This document is prepared by William Earle Shelander, Ace Shelander and Harriete Estel Berman. — This document is prepared by **William Earle Shelander**, **Ace Shelander** and **Harriete Estel Berman**.

************************************************************************************************************************************************************************************

DISCLAIMER

THE COYRIGHT OWNER HAS PREPARED THE FOLLOWING MATERIALS AS AN INFORMATIONAL AID TO EDUCATE THE READER ABOUT COMMON SITUATIONS THAT GENERALLY ARISE IN THE ARTS AND CRAFTS FIELD. THESE MATERIALS, INCLUDING ALL SAMPLE AGREEMENTS, CANNOT AND DO NOT ADDRESS ALL OF THE LEGAL ISSUES THAT MAY BE PERTINENT TO ANY INDIVIDUAL CIRCUMSTANCE. THE READER SHOULD NOT ASSUME THAT THE INFORMATION CONTAINED HEREIN WILL SATISFY ALL OF THEIR NEEDS. LAWS VARY FROM STATE TO STATE, AND THESE MATERIALS ARE NOT A SUBSTITUTE FOR OBTAINING LEGAL ADVICE FROM A LICENSED ATTORNEY IN YOUR STATE. THE READER IS ENCOURAGED TO SEEK SUCH LEGAL ADVICE PRIOR TO USE OF THESE MATERIALS. SNAG AND THE COPYRIGHT OWNER DISCLAIM ANY RESPONSIBILITY FOR ANY AND ALL LOSSES, DAMAGE, OR CAUSES OF ACTION THAT MAY ARISE OR BE CONNECTED WITH THE USE OF THESE MATERIALS AND/OR FORMS.

More information about Jury Ranking Systems below: