General Applications

Consensus Analysis is a statistical technique that determines if the answers given to a set of questions, by a group of people, indicate that the group is in general agreement.  Several examples will show this.

• Twenty students have each written a one-paragraph essay.  A panel of 12 judges reads each of the essays and indicates whether the essay passes or fails the writing test.  The judges have all been told the test criteria for passing.  The Consensus Analysis is used to look at the exam evaluations and determines if the judges are applying the criteria uniformly.
• A class of 5 year old children are presented with 25 different prepared foods.  Each child is asked whether he or she likes each food.  The Consensus Analysis evaluates whether there is agreement among this group of children as to what constitutes food preferences.

In general, the size of the sampled population (e.g., judges and children) ranges from a dozen to two dozen.  There should, in general, be more evaluation items  (e.g., essays and prepared foods) than the sampled population.

The evaluation items must be of a similar type and it is assumed that they are all of equal importance.  In the essay-evaluation example, this is clear since all of the evaluation items were same-sized paragraphs (presumably on the same topic).  In the food-preference example, it is not as clear that the evaluation items are sufficiently similar to meet this assumption.

It is possible to do follow-up analyses, particularly when there appears to be no consensus.  For example, what are the items for which there is agreement versus those where there is a mixture of opinion.

Statistical Analysis Software

Software to run consensus analyses is free and available on the Web. The software program is called "PAST."  It is available at this link: http://folk.uio.no/ohammer/past/download.html

This software runs on PCs. It doesn't require installation, so you can keep it on a USB drive and simply plug it into any PC to run it.  Follow the download instructions on the PAST website.

Running Consensus Analyses with PAST

Data Configuration and Data Entry

A simple use of PAST is shown here with an example of 12 judges reading essays for 20 students.  The objective is to see if all the judges are consistent in their evaluation of the essays.  This agreement does not need to be perfect.

The samples are the columns (e.g., judges) while the evaluated items are the rows (e.g., student essays). In each cell there is a 1 for pass or a zero for fail.

• Each column and row can (and probably should) have a short descriptive header. The labels must not have any blanks (e.g., "judge1" or "j01", not "judge 1" or "j 01").
• It is often convenient to enter your data into Excel.
  • Open Excel and enter your data. Save the worksheet holding your data.
  • Open PAST and make sure there is a check in the Edit labels box (if you don't do this, your labels will become part of the data).
  • Return to Excel and highlight the entire data area. Click on copy.
  • Return to PAST and click on paste.
  • Uncheck the two edit boxes. Your data are now ready for analysis. (Screen Shot)

Running and Interpreting a Consensus Analysis

• Inside PAST, highlight the area with your data.
• Choose the Multivar menu and click on Principal Components. (Screen Shot)
• A popup window will appear.  Click on the Correlation radio-button under Matrix.
• Look at the column of values labeled "Eigenvalue."  Use the first value (next to the label PC 1) and divide it by the second value (next to the label PC 2). (Screen Shot)
• If the value resulting from the division is 3 or greater, you can conclude that there is a consensus among the respondents in their answers to the questions.
• You should then click on the View loading button.
• When the "PC loadings" popup window appears, click on the Correlations radio-button under Values.
• Check if all of the Correlation bars are positive.  If so, this also shows that there is consensus in the group. (Screen Shot)

If you did not get a consensus (i.e., the division of PC 1 by PC 2 was less than 3), you may have one or two people who don't agree with the other people.  You should be able to see who they are by looking at the loadings.

You should return to your original data and examine the values with a critical eye.  For example, you can calculate what percentage of the judges voted "yes" for each exam. This can be added as a new column and the rows can then be sorted by this percentage value.  Both high and low percentages show broad agreement among the judges (most saying either yes or no).  It is the intermediate percentages (highlighted in yellow in this Screen Shot [Trouble with this link]) that should be scrutinized to find out why there was a difference of opinion.

Consensus Analysis is a tool that provides a starting point.  Make sure that you continue your analyses by using the power of spreadsheet analyses and by asking critical questions.