Interpreting results

The purpose of the report summary is to provide you with a useful and interpretable overview of your results, in a clear and simple format. The following guides are designed to help you interpret your results.

• Report layout

• Using the tables

• Interpretational concerns

Report layout

The top section of each page (shown in Figure 1) presents basic information to identify the class and Unit to which the questionnaire concerned was administered. You will find listed

• name of the 'Unit Co-ordinator' 

• the department 

• the Unit 

• the degree program

• the semester of the questionnaire's administration 

• the number of students enrolled in the Unit 

• the number of questionnaires that were completed 

Figure 2:  Top section of the report

The number enrolled and the number of questionnaires completed are critical pieces of information. If the number completed is less than about 60% of the number enrolled then your responses might not be representative of the perceptions of those students who have experienced the Unit. Of course it could be that those students not completing the questionnaire never come to class anyhow. But the point is that you need to consider representativeness.  These two numbers give you a good initial guide.

The panels or boxes (shown in Figure 2) present the response analyses for single questions from the questionnaire. The heading for each panel is simply the question to which that panel's analysis refers. The order of the panels parallels that of the questions on the questionnaire. Each panel shows

• the number of students who gave each of the five possible responses 

• the percentage who gave each possible response (calculated over the number answering the question, which might be less than the number completing the questionnaire) 

• a graphical representation of those percentages 

• a median response 

• a semi-interquartile range (SIR)

•  

Figure 2:  Panels or boxes in the response analysis of each item in the questionnaire

The median is an indication of the "central point" in a distribution of responses. Theoretically, it is that scale value which divides a top 50% of the responses from a bottom 50%. For each question, it is calculated by assigning to each of the five "answer" responses the point values shown to the left of each response label in the panels (the "cannot answer" responses are excluded). What it all means is that the higher the median for any particular question, the more the bulk of the students will have responded towards the ' A' end of the scale.

The semi-interquartile range (SIR) is an indication of the spread of responses in a distribution. It is also calculated using the same assigned point values for the response categories. For the most part SIRs will be between about 0.4 and 0.8. The smaller the SIR for any one question the more the bulk of the students will have clustered around one or two responses. That is, a small SIR value means that the students are in agreement on the question.

Using the tables

In interpreting the responses for any given question, it is very important to look at the TOTAL pattern of student responses. Attending to just one aspect can lead you to misinterpret what is being indicated. So, assuming that you have enough students responding to the questionnaire, that those students are a reasonable proportion of the number enrolled, and that they have sufficient experience of the Unit, what do you do with the response analyses? How do you interpret each panel?

 Check that most of the students responding to the questionnaire also responded to the question. If the total figure at the bottom of the Count column is less than the No. Questionnaires Completed, then the difference is the number of students who simply left that question blank. The more that substantial numbers left a question blank, the less you can interpret the remaining responses as representing your students' collective judgment. 

Look for the response positions that attracted the biggest percentages of student responses. Since all of the questions were positively phrased, you would look to see whether the bulk of the students responded towards the ' A' end of the scale. The larger the percentage of students who observed that the Unit consistently exhibited the characteristic, the more positive is the interpretation that you can make. If large percentages responded towards the 'E' end of the scale, then the question might indicate an aspect of the Unit that is worth reviewing.

 Look at the spread of responses shown in the graph. If the clear majority of the students opted for one or two of the five response positions then you can conclude that there is a good degree of agreement amongst the students on the question. If the responses are spread, then there is disagreement amongst the students. The narrower the spread of responses, the clearer is the interpretation that you can take from those responses.

 Finally, you could look at the median and the semi-interquartile range (SIR). You would be looking for high value medians and low value SIRs. This is not really anything more than you will already have learned from doing "2" summary values. However, there is an important caution. Sometimes you might find two "humps" in the responses to a question. That is, there might be two response positions that account for the bulk of the students, but these two are distant from one another on the scale. The graph would have two peaks. This often means that the students comprise distinguishable sub-groups. The caution is that looking at the median and SIR alone can mask this. So always look at the percentages and the graph to get a proper overview of what the median and SIR are summarizing

In summary, what is the ideal for which you should be looking?

• Most of the enrolled students completing the questionnaire 

• Very few students leaving a question blank 

• The clear majority of responses sited at the one or two positions at the 'A' end

• Small percentages of responses sited outside those one or two positions

Interpretational concerns

If having a high percentage of responses at the 'A' end is good, is having a low percentage necessarily bad?

There are two points to be made here. First, not all questions are of equivalent priority of importance. It is probably advisable to examine the nature of the question and consider its relative importance.

Second, some questions reflect characteristics that while certainly part of a well-organized Unit might yet be quite uncommon in general.  So, while getting high positive percentages on those questions would be a mark of a 'good' Unit, getting low percentages might simply mean that it is on a par with the majority of other Units.

What should one make of small numbers of negative responses when the clear bulk is positive?

Behavioural measures will quite normally contain a degree of variability. In the final analysis the questionnaires are sampling students' recollections and judgments. There is no guarantee that those recollections and judgments will be entirely error free. In interpreting questionnaire results most attention should be paid to the responses of the bulk. Any responses that are “deviant outliers” compared to that bulk should sensibly be ignored.

Even when the bulk of responses are clearly at the positive 'A' end of the scale, how good is good?

There are two answers here. First, in an absolute sense if all of the responses are at the top end of the scale, then it is a clear indication that something is right. But second, in a relative sense, you might still be interested to know how the responses compare to those from other Units. How common is it for other responses to be the same this Unit, or better or less?

When the responses are toward the 'E' end of the scale, how does one know what could be done, specifically?

First, should reflect on what actually happened in the Unit, in relation to the question, and generate some possibilities to consider. Second, you might ask the students for assistance, directly. Share the results of the questionnaire with the students who responded to it in the first place. Ask them what they think needs to be done in relation to any "negative response" questions. It might take a bit of courage to do this, but you will almost certainly score points with the students for doing it. You might just get some really useful ideas that could improve the Unit. Third, you could contact the Higher Educational Development Unit (HEDU) for assistance.