MCQs – A contentious Issue!

Filed under Dr Mohammad Zafar Yab, MCQ, Statistics

1. Options corresponding to each question/ statement are randomized.
2. Student selects an option in a questions/statement randomly (i.e. by shear guess work). Essentially it means that the test is administered to completely ignorant persons about the subject that they are tested for (i.e. subject - insane). As an example, imagine a PhD level high energy physics MCQ based questionnaire administered to a matriculation student. What should we expect when he is asked to select one (and only one) option for each statement? – A non-knowledge based random selection!
3. The test is administered to a laaa…rge number of students

• For MCQs with only two options (i.e. True or False) no matter how large is the number of statements over 50% of the subject-illiterate will pass. For effective use of true-false type questions other measures such as negative making should help; but needs a careful analysis to avoid a trap one might get into.
• MCQs with three options do not appear to be practicable when administering quizzes, mid-semester or final examination during the conduct of a course because of time required and size of the class. For example even for a 26 statements questionnaire (Paper) nearly 6% will pass, some of them getting even C+ grade – not a very comfortable situation. Increasing the number of statements in questionnaires to 50 will still leave one percent of the subject-illiterate to get through. Of course, the pass percentage will be enhanced to the extent of knowledge of the students to whom the test is administered.
• Questionnaire consisting of 18 statements each having four options will still allow nearly 2% to pass; whereas, that with five options will let less than 0.5% (5 out of one thousand) to get through.
• An increase in the number of statements and options both allow lower numbers to pass, perhaps, at a prohibitive cost. For example, a 5-options 35-statements questionnaire leave very nearly ZERO % (actually 0.003% - i.e. 3 in 100,000) unwanted qualifiers. Attaining such an accuracy has to be weighed against the time required to conduct an exam within a short time usually allocated for such activity in a class.

• Some instructors believe that the MCQs do not test knowledge. This happens with instructors who lack in know-how about writing MCQs. (Dr. Syed Makhdoom Ali Shah may help).
• Some reckon that the results are not reliable when administered to a small group. Apparently a genuine concern. Let us not forget that the theoretical results are based on assumption 3. For small groups (such as of size 30 or so) the percentage of students obtaining a particular grade will hover around the true value; and will vary from group to group (based on the standard error of the estimate). Thus we can expect this to happen more often in real life scenarios where small samples are involved. Nevertheless, this variation will be insignificant for affecting most decisions. Certainly, increasing the group size must reduce this uncertainty and alleviate any fear.