Multiple Choice Reporting Guide
ADVANTAGES AND LIMITATIONS OF MULTIPLE-CHOICE ITEMS
The Advantages
Here’s some key benefits of multiple-choice questions.
Quick and simple to answer
As long as you’ve used clear language and you’ve provided easy to follow answer
options for your respondents to select from, then multiple choice questions should be
quick and simple for them to complete.
You can potentially ask more questions
Considering how long it would take someone to answer a free text open-ended
question, if your survey included just multiple choice questions, which are a lot quicker
to answer, you could potentially include a lot more questions in your survey.
Mobile friendly
Compared with other questions types, notably ones with text boxes, multiple choice
questions are much easier to complete on smartphones and tablets. And when you
consider that Brits are now spending 2½ hours online on their smartphones daily, it’s
more important than ever to ensure your online survey is simple to complete on a
mobile device.
Simple to analyse
When you’ve only given respondents a finite list of answers to choose from there are no
worries about having to deal with irrelevant or stupid answers that you might face with
an open-ended question.
Subsequently, your data will be much easier to analyze, interpret and draw conclusions
from.
The Disadvantages
Despite the many benefits of multiple-choice questions, they do have some
shortcomings, so it’s prudent to be aware of these before planning your next survey.
They can be time-consuming to create
Although they’re quick to process, given the many types of multiple choice questions
that are available, and the many options and steps you need to go through, multiple
choice questions can take a bit longer than other question types to set up. So, it’s best
to be aware of this before you get started.
The data they produce is solely quantitative
If you’re after data that is quick and simple to gather, and draw insight and conclusions
from, then quantitative data is ideal for you.
However, if you’re looking to get a deeper insight into the reasons and motivations why
people have answered in the way they have, you’re not going to get that with multiple
choice questions. Instead, you would need to include open ended questions that would
encourage respondents to leave this kind of qualitative feedback.
They can limit the respondent in their answers
Depending on what your survey is about, there can be some occasions when there’s
not a suitable answer available from the options you have provided for a respondent to
select. In this case it can be helpful to provide them with an ‘other’ option textbox after
your question.
Reference: https://www.smartsurvey.co.uk/survey-questions/multiple-choice-questions
QUALITIES DESIRED IN MULTIPLE CHOICE ITEMS
Strong assessment items are made up of five elements:
1. Standard
2. Stimulus
3. Stem
4. Key
5. Distractors
Standard
The standard establishes the purpose and topic of the assessment item. The student
should be able to read the standard and know immediately what to expect in terms of
topic and desired action. This is usually at the head of a section, with problem sets
below it. For example, a standard for a geometry section might be “Use the
Pythagorean Theorem to identify the hypotenuse for the following.”
The standard establishes the purpose and topic of the assessment item. The student
should be able to read the standard and know immediately what to expect in terms of
topic and desired action. This is usually at the head of a section, with problem sets
below it. For example, a standard for a geometry section might be “Use the
Pythagorean Theorem to identify the hypotenuse for the following.”
Stimulus
The stimulus provides the necessary background information to understand the
problem. For math, it is the original equation or problem. For word-based problems, it is
the background material that sets up the challenge, whether that’s a sentence or a
paragraph. For example, in the previous example, the Stimulus would be the actual
description of “A right triangle has one side with a distance of 4 and the other side with a
distance of 6.”
Stem
The stem asks the actual problem to solve. It is critical because it provides the exact
direction that brings the prior elements to conclusion. If the stem does not make sense
in the context of the standard, then there is an issue with the way the assessment is
written.
Confusing stems do a disservice to students, wasting valuable test time and losing
clarity in the process. Using the above example, a proper stem would be “What is the
hypotenuse of this triangle?” However, a misleading stem would ask a question that is
tangentially related (or even unrelated) to the topic; for the triangle example, it might be
“What are the other angles of the right triangle?” While this may be a valid question in
terms of the overall subject, such a stem changes the expectation of the standard and
confuses the student.
One additional not on stems — it may be tempting to use negatives (not, except etc.) in
the stem. However, this type of language tends to create confusion about the actual
purpose of the assessment item. It is considered best practice to avoid any type of
negative terminology in the stem and instead keep it as straightforward and clear as
possible.
Key
The key is the answer. It should be completely clear and not leaving any doubt for
someone who has mastered the skill.
Distractors
Distractors are the other options for the assessment item. There is an artform for
crafting proper distractors. For someone who has mastered the skill, a distractor should
be obviously wrong. However, distractors should be plausible enough that they could be
seen as the correct answer if the test-taker has missed a step or applied a common
misconception.
Reference: https://www.teachthought.com/pedagogy/quality-multiple-choice/
APPLICATION OF CRITERIA TO MULTIPLE CHOICE ITEMS
Multiple choice test questions, also known as items, can be an effective and efficient
way to assess learning outcomes. Multiple choice test items have several potential
advantages:
Versatility: Multiple choice test items can be written to assess various levels of learning
outcomes, from basic recall to application, analysis, and evaluation. Because students
are choosing from a set of potential answers, however, there are obvious limits on what
can be tested with multiple choice items. For example, they are not an effective way to
test students’ ability to organize thoughts or articulate explanations or creative ideas.
Reliability: Reliability is defined as the degree to which a test consistently measures a
learning outcome. Multiple choice test items are less susceptible to guessing than
true/false questions, making them a more reliable means of assessment. The reliability
is enhanced when the number of MC items focused on a single learning objective is
increased. In addition, the objective scoring associated with multiple choice test items
frees them from problems with scorer inconsistency that can plague scoring of essay
questions.
Validity: Validity is the degree to which a test measures the learning outcomes it
purports to measure. Because students can typically answer a multiple choice item
much more quickly than an essay question, tests based on multiple choice items can
typically focus on a relatively broad representation of course material, thus increasing
the validity of the assessment.
Reference: https://cft.vanderbilt.edu/guides-sub-pages/writing-good-multiple-choice-
test-questions/
Rule #1: Test comprehension and critical thinking, not just recall
Multiple choice questions are criticized for testing the superficial recall of knowledge.
You can go beyond this by asking learners to interpret facts, evaluate situations, explain
cause and effect, make inferences, and predict results.
Rule #2: Use simple sentence structure and precise wording
Write test questions in a simple structure that is easy to understand. And try to be as
accurate as possible in your word choices. Words can have many meanings depending
on colloquial usage and context.
Rule #3: Place most of the words in the question stem
If you’re using a question stem, rather than an entire question, ensure that most of the
words are in the stem. This way, the answer options can be short, making them less
confusing and more legible.
Rule #4: Make all distractors plausible
All of the wrong answer choices should be completely reasonable. This can be very
hard to accomplish, but avoid throwing in those give-away distractors as it detracts from
the test’s validity. If you’re really stuck, get help from your friendly SME. (BTW, this word
can also be spelled as “distracter.”)
Rule #5: Keep all answer choices the same length
This can be difficult to achieve, but expert test-takers can use answer length as a hint to
the correct answer. Often the longest answer is the correct one. When I can’t get all four
answers to the same length, I use two short and two long.
Rule #6: Avoid double negatives
No big news here, right? Don’t use combinations of these words in the same question:
not, no, nor, the -un prefix, etc. For example, this type of question could confuse test-
takers: ‘Which of the following comments would NOT be unwelcome in a work
situation?’ Flip it around and write it in the positive form: ‘Which of the following
comments are acceptable in a work situation?’
Rule #7: Mix up the order of the correct answers
Make sure that most of your correct answers aren’t in the “b” and “c” positions, which
can often happen. Keep correct answers in random positions and don’t let them fall into
a pattern that can be detected. When your test is written, go through and reorder where
the correct answers are placed, if necessary.
Rule #8: Keep the number of options consistent
Did you ever have to convince a SME that he or she can’t have answer choices that go
to ‘h’ in one question and ‘c’ in the next? It’s something of a user interface issue. Making
the number of options consistent from question to question helps learners know what to
expect. Research doesn’t seem to agree on whether 3 or 4 or 5 options is best.
Personally, I like to use 4 options. It feels fair.
Rule #9: Avoid tricking test-takers
As faulty as they are, tests exist to measure knowledge. Never use questions or answer
options that could trick a learner. If a question or its options can be interpreted in two
ways or if the difference between options is too subtle, then find a way to rewrite it.
Rule #10: Use ‘All of the Above’ and ‘None of the Above’ with caution
I hate this rule because when you run out of distracters, All of the Above and None of
the Above can come in handy. But they may not promote good instruction. Here’s
why. All of the Above can be an obvious give-away answer when it’s not used
consistently. Also, the All of the Above option can encourage guessing if the learner
thinks one or two answers are correct. In addition, the downside to None of the Above is
that you can’t tell if the learner really knew the correct answer.
Reference: https://theelearningcoach.com/elearning_design/rules-for-multiple-choice-
questions/
VARIATION OF MULTIPLE-CHOICE ITEMS
1. MCQs with Single Correct Answer
In this type of MCQs, only one alternative is the correct answer and the remaining
alternatives are incorrect.
2. MCQs with the Most Suitable Answer
In this type of MCQs, the alternatives differ in the degree of correctness. The most
suitable alternative is the correct answer and the other alternatives serve as distractors.
3. MCQs with Negative Phrasing
In this type of MCQs, the learner has to either identify the incorrect answer or the
alternative which is irrelevant to the question stem.
4. MCQs with Multiple Responses
In this type of MCQs, the learner has to select more than one correct answer.
5. MCQs with Combined Responses
In this type of MCQs, one or more alternatives are correct answers and the remaining
serve as distractors. The learner has to identify the correct answer from the set of
options which has combinations of alternatives. Studies suggest that items of the
combined-response variety have low reliability, so it is better to rewrite it as multiple
true/false.
Reference: https://blog.commlabindia.com/elearning-design/varieties-of-multiple-choice-
questions-in-elearning
