Assessment
for Learning Using
Multiple Choice Questions
Dr. Mari Chikvaidze
The 7th Annual Research Student Conference, School of Education, The University of
Northampton, Boughton Green Road, NN2 7AL, Northampton, UK
Friday, 24 June 2016
My Background
• A citizen of Georgia, 3rd year in London
• A wife to a computer scientist
• And a mum of two: Mia (5) and Neo (1)
• A bio-physicist and a qualified maths and physics teacher
• An educational tourist (with MSc and PhD)
• Advanced skills tutor at the Brilliant Club
• A participant of the first cohort of Researchers in Schools
• A honorary research associate at KCL
• An NQT at Lampton academy (just back from the maternity leave!)
• A student at the University of Northampton (PGCE top-up)
Acknowledgments
• James Underwood (The University of Northamton)
• Syed K. Husain and Sarah-Jane Gan (Lampton Academy)
• Dr. Nikesh Solanki and Dr. Mauro Mantega (Researchers in Schools)
• Michael Slawinsky (Researchers in Schools)
Aims and Objectives
What Do I want to achieve?
qIntroduction to the Multiple Choice Questions (MCQs)
qMCQs for AfL in maths
qResearch questions
qIs MCQs a useful tool for assessment?
qHow to design effective MCQs?
qChoice of methods
qPreliminary results
qDiscussion and conclusions
qFuture work
qQuestions and suggestions
Multiple Choice Questions
• MCQs is a form of assessment in which learners are asked to select the best
possible answer(s) out of the set of multiple choices. (Kehoe, 1995)
• A multiple choice test item consists of a question or a problem, known as a
steam, a list of suggested solutions or called alternatives, the correct
answer or best alternative, know as a key, and the incorrect answer(s) or
inferior alternatives, known as distractors. (Burton et al., 1991, p.3)
• Potential advantages of using MCQs for the summative assessment, include
versatility, reliability and validity. (Burton et al., 1991, p.4 and p.5)
• Using Multiple-choice questions for “scanning the class instantaneously
using mini-whiteboards or ABCD cards to follow up misunderstanding”, has
been suggested by Black and Wiliam (2001) in their famous “Inside the
Black Box” paper.
• The importance of the content of questions for making MCQs a formative
assessment for learning tool has also been discussed. (Black, P. and Wiliam,
D., 2001)
MCQs for AfL in Mathematics
• Multiple choice questions are ‘one of the most common forms
of teacher-constructed tests’ (Kehoe, 1995) and have been
widely used for the summative assessment in mathematics.
• The learning outcome tested, may include high level thinking
skills, such as analysis and evaluation, together with the lower
level, recall of facts (Burton, et al., 1991, p.8)
• One example from the literature, where MCQs have been used
for assessing learning formatively, is a ‘technology assisted
formative assessment tool’ developed by Derek Cheung (2006)
Research Questions
1. Is multiple choice questions useful for the formative
assessment in mathematics?
2. How to design effective multiple choice questions, to identify
and address common misconceptions?
Research Design
• This project came about from the need
for improvement of GCSE results for the
C/D borderline students at Lampton
academy.
• An online tool called ExamWizard has
been used for extracting past exam
questions in statistics GCSE, along with
the mark schemes and examiner’s
reports.
• The exam revision quizzes have been
designed using MCQs for every sub-
topic / skill in statistics.
• Pupil’s misconceptions have been
sourced from examiner’s report and
presented among the alternatives.
• Finally a case study has been carried out
by making use of the quizzes from the
foundation tier only.
Example Question
Benjamin rolls a fair 6-sided dice.
Benjamin then spins a fair 4-sided spinner.
He adds the score on the dice and the score on the spinner to get the total.
(b) Complete the sample space diagram to show all the possible totals.
(2)
(c) What is the probability that the total is 2?
A) About 4% B) Unlikely C) Impossible D) 1/24
Example Lesson
Wearing my researcher’s hat
• It was with my researcher’s hat on, that I
discovered how useful formative assessment via “As a newly qualified
multiple choice questions is: teacher with a PhD in my
subject, I believe, I wear
two hats.”
• not only for spotting and clarifying misconceptions,
• but also, for creating stimulating, whole-class
discussions.
Wearing my researcher’s hat
• The researcher’s hat is what gave me the edge:
1. to view my research question in light of the existing
literature
2. to suggest a novel way of designing exam revision
quizzes - using examiners reports as a source of
misconceptions
3. to find a context in which to ask the question of how
effective these quizzes are
With my pragmatic teacher’s hat on…
“By using this
• I rejected ‘positivist rigor’ natural sciences methodology, I aspire to
create a classroom where
approach and accepted that my sample size (low pupils believe that
ability year 10 classes), is way too small for choosing the wrong
answer, out of the set of
making statistical inferences. multiple choices, is an
• Instead, I chose to use the qualitative research opportunity to participate
in meaningful discussions,
methods, to help me understand how to design and learn from them.”
and use multiple choice questions to assess
pupil’s understanding on the spot.
With my pragmatic teacher’s hat on…
• In order to collect the data and answer my
research questions I used:
1. Direct lesson observation of a colleague, to
evaluate usefulness of multiple choice questions
without having to participate.
2. Anonymous pupil questionnaires to capture their
attitude towards learning statistics using multiple
choice questions.
3. Semi-structured teacher interviews to help me
draw a rich, qualitative picture, without being
subjective.
• For data analysis used:
• Coding of the notes and the interviews.
• Cross-comparison of the pupil questionnaire
responses.
Direct Observation
• Having adopted constructivist approach to learning and instruction, I
decided to analyse a small sample, but in great detail!
• I conducted an informal lesson observation of a colleague who agreed
to use MCQs in her revision lesson with the low ability year 10 class.
• In my observation notes I accounted for the presence of the observer.
Pupil Questionnaires
• Two year 10 classes have been asked to complete a
questionnaire, my colleague's year 10 to 11 class (13
pupils), and my own year 9 to 10 class (11 pupils),
both bottom sets.
• The questionnaire comprised 12 closed questions
and opinion scale which made use of the following
codes: sa = strongly agree, a = agree, u = undecided,
d = disagree and sd = strongly disagree)
• The results have been analysed and cross compared
• The main limitation of such questionnaire is that
people tend to answer near the middle, as they
don't tend to seem extreme
Pupil Questionnaires
Pupil Questionnaires
Pupil Questionnaires
Interviews
“(Multiple choice questions) …is a
good assessment for learning tool.
…It is a really quick way to assess
whether you need to do a revision
lesson.”
Interviews
“… You may come across
students who may only
care about what the
answer is and may not be
open to having a
[discussion about]
misconceptions. This in
return could work against
the entire purpose of
having the MCQs.”
“I may consider doing a “spot the mistake”
activity with the kids in future to see if they
are able to identify, explain and rectify
common misconceptions made Hopefully
this awareness will help them develop a
more enquiring mind with respects to their
exams”
Interviews
“It was a great opportunity to see how
the students react to a new activity
“-Is there anything else you’d like to
like this. It also really highlighted the
share regarding today’s lesson or the
importance of students knowing what
project in general?”
the concept is, as oppose to knowing
what the single correct answer is.”
Interviewer Respondent
Coding Results
• The themes in this table
emerged as a result of coding
of 2 interviews, observation of
a colleague and self-reflection
notes
• The frequency of codes and
the relationship between them
has been analysed to generate
4 main themes
• Each theme is supported by a
quotation
Limitations
• Despite their usefulness, there are many disadvantages
to using multiple choice questions.
• One such disadvantage, often mentioned in literature, is
the limited types of knowledge that can be assessed.
• Another drawback of using MCQs is random guessing of
answers; the likelihood of selecting ‘correct’ answer out
of the 4 alternatives is as high as 25%.
• A disadvantage that emerged from this study is that
students tend to copy from each-other, when the
A/B/C/D cards are not shown by the entire class
simultaneously.
Limitations
• Maximum data collection was limited by the fact that I was on a
maternity leave and had little to no access to direct
conversations with teachers and pupils in my school.
• I am fully aware that the chosen sample size is not large enough
for making any statistically significant claims.
• In this initial exploratory case study I did not expect to reach any
conclusive, generalisable outcomes.
• Two interviews and a single observation cannot be taken as
representative of the entire Maths department, let alone the
wider cohort of teachers.
• I understand that the view expressed by the interviewees, as
well as my own interpretations are based on certain beliefs and
biases.
• I fully agree with Yates (2003) and acknowledge that the research
design, as well as the choice of methods has been heavily
influenced by my background and my current epistemological
standpoint.
Discussion and Conclusions
• A case study presented here, explores the potential of using MCQs to assess
learner’s ‘conceptual understanding’ (Skemp, 1976) of mathematics, in a
formative way.
• In this project, multiple choice questions have been designed, using examiners
reports, annually published for statistics GCSE exams.
• These reports proved useful for sourcing pupil’s misconceptions and using them
as distractors, thereby generating stimulating whole-class discussions.
• Teaching resources, including research-informed lesson plans that make use of
MCQs, have been designed and partly evaluated.
• A methodological framework has been developed for teachers of all subjects, to
source pupil’s misconceptions in a systematic manner, from examiner’s reports.
• The aim of addressing the mathematical learning needs of low-attaining students
and those experiencing particular difficulties in maths has been achieved.
• The preliminary findings of this small scale study suggest that well-designed
multiple choice questions work well for AfL in low ability year 10 classes of the
Lampton Academy.
Future Work
• Encourage more teachers to use the resources, this will ideally give a
quantitative dimension to this study, generating statistical data.
• The scope of the use of multiple choice questions can be extended
beyond the exam revision period, and deployed in everyday teaching.
• For instance, MCQ probes can be used as hinge-point questions, after
teaching a new concept, to ensure that students do not ‘misconceive
the teacher’s meaning because they interpret new teaching in terms
of their existing understandings’. (Taber, 2009)
• Finally, studies of the similar scale and nature can be carried out as I
try out different activities in my NQT year.
Wearing two hats
I believe, wearing both hats at the same time
is what helps me narrow the gap between
theory and practice and develop as an
effective teacher researcher.”
Questions and Suggestions
