Advanced Research Methods

and Scientific Writing

Collecting Data through Questionnaires
 Learning Objectives
By the end of this lecture, you should be able to:
1. Define the scientific method and understand its importance in scientific
inquiry.
2. Identify and describe the key steps of the scientific method.
3. Apply the scientific method to formulate hypotheses and design
experiments. Learn hypothesis-based science.
4. Analyze how the scientific method fosters critical thinking and reliable
conclusions.
5. Recognize the role of the scientific method in various scientific
disciplines and everyday problem solving.
 Learning outcomes
• Understand the advantages and disadvantages of questionnaires as a
data collection method;
• Be aware of a range of self-administered and interviewer-administered
questionnaires;
• Be able to select and justify the use of appropriate questionnaire
techniques for a variety of research scenarios;
• Be able to design, pilot and administer a questionnaire to answer
research questions and to meet objectives;
• Be able to apply the knowledge, skills and understanding gained to
your own research project.
 Preamble
• Within research, the greatest use of questionnaires is made within
the survey strategy.
• Generally, data collection with questionnaires is where the
person answering the question actually records their own
answers.
• Questionnaire is used as a general term to include all techniques
of data collection in which each person is asked to respond to
the same set of questions in a predetermined order.
• It, therefore, includes both structured interviews and telephone
questionnaires as well as those in which the questions are
answered without an interviewer being present, such as online
questionnaire.
 Response, Reliability and Validity
• The design of your questionnaire will affect the response
rate and the reliability and validity of the data you collect.
• Response rates, validity and reliability can be maximized
by:
– Careful design of individual questions;
– Clear and pleasing layout of the questionnaire;
– Lucid explanation of the purpose of the questionnaire;
– Pilot testing;
– Carefully planned and executed administration.
 Questionnaires
• Standardized, structured instrument for collecting/gathering
data.
• … a prepared set of questions (or measures) to which
respondents or interviewers record answers.
Purpose
– To obtain information that cannot be easily observed or is not
already available in written or electronic form
Types of Questionnaires
 Types of Questionnaires
• The design of a questionnaire differs according to how it is
administered and, in particular, the amount of contact you have with
the respondents.
• Self-administered questionnaires are usually completed by the
respondents.
• Such questionnaires are administered electronically using the Internet
(Internet-mediated questionnaires) or intranet (intranet-mediated
questionnaires), posted to respondents who return them by post after
completion (postal or mail questionnaires), or delivered by hand to
each respondent and collected later (delivery and collection
questionnaires).
 Types of Questionnaires
• Responses to interviewer-administered questionnaires are recorded
by the interviewer on the basis of each respondent’s answers.
• Questionnaires administered using the telephone are known as
telephone questionnaires.
• The final category, structured interviews (sometimes known as
interview schedules), refers to those questionnaires where interviewers
physically meet respondents and ask the questions face to face.
• These differ from semi-structured and unstructured (in-depth)
interviews, as there is a defined schedule of questions, from which
interviewers should not deviate.
 Types of surveys vs Effort
• Mail/self-administered
– cheapest, wide coverage, standardized, low response rate (?)

• Telephone
– medium cost, wide coverage, medium response rate,
standardization depends on interviewer

• Face to face
– most expensive, coverage depends on personal contact, highest
response rate
 The Choice of Questionnaire
Influenced by a variety of factors related to research question(s) and
objectives and in particular the:
– Characteristics of the respondents from whom you wish to collect
data;
– Importance of reaching a particular person as respondent;
– Importance of respondents’ answers not being contaminated or
distorted;
– Size of sample you require for your analysis, taking into account the
likely response rate;
– Types of question you need to ask to collect your data;
– No. of questions you need to ask to collect your data
 The Choice of Questionnaire
Your choice of questionnaire will also be affected by the
resources you have available, and in particular the:
– Time available to complete the data collection;
– Financial implications of data collection and entry;
– Availability of interviewers and field workers to assist;
– Ease of automating data entry.
 When to use questionnaires
• Questionnaires are usually not particularly good for exploratory or
other research that requires large numbers of open-ended questions.
• They work best with standardized questions that you can be confident
will be interpreted the same way by all respondents.
• Questionnaires tend to be used for descriptive or explanatory
research.
• In contrast, explanatory or analytical research will enable you to
examine and explain relationships between variables, in particular
cause-and-effect relationships.
 When to use questionnaires

Questionnaires are useful in gathering information from

key respondents about:
– Attitudes
– Beliefs/Opinions
– Behaviors
– Characteristics
 When to Use Questionnaires

• Questionnaires are valuable if

– Respondents are widely dispersed
– Many members are involved with the project
– Explanatory work is needed
– Problem solving prior to interviews is necessary
– Others…
 When to use questionnaires
Best used when:
– There is a large sample
– You want fairly straightforward information
– You want standardized data from identical questions
– You are more interested in what occurs rather than why or
how
 Merits of Questionnaires
• There is low cost even when the universe is large and is
widely spread geographically.
• It is free from the bias of the interviewer; answers are in
respondents’ own words.
• Respondents have adequate time to give well thought
out answers.
• Respondents, who are not easily approachable, can
also be reached conveniently.
• Large samples can be made use of and thus the results
can be made more dependable and reliable.
 Demerits of Questionnaires
• Low rate of return of the duly filled in questionnaires; bias due to no-
response is often indeterminate.
• It can be used only when respondents are educated and cooperating.
• The control over questionnaire may be lost once it is sent.
• There is inbuilt inflexibility because of the difficulty of amending the
approach once questionnaires have been dispatched.
• There is also the possibility of ambiguous replies or omission of replies
altogether to certain questions; interpretation of omissions is difficult.
• It is difficult to know whether willing respondents are truly representative.
• This method is likely to be the slowest of all.
 Limitations of questionnaires
• Can be superficial
– difficult to capture the richness of meaning

• Cannot deal with context

– information is collected in isolation of environment

• Information is not causal

– cannot attribute cause-effect relationships

• Information is self-report
– which does not necessarily reflect actual behavior
 Designing the Questionnaire

Stages that must occur for a question to be valid & reliable

 Steps in Questionnaire design

Step 1: Initial Considerations

Step 2: Clarification of Concepts

Step 3: Typology of a Questionnaire

Step 4: Pre-testing of a Questionnaire

Step 5: Administering a Questionnaire

 Questionnaire Design:
Initial Considerations

• Clarify the nature of the research problem and objectives.

• Develop research questions to meet research objectives.
• Define target population and sampling frame.
• Determine sampling approach, sample size, and expected
response rate.
• Make a preliminary decision about the method of data
collection.
 Designing a questionnaire

• Is a questionnaire (the most) appropriate?

• Identify the resources that are available.

• Decide what information you need.

• Select items for inclusion.

• Design the individual questions.

 Research Questions
Examples:
• Is staff motivation a problem in this organization?
• Do employees in this organization support diversity in the
workplace?
• What factors influence productivity in this organization?
• What are the most important factors influencing the purchase of
a laptop computer?
• What are the good and bad issues about the government’s
policies on fighting corruption?
 Questionnaire Language
Questionnaire language should be:
– Simple
– Specific
– Free of bias
– Not patronizing
– Technically accurate
– Addressed to those who are knowledgeable
– Appropriate for the reading level of the respondent
 Broad considerations
• Sequencing of questions.
• Identification of concepts.
• How many questions are required to capture each
concept?
• Question wording.
• Overall length of questionnaire.
• Placing of sensitive questions.
• Level of measurement.
• Open-ended vs. close-ended questions.
 Type of information

• Knowledge - what people know

• Opinions, attitudes, beliefs, values - what people think about
an issue
• Behavior - what people do
• Attributes - what are people’s characteristics
• Remember - based on self-report
 Knowledge
What is the recommended interval between maintenance/service of a
grader?
❑ 200 hours

❑ 300 hours

❑ 500 hours

❑ Not sure

At what age should kids join school?

❑ 2 years

❑ 3 years

❑ 4 years

❑ Not sure
 Opinions

What do you think are the major issues affecting

Uganda’s economy at the moment?
_____________________________________________________
_____________________
Whom do you think will be elected speaker in the 12th
parliament?
_____________________________________________________
__________
 Behavior

• Do you have an insurance plan for your family?

❑Yes
❑No
• Do you often use the gym for your personal fitness?
• Do you use fortified products in your family?
 Attributes

• When did you graduate from university?

___________________________________________

• Where do your kids go to school?

________________________________
• Do you earn income from extra jobs?
_____________________________
 Question Types
Questions are designed as either
– Open-ended
• Try to anticipate the response you will get
• Well suited for getting opinions
• Useful in explanatory situations

– Closed
• Used when all the options may be listed
• When the options are mutually exclusive
Open-Ended and Closed Questions

Open-ended Ease of: Closed

Slow Speed of completion Fast

High Exploratory nature Low

High Breadth and depth Low

Easy Ease of Preparation Difficult

Difficult Ease of Analysis Easy

 Types of questions
• Open-ended question(s)
– What…? Why…? How…?
– No predetermined responses given
– Able to answer in own words
– Useful for exploratory research and to generate ideas
– Flexible
– Requires skill in asking questions and interpreting results
– Answers can lack uniformity and be difficult to analyze
 Open-ended questions
• What do you think about the quality of products made in
Uganda?
• How do you typically decide which restaurant you will eat
at?
• Please list up to three things you like about your job?
1. -------------------
2. ------------------------
3. --------------------------------
 Types of questions
Close-ended question(s)
• Designed to obtain predetermined responses
(Yes/No; True/False; strongly agree-strongly disagree, etc..)
• Easy to count and analyze
• Easy to interpret
• May not have catered for all possible answers
• Questions may not be relevant or important
 Closed-ended questions
The curriculum of Mak encourage graduates to become job
creators:

Which country in Europe have you visited in the last one

year?
a) Belgium, b) Germany, c) the UK, d) Holland, e) Spain.

• Please rate the quality of groceries from Vivo Supermarket:

Poor Fair Good
 Opening questions
Screening or filter questions
• Are used to ensure respondents included in the study are
those that meet the predetermined criteria of the target
population.
• E.g. Tonight we are talking with individuals who are 18 years
of age or older and have 50% or more of the responsibility for
banking decisions in your household. Are you that person?
___ Yes ___ No.
 Opening Questions
Rapport Questions
• Are used to establish rapport with the respondent by gaining
their attention and stimulating their interest in the topic.

• Have you watched any good movies in the last month? __

Yes __ No.

• What is your favorite restaurant in Kampala?

 Filter questions
• Filter questions useful to ensure respondents only
answer relevant parts of questions
• Unfiltered:
– If you use a drawing software program, which one do
you use?
• Filtered:
– Do you use a drawing software program?
❑ No - skip to next question
❑ Yes - which one?
 Branching Questions
Are used to direct respondents to answer the right questions
as well as questions in the proper sequence.
– Have you seen or heard any ads for mobile phones in the last 30
days?
– If “No”, go to question #10.
– If “Yes”, were the ads on radio or TV or both?
– If the ads were on TV or both radio and TV, then go to question #6.
– If the ads were on radio, then go to question #8…
 Getting the question right
• Question wording
– Questions need to be clear, simple and precise
– Poorly written questions lead to ambiguity and
misunderstandings and can be wasteful
• Responses
– open, closed, what type of response set will you use?
 Common problems with wording
• Leading questions: a question that suggests or implies a
certain answer.
– Do you prefer being examined by a doctor of your own sex?

• Better: Would you rather be examined by a:

❑ Male doctor
❑ Female doctor
❑ Either/doesn’t matter
 Leading questions
• The bandwagon effect
• e.g. Most Ugandans have stopped eating junk food. Do
you eat junk food?
• Partially mentioning some alternatives
• e.g. which fast food restaurant do you prefer, KFC or
others?
• Questions with the phrase:
• Like “… Don’t you think that ..”
 Common problems with wording
• Vague questions:

– Taken altogether, how happy are you with your stay in hospital?

– Have you seen a doctor during the past year?

• Vague questions

– Taken altogether, how happy are you with your stay in hospital?

– Overall, how would you describe the care you received in

hospital?
 Common problems with wording

• Have you seen a doctor during the past year?

• In the last 12 months, have you visited a Doctor?
• How long has it been since you last visited a Doctor?
(within the last month, between 1 and 12 months ago, more
than 12 months ago)
 Common problems with wording
Biased or value-laden/loaded questions:
• A question that is designed to suggest a socially desirable
answer.

• Usually it is emotionally charged.

– E.g. Do you think fizzy drinks are bad for teeth?
– E.g. In your opinion is it fair that Multiplex should be harassing car
owners with parking tickets?
• Framing question such that honest answer is painful or embarrassing →
use counter biasing statement.
 Common problems with wording
• Biased or value-laden questions:
– Do you think technology innovations is a waste of time?
– What do you think of technology innovations?
• Threatening questions
– How often do you smack your child?
– Do you know enough about treating patients at risk of
stroke?
 Common problems with wording
Threatening questions
– How often do you smack your child?
– How often do you use each of the following to discipline
your child… …?
– Do you know enough about treating patients at risk of
stroke?
– How would you rate your knowledge of X for treating
patients at risk of stroke (I know very little, I need to learn a
little more, I need to learn a lot more etc..)
 Common problems with wording
Double-barreled questions
• Two concepts in one question
• Example: Have you had a neck ache or a back ache since your
last visit?
• Double-barreled questions
– Have you had a neck ache or a back ache since your last visit?
– Since your last visit, have you had any of the following symptoms (tick as
many that apply):
❑ Neck ache
❑ Back ache
❑ Headache
 Common problems with wording
Negative questions
• Avoid using negative wording ‘not’, ‘rarely’, ‘never’, or
words with negative prefixes ‘in-’, ‘im-’, ‘un-’.
– Doctors should not be required to see patients outside surgery hours:
agree/disagree
– Doctors should be required to see patients outside regular hours:
Agree/Disagree
 Common problems with wording
Complex questions
– On a scale of 1 to 10, please rate for each of the 12
categories listed below, your level of knowledge,
confidence and experience.
– Please complete the table below about your level of
knowledge, confidence and experience in each of the
following areas.
– Please complete the table below about your level of
knowledge, in each of the following areas.
 Responses
• Closed-ended questions are usually followed
by a set of responses
• Choose type of scale:
– nominal
– ordinal
– continuous (summed items with ordinal response
scale)
 Responses
Nominal
• Are you:
❑ Male

❑ Female
• What is your marital status:
❑ Single

❑ Married

❑ Widowed

❑ Divorced

❑ Separated

❑ Other
 Responses
• Limited choices of responses, lack of consistency in what a yes/no,
agree/disagree response means
Do you have trouble climbing stairs?
• Attitudes and behaviors lie on a continuum
To what extent do you experience difficulty when climbing stairs in your
house?
❑ None

❑ A little

❑ Quite a bit

❑ A lot

❑ I do not have stairs in my house

 Responses
Ordinal
• What is the highest level of education you have reached:
❑ Did not complete primary school
❑ Completed primary school
❑ Did not complete O-Level
❑ Completed O-Level
❑ HSC or equivalent
❑ University
 Problems with responses
Effort required to answer questions
– E.g.: During your last consultation with your doctor, did the
doctor discuss medications to help lower your blood
pressure?
– What is meant by discuss?; relies on recall of discussion
– Many respondents will tick a response that is ‘satisfactory’;
that is, to just ‘tick a box’.
 Problems with responses
Fatigue/boredom/disinterest
– agree with everything
– just say ‘don’t know’
– always choose first response
– ‘randomly’ respond without considering the question
• Social appeal
• Aversion to extreme ends of the scale
 Problems with responses
Minimizing fatigue/boredom
• Keep task simple
– E.g. easier to recall more recent events

• Keep words short and easy to understand

• Maintain motivation of participants
– ensure task is relevant

• Ask people to justify their responses

 Problems with responses

• Minimizing social desirability

– is difficult
– instruct that it is ok not to know something
• Aversion to extreme ends of scale
– Avoid absolutes ‘never’, ‘always’
– Expand number of categories by including throw away
categories at the end:
• never, almost never, infrequently, sometimes, usually, almost always,
always
 Problems with responses
Minimizing ceiling effects
• ‘Average’ response doesn’t have to be middle response
 Problems with responses
Halo effects
– The tendency for an impression created in one area to
influence opinion in another area.
– often occurs when evaluating individuals
– judgments made on aspects of a person’s performance
influenced by overall impression of the person
– a global summary just as informative
 Problems with responses
Order effects
– Maybe more likely to endorse first or last response
– Preceding questions may influence responses to questions
that follow
 Problems with responses
• Randomize order of response sets between individuals
• Randomize order of items within questions
• Maybe possible to randomize order of questions
• Don’t always present ‘positive’ or ‘negative’ sounding
response first.
• Easier to randomize in computer-assisted interviews than
paper and pen questionnaires.
 Ordering questions
• Sequence should be logical to the respondents and flow
smoothly from one question to the next.
• Questions tend to flow from:
– general to specific
– impersonal to personal
– easy to difficult
 Questionnaire Design

• Question Sequence
– Order Bias
– Funnel Technique
– Filter Bias
From general
• Question Layout to Specific
 Do’s of Questionnaire Design
• Include instructions to participant.
• Awaken interest and motivate participation.
– Place interesting questions and/or attention-getting questions early
– Place classification questions at end
• Obtain trust and willingness to disclose non-sensitive information
first.
– Place sensitive or ego-involving questions in middle
• Start with simple questions and then move toward more
complex.
– Use funnel approach (general to specific)
• Use transition statements between different topics.
 Validity and reliability
• Validity
– question measure what you claim it measures
– problem with self-report
• Reliability
– results are reproducible or consistent with similar groups of
respondents, over time and when other people administer
the questionnaire
 Internal Validity
• Internal validity in relation to questionnaires refers to the
ability of the questionnaire to measure what you intend it to
measure.
• This means you are concerned that what you find with your
questionnaire actually represents the reality of what you are
measuring.
• This presents you with a problem as, if you actually knew the
reality of what you were measuring, there would be no point
in designing your questionnaire and using it to collect data!
 Content Validity
• Content validity refers to the extent to which the
measurement device, in our case the measurement
questions in the questionnaire, provides adequate
coverage of the investigative questions.
• Judgment of what is ‘adequate coverage’ can be made in
a number of ways.
– One is through careful definition of the research through the
literature reviewed and, where appropriate, prior discussion with
others.
– Another is to use a panel of individuals to assess whether each
measurement question in the questionnaire is ‘essential’, ‘useful but
not essential’, or ‘not necessary’.
 Criterion Validity
• Criterion-related validity, sometimes known as predictive validity, is
concerned with the ability of the measures (questions) to make
accurate predictions.
• This means that if you are using the measurement questions within
your questionnaire to predict customers’ future buying behaviors, say,
then a test of these measurement questions’ criterion related validity
will be the extent to which they actually predict these customers’
buying behaviors.
• In assessing criterion-related validity, you will be comparing the data
from your questionnaire with that specified in the criterion in some
way.
• Often this is undertaken using statistical analysis such as correlation.
 Construct Validity
• Construct validity refers to the extent to which your
measurement questions actually measure the presence of
those constructs you intended them to measure.
• This term is normally used when referring to constructs such
as attitude scales, aptitude and personality tests and the like
and can be thought of as answering the question:
– ‘How well can you generalize from you measurement questions to
your construct?’
– Because validation of such constructs against existing data is
difficult, other methods are used.
 Layout
• Just as important as wording
• Aim for a professional look
• Tips:
– cover letter/introductory page giving study title, organization, aims
of the survey
– enough space for open-ended questions
– font large enough to read without strain
– consistent and clear instructions
– don’t split questions or answers across pages
– enough white space
 Piloting a questionnaire
• Pilot
– Discuss it with your colleagues
– Pilot it with the target group and as you intend to administer it

• Evaluate and modify on basis of pilot

• Conduct survey, including protocol for maximizing
response rates
 Pretesting Questionnaires
• Objective is to identify possible shortcomings of the
questionnaire.
• Approaches could be formal or informal.
• Can assess:
– Clarity of instructions
– Cover letter
– Clarity of questions
– Adequacy of codes and categories for pre-coded questions.
– Quality of responses
– Likely response rate
 Pretesting Questionnaires
No hard and fast rules:
– Ability to preform meaningful analyses
– Time to complete questionnaire
– Cost of data collection
– Which questions are relevant
– Whether questions have been overlooked
– Sources of bias.