Surveys and Questionaires
I. Quantitative Methods
Functions of Quantitative Designs
Purpose of...
Experimental Research is to draw conclusions about cause and effect
Comparative Research is to examine relationships between variables
Descriptive (Survey) Research is to summarize characteristics of a particular sample or population
Types of Surveys
• Cross-Sectional –data is collected once from a variety of different groups
• If everyone in the class was surveyed to know the difference between 1st year master’s student’s perceptions and doctorate student’s perceptions. PhD students were at some point 1st year master’s students. So both masters and PhD people would be asked the same questions at the same time. Data is being collect at one point in time but it’s just different people/groups.
• Longitudinal –data is collected over several points in time from a single group
• Trend Studies–survey similar samples at different points in time
o Ex: I could sample the EDPS 6030 cohorts every year; sample Jane Doe first year of master’s and then sample Jane Doe again but when she’s a PhD student.
• Cohort Studies--survey a different sample from the same group at different points in time
o Ex: I could sample 4 students from this group of EDPS 6030 students, with different students in each sample
• Panel Studies--survey the same sample at different points in time
o •Ex: I could sample the current entire group of EDPS 6030 students every year
• ****this is the best way to do a longitudinal study***
• They are hard studies to do as you lose samples over time
Survey Research Methodology
Remember, whenever you do a study, the first thing to look at is WHAT IS THE QUESTION? When going in to the method section the reader must know what the question, or hypothesis is. This is called the focus or unit analysis.
What is the focus (unit of analysis) of the survey?
• Focus should be interesting and important enough for individuals to want to respond
• Objectives should be clearly defined, and all questions should relate to the objectives
Who made up the sample?
• Target Populations vs. Actual Populations
• Demographic Information
• Sampling Response Rates
How is the survey administered?
• – Direct (have a clipboard and go up to people somewhere and ask questions face to face
• – Mail (a lot of $ is spent putting together survey and mailing it out; some come back some don’t) Only a 30% return rate
• – Telephone - going by the sideway due to the no call list
• – In Person – same as direct
• – Internet (60-80% return rate)-easier b/c you can use survey sites, such as survey monkey.
What is explained to the subjects in the cover letter?
• The cover letter usually should say:
o Who is doing it
o What it’s about
o How long it will take
o Why they are being asked
o When it should be returned
• Should explain enough information to tell the person why they should spend their time completing the survey.
Individual Survey Items
• Should be as unambiguous as possible (people should be able to understand without having to ask questions)
• Should be short and well-organized
• Should not ask more than one question at a time
• Should not contain biasing or overly technical terms
• Should not use leading questions
• Should avoid using double negatives
• Should not overuse contingency questions (if you have choose this, or if you haven’t, choose this)
SO, SURVEY QUESTIONS SHOULD BE CLEAR, SHORT AND UNBIASED.
• Responses (Use reverse scoring to get a lot more truthful response)
Survey Item Types
Closed-ended (easy to analyze: check the box, choose a, b, or c.)
• Advantages
o Consistent and unambiguous
o Easier to score and interpret
o More likely to elicit responses
• Disadvantages
o Limits amount of “data” received (you only get the data you asked for, have you smoked. They can say yet but it doesn’t let you, the tester, know if it was in high school or now or yesterday or two minutes ago).
o More difficult to construct good questions
o More questions are usually required
Open-ended
• Advantages
o Allows more freedom in individual responses
o Easier to construct
• Disadvantages
o More difficult to score and interpret
o May elicit more ambiguous responses
o Less consistency in individuals’ response rates and response contents
Analyses of Survey Data: Closed-Ended Questions
Descriptive Statistics
Percentages
Mean, Median, Mode
Frequency Distributions
Comparative Statistics (the P value should be less than .05)
Correlations (r, R)
Chi Square (χ2) (
Inferential Statistics (the P value should be less than .05)
T-tests (t)
ANOVAs (F)
MANOVAs(F)
Analyses of Survey Data: Open-Ended Questions
Use of Qualitative Analysis Methods
• Content Analysis
• Triangulation (support for the idea from at least three different sources of data)
• Emergent Patterns
• Emergent Hypotheses (quotes seen are trying to back up their data)
Validity & Reliability
Pretesting: good surveys are pretested to see if potential respondents understand what I’m asking. It’s done to make sure we haven’t screwed something up.
Reliability Checks: need someone to check how reliable the data has been coded or entered in to a program.
Validity Checks
• Internal Validity (check all of the threats to internal validity. Check to make sure questions won’t bias the participant; that location doesn’t bias the participants; where it’s sent doesn’t bias them
• External Validity (am I asking questions that can be asked to anyone; is it applicable to those outside of the specific group it was written for).
March 5
Correlation Designs: Used to measure relationships between two or more variables (denoted by little 'r')
****you cannot draw conclusions about cause and effect with correlational designs!!!!****
*There are two kinds:
1. Explanation
2. Prediction
B. Robinson et al (2007) - researchers that are making conclusions about cause and effect on correlational information
C. Three necessary conditions for causal relationships:
1. Cause is related to effect
2. No plausible alternative explanation for the effect exists other than the cuase
3. Cause is beore effect
D. Look at Percentage of Nonintenrvention Articles on powerpoint for today's notes to see graph.
**Finished taking notes in Word as I became frustrated trying to format every line!
Here are our cohorts notes:
orrelational Designs
–Used to measure relationships between two or more variables (r)
• Explanation
• Prediction
–No conclusions about cause and effect may be drawn
More and more researchers are making causal conclusions inappropriately.
They are making causal conclusions from correlational data, which you can NOT do.
Analyzing Data
• Correlation Coefficients
– “r”can range from -1 to +1
– Negative correlation = as one variable decreases, other increases
The negative doesn't mean the correlation is any less strong, it simply goes in the other direction.
ex. Typing skill and typing errors
– Positive correlation = as one variable increases, other also increases
ex. Standardized test scores and GPA
– Zero correlation = no relationship between the two variables
Plotting Correlational Data --Scatterplots
The more scattered the dots are on the graph, the closer the correlation coefficient gets to zero
Interpreting Correlations
• Correlation coefficient “r”
–Ranges from -1 to 1
–Indicates whether relationship is positive or negative
–Statistical significance (p-value) depends on size of relationship and size of sample
Magnitude of r Interpretation
Look at the absolute value of r to determine the magnitude of r
So what does r mean, anyway?
.00 to .40 weak relationship
.41 to .60 moderate relationship
.61 to .80 strong relationship
.81 or above very strong relationship.
More on CorrelationalDesigns
• Predicting from Multiple Variables
–Can compute several individual correlation coefficients to produce a correlation matrix (a table showing how the different variables correlate)
• Or can conduct a Multiple Regression Analysis - puts all the variables together for one coefficient (R)
–Yields coefficient R, which can be interpreted similarly to the simple correlation, r
–Also yields R2(coefficient of determination)
Coefficient of determination = Effect size estimate for correlational studies = how much of the result we can explain by the effect of all the other variables
ex. Reading Comprehension has many variables. R2 = How much of the reading comprehension can we explain through these other factors?
Imagine trying to dissect the concept “Reading Comprehension.” It is made up of several related factors, such as:
•Fluency
•Intrinsic Motivation
•Verbal IQ
•Working Memory Capacity
•Background Knowledge
If we sum up the portion of those components that uniquely overlaps with reading comprehension, we can explain a big part of reading comprehension. That is essentially what R2 in a multiple regression does.
So R2tells you, given all of the factors we’ve entered into the equation, how much of Reading Comprehension can be explained by those factors.
Other Uses of Correlational Data
• Structural Equation Modeling (aka. SEM, Path Analysis, Hierarchical, Stepwise) –maps out relationships among several variables
–Instead of lumping everything into a multiple regression, we can put them into a structural equation model; Allows researchers to see the “big picture”as well as relationships among individual variables
• Factor Analysis –how do individual variables or items in a measure combine to create “mega-
variables”
–E.g. Several items on a questionnaire might relate to your interest in a topic. Instead of treating each item as an individual variable, we combine them as one “factor”
•Path Modeling - shows how all of the factors correlated together AND how they work together to predict supportive computer use (looks like boxes with arrows going every which way showing correlational values between each combination)
Feb. 28 Notes
Other Comparative Designs
Review of Terms:
Random Selection/Sampling vs. Random Assignment
Random Selection = How do I get my sample? All people in your population have an equal chance of being selected
Random Assignment = Once you have your sample, how do you assign them to a group?
Internal Validity vs. External Validity
Internal Validity = control of the experiment
External Validity = generalizability of your experiment
You want to have the correct balance between the two
Independent, Dependent, & Extraneous Variables
Independent Variable = What you are manipulating
Dependent Variable = depends on the independent
Extraneous Variables = the things that mess everything up
Between vs. Within Subjects Variables
Between subjects variable = looking for a difference between subjects - they don't get the same experience in both groups - but you need to make sure that both groups are as similar as possible to confirm that the only differences between groups is your independent variable
Within subjects variable = Every individual gets both experiences in the experiment - ex. Pre vs. Post - Within is more ideal because you know your groups will be consistent for both independent variables
Factorial Design = measures the impact of more than one independent variable at once
Benefit: you can see if there is an interaction between the different independent variables
(No more than three independent variables, otherwise it gets too complicated and you can't tell what variables are interacting with which variables very clearly)
Experimental vs. Quasi-Experimental Designs
"True" Experimental Designs = involve random assignment to conditions manipulated by experimenter
Quasi-Experimental Designs = involve comparisions of groups in pre-selected conditions or groups - I design the study before I collect the data
Causal Comparative Designs = are ex post facto quasi-experimental designs; They involve comparison of pre-selected conditions/groups after the fact
Time-Series Design
Measure dependent variable a lot over time
Experimental Designs, cont.
Single Subject Designs
-Like mini longitudinal experimental designs, on individuals or small groups of individuals
-Similar to pretest/posttest designs; examines DV before and after IV
-Used when it doesn’t make sense to pool effects across individuals - ex. When working with children with special needs, the specific behaviors and needs of one child are not the same as others' - But tracking that one child over time may help develop strategies to help that specific child - you're not trying to generalize your findings, you're just trying to help that one individual
--Special populations
--Focus on success of specific interventions with specific individuals
Possible Single-Subject Designs
A-B Design = baseline, followed by intervention
A = Baseline
B = Intervention
But what happens after the intervention is removed? Does behavior go back to baseline?
A-B-A Withdrawal Design = baseline, followed by intervention, concluded with baseline
When the intervention is removed, does behavior go back to baseline?
Ethical issue: is it OK to intervene but then leave subjects back at baseline behavior, especially if we know that the intervention is needed?
A-B-A-B Design = one step further; instead of leaving subjects back at baseline, present intervention again (more ethical)
Multiple Baselines Designs = alternative to ABAB design; used when it’s not ethical to leave the subject at the baseline condition and when measures on multiple Dependent Variables (DVs) are taken.
-Taking baselines for multiple behaviors at same time – whether it’s one behavior in multiple individuals, multiple behaviors in one individual, one type of behavior in one individual in multiple settings, etc.
-Difficult to use, because must ensure that multiple DVs aren’t related to one another
Issues to Consider: Internal Validity
Is Treatment Standardized?
-# Variables Changing Across Conditions ? - Is the behavior changing because of my intervention, or is there another explanation?
Condition Length?
-Degree and Speed of Change? - You want to show a stable trend. Do you have enough data points to see a stable trend in the experiment results? - The "magic" number is 7 measurements to see a stable trend
Experimenter Effects?
-Because this is a one-on-one experiment, the experimenter is likely to have an impact on the individual
Practical Significance of Results?
-Practical significance is looked at more than statistical significance because of the single-subject design. - ie. Did it help the student improve?
Single Subject Designs and External Validity
It is very difficult to generalize the results of these designs beyond the sample studied – WHY?
-Because it was only designed for one person for a specific purpose. It is not necessarily meant to be generalized.
Thus, it is VERY IMPORTANT for this type of research to be replicated before strong conclusions are drawn.
*We now have all the information we need for quiz 3 if we want to take it now.
No comments:
Post a Comment