Feb. 6 - Meta-Analysis

Reviewed the Quiz #1.

Meta-Analysis and other Methods of Research Synthesis
• Levels of Research Synthesis

• Literature Reviews
o -- allows for in depth discussion of individual studies’ findings and their theoretical implications
o -- no weight given to statistics in the discussion
• Numerical Reviews (“Vote Counting”)
o Studies that are statistically significant “cast votes” for effects
o Statistical significance depends on many factors
• Meta-Synthesis (synthesizing qualitative research findings)
o Used in qualitative research (qualitiative research is concepts and ideas and general results)
o It is NOT collection and aggregation of research results
o It is “the bringing together and breaking down of findings, examining them, discovering the essential features, and, in some way, combining them into a transformed whole” (Schreiber et al., 1997, p. 314)
• In qualitiative research the goal of the study is to try and understand it
o One of the important issues is how do they actually go about doing the study.
• What is the selection criteria? How am I going to select studies on whatever? What is my inclusion criteria? (is it published in a peer review journal? You want to do a peer review so they have been criticized, or passed a level of critique. There may be a restriction/criteria selection on date, participants or methods used.
o Once you have chosen, determined the critiera, you start to code and categorize all of the relevant information.
o Teams or Individuals? In general, teams are better. Results are a lot more valid.
o Audit Trail: a list or description of exactly what you did.
o Triangulation: you want to approach a topic or finding from at least three different perspectives

II. Meta-Analysis
• Origins of Meta-Analysis – started in agricultural research

• Definition
o Statistical technique that enables the results from a number of studies to be combined to determine the average effect size of a given independent variable.
• Is the effect significant or not? What was found, is it significant and how big was the effect? Remember, if it’s significant or not, that is the P value. ‘N’ indicates sample size.
o Supposedly more objective than narrative literature review

• Advantages
o Use stats to organize and extract info (organizing numbers)
o Eliminate selection bias (???) this is a bold statement;
o Makes use of all info in a study (???) only if you include it or pay attention to it
o Detects mediating characteristics

• Limitations
o No guide for implementation
o Time-limited
o No real rigid methodological rules
o Only as good as the studies it is based on

QUIZ QUESTION: META ANALYSIS ARE ONLY AS GOOD AS THE STUDIES I’M USING TO GET MY RESULTS.

III. Meta-Analysis Methodology

1. Research Focus – what’s the question that needs to be answered

2. Sampling – not recruiting partipants. You’re going out and finding studies. The studies that your meta anlaysis is based on make up the sample.
• Inclusion Criteria? see above re: peer reviewed, dated, etc.

3. Classifiying and Coding – not classifying qualitative or abstract data. Specific findings from other studies.

4. Role of the “Audit Trail” – pertains just as much to meta-analysis as it does to meta-synthesis (qualitative and quantitative are equally important).

5. Data Analysis
• Significance vs. Effect size vs. Effect Magnitude
o Is the effect size small, medium or large?
o Of what magnitude is the effect? Small = weak or large = really important and dramatic.
• Comparison to Zero Effects (Cohen’s d is the effect size)
• Positive vs. Negative Effect Sizes – usually only see positive effect sizes. Negative is usually the opposite of what you are wanting.

6. Interpretation and Beyond
• Raises Researchers Consciousness? – what are the important questions? What is important to look at?
• Highlights Gaps in Field Knowledge Base (lets researchers know what they need to go out and research)
• Motivates Future Research
• Implications for Practice?
Synthesis is qualitative; analysis is quantitative