Program Evaluation: takes components of any or multiple of the design discussed earlier this semester into one giants research project.
Assessment:
Program Evaluation (if a program is set out with certain goals you actually meet those goals) and Outcomes Assessment (want accountability to wehre did it actually do anything good; did I improve anything-refers to the idea that not only did I implement the program I improved something in some way)
In both of theses: formative assessment is as you go along (quizzes along the way)
Summative is at the end of something (final research paper)
It's important to have both kinds of assessment b/c you need to know before you get to the end that there is understanding as you go along so adjustments may be made to ensure the final project is going to be successful.
Both Program and Outcome Accountability are important
-you could have a program that does everything it says it's going to do, but it doesn't produce any outcomes
**DARE is an example of this problem.
-or you could have a program that doesn't do what is says it's going to do, but there are changes in outcomes
**so then how do you know what the changes in outcomes are due to, the program or some extraneous variable?
Taking a Look at Program Evaluation
Program Elements
A. Structure
1. Definition of the program
-the program is a developmental educational....
-this program is an integral part....
-this program includes
2. Rationale for the program
-importance of the program
-reasons why students need waht the program offers
3. Assumptions of the program
-identify and briefly describe the premise upon which a programs rests
**ethical and legal practice
4. Resources
-human
-financial
-political
a. program components
-training
-student interventions
-support for system
DOMAINS
Domains identify conceptual aspects of focus or priorities for the program.
-The identification of domains:
1. provides a description of areas of importance
2. provides a structure to organize goals and competencies
3. all assessments should be related to all domains of the program. Ex. don't want someone to throw an extra measurement into the research that has nothing to do with the program being studied.
Competencies are knowledge, skills, attidues (descriptions) of student behaviors that are linked to stated goals for their education. They are based on the assumption that the behaviors can be transferred from a learning situation to a real life situation.
Competencies are:
1. organized into goal content areas
2. arranged developmentally
3. measurable
Components of Program Evaluation
Process vs. Outcome Evaluation
-formative = evaluation of the process of learning
-summative= evaluation of final product (the end)
Stakeholders are anyone who has something to gain from the program. Anywhere from desingers, evaluators teachers and students.
Observations & Interviews 4/2
Interview Process
• Who is our interviewee?
• Design the interview protocol
• Where will the interview take place?
• Memorize the questions.
• Wording of the question:
o No leading questions
o Open-ended questions
• Obtain consent before starting
**respect in quantitative is called control; in qualitative it’s called rigor***
• Demonstrate Respect
o For individual, culture, community, etc.
o Develop rapport
o Stick to protocol as much as possible
• Record comments faithfully
o Use a recorder or videotape if possible
Interview Types
Structured or Semi-Structured
• “Verbal questionnaires”
• More Formal
• Specific protocol prepared in advance
• Designed to elicit specific information
Informal Interviews
• More informal, more like a conversation
• Still involve planning
• Move from non-threatening to threatening questions
• Designed to determine more general views of respondent
Types of Interview Questions
• Demographic Questions –who are these people; where are they from; what’s their occupation
• Knowledge Questions
• Experience or Behavior Questions
• Opinion/Value/Feeling Questions
• Sensory Questions
Interview Questioning Tips
• Ask questions in different ways – watch the person’s facial expression and it will let you know if there is comprehension
• Ask for clarification
• Vary control of flow of information
• Avoid leading questions
• Ask one question at a time
• Don’t interrupt
Observation Process
1. What is the research question? - be very clear of what the question is, what the specific look is, what are you watching for
2. Where is the observation taking place?
3. Who, what, when, and how long? -what: be aware of what on task behavior looks like and what off task behavior looks like
4. What is the role of the observer?
What is the observer’s role?
• Participant Observation
o Complete Participant – acts just like everybody else in the class; may participate in quizzes, asking questions, etc. no one knows who she is
o Participant-as-Observer – acts like a student, may engage in discussion, but doesn’t actually hand in anything and isn’t graded in anyway. Still a participant, but not 100% participant.
• Non-Participant Observation
o Complete Observer - may be sitting with a clipboard checking off things on a rubric
o Observer-as-Participant may be a little more covert, not let others be aware that observation is going on
• Minimize Bias – done in how a rseracher collects data
Collecting Observation/Interview Data
Note Taking Methods
• Field Jottings
• Field Diary
• Field Log
Descriptive vs. Reflective Notes
• Descriptive notes describe the subjects, settings, events, activities, observer behaviors
• Reflective notes reflect on analysis, method, ethics, frame of mind, points of clarification
Data Coding Schemes
Sample Field Notes Form
Research Question:
Date/Time:
Setting:
Descriptive Notes: Reflective Notes:
Sample Interview Protocol
Research Question:
Time of Interview:
Date:
Setting:
Interviewer:
Interviewee:
Position of Interviewee (if relevant):
(Briefly describe project to interviewee and obtain consent.)
Sample Interview Questions:
1. What has been your role in the incident?
Interviewee’s Response Interviewer’s Reflective Comments
2. What has happened since the event that you have been involved in?
Interviewee’s Response Interviewer’s Reflective Comments
3. What has been the impact on the community of this incident?
Interviewee’s Response Interviewer’s Reflective Comments
4. To who should we talk to find out more information about the incident?
Interviewee’s Response Interviewer’s Reflective Comments
(Thank individual for participating in this interview. Assure him/her of confidentiality of responses and potential future interviews.)
For Quiz 7 last question:
As a class, come up with a focus topic for a one-question interview related to the use of technology in education. Our broad focus is availability of technology in workplace.
Next, write the question. Make sure it is clear, unbiased, uses respectful terminology, etc., and that it gets at your focus topic. How does the availability of technology influence your productivity? Prep with how can you use what’s available to do your job? How do you use technology to influence your productivity? Everyone finds one person; record who they are, what their career is, the time, the date,. Tell them the purpose of the study, ask them the question, record what they actually say and then record my reflective notes as well. There is a response box for this on questions #7 on the quiz.
For your answer to the last question on quiz 7, you should interview one person, and record both descriptive (transcript) and reflective notes for this interview.
Subscribe to:
Post Comments (Atom)
No comments:
Post a Comment