Research> Grants FY11

QM Research Grants FY11

1) The Development of Technological Pedagogical Content Knowledge (TPACK) in Instructors Using Quality Matters Training, Rubric, and Peer Collaboration:  Cheryl Ward, Principle Investigator, University of Akron

Findings:

Pedagogy is central to the quality development of online course design and that the TPACK (Technological, Pedagogical, Content Knowledge) conceptual framework is key in enabling instructors to develop new schema for a re-conceptualization of content, pedagogy and technology.

Posited that use of the Quality Matters process helps instructors develop this complex knowledge that enables them to discuss, develop, and implement more effective online learning.  Therefore, purpose of the project was to study the process of how the QM rubric and QM training help instructors develop TPACK knowledge that enables them to discuss, develop, and implement more effective online learning.

The research questions guiding the study were:

• Is QM rubric consistent with TPACK framework to help instructors construct knowledge in quality design and online instruction?
• How is QM rubric implemented and integrated as a catalyst to inform and guide online instructors for quality design and instruction?
• An alignment between the QM rubric and the TPACK conceptual framework was done to determine if any gaps existed between the rubric and the six areas of the framework.  Three professors of instructional technology and three instructional technology students did independent alignment processes with the TPACK framework and the QM Rubric. 

One premise of this study was that the QM Program can inform and facilitate knowledge growth in the TPACK areas.

• The alignment indicated that the QM Rubric is fairly well aligned with the TPACK conceptual framework. It was interesting that a rubric that purports to only address the design elements of an online class align so highly with the pedagogical elements in the TPACK conceptual framework.  
• This alignment supports initial contentions that the elements of the QM Rubric foster discussion and knowledge development in more areas than just design of these environments.  Technological, pedagogical and content discussions are overlapping and connected in a dynamic way that result in inability to discuss or work on them in isolation for online course development.

Four participants were recruited after they finished QM training. Multiple data resources collected in the study revealed that becoming online learners themselves for the QM training helped the participants understand the needs of online learners.

• The QM rubric increased the instructors’ knowledge of the importance of aligning learning objectives to assessment, instructional activities, and technology integration. 
• Even though QM training and the QM rubric did not specifically introduce the Technology Pedagogical Content Knowledge conceptual framework to the instructors, it is clear that their knowledge increased in the areas of technology, pedagogy and content. 
• The learning experiences they shared with the researchers about the QM training also demonstrated that they grew to be more sophisticated online instructors because of the way they designed, modified, and implemented their online courses through the knowledge they gained.

The data analysis results from this study suggest a developmental model that depicts a few key transitional points in order to become an effective online instructors, and how QM training can effectively consider these transitional points to deliver the training more efficiently to enhance the quality of online course with more explicit guidelines to not only course design, but permeate to the other aspects of online teaching and learning.
 
2) Evaluating the Impact of the Quality matters Review Process on Student and Faculty Perceptions of Course Quality:  Tina Parscal, Principle Investigator, University of the Rockies

Findings:

Research Questions:

• What is the extent to which QM certification impact student perceptions of a course?
• Which standards have the greatest impact on student perceptions of a course?
• In an enterprise model of course development, what do instructors perceive as the positive and negative aspects of teaching the same course before and after the QM course review process?

UOR commitment to QM at institutional level:  Although a QM review is generally a 6-8 week process per course, as the phases dictate, chronological progress is dependent on the outcome of each particular state.  If a course suggestions need to be applied, the overall length of the process would grow accordingly.  These dependencies are critical to the overall lifecycle, but also impacted the data collection phases and survey launches associated with the grant research project.

10-12 participants (faculty and staff) are selected per quarter to complete the APRR training.
Standard with the most impact:  All 8 standards identified at least once
Strengths of QM process:  Collaboration (4), Alignment (4), Consistency (4)
 
Student survey responses:

• 4.3 (.68)  The instructions on how to get started in the course were clear.  (Standard 1)
• 4.18 (.88)  The course outcomes seemed achievable.  (Standard 2)
• 4.31 (.82)  The course requirements for student interaction were clearly stated.  (Standard 5)

Unintended results of this study:  In this research we aimed to determine student and faculty perceptions of quality as a result of the QM review process, we discovered that the majority of our courses are QM recognized on the first submission. Although this is not supportive of the grant research process; per se, it does indicate that embedding QM standards in the tools and templates that faculty and the curriculum team use to develop online courses are effective in support the development of high quality courses. This has led to high level of student perception of quality on pre-tests.

3) The Faculty Experience in Preparing Faculty to Use the Quality Matters Model for Course Improvement, Carol Roehrs, Principle Investigator, University of Northern Colorado

Findings:

A descriptive qualitative study to gather information on most efficient and effective ways to use the program by examining the experiences of faculty with QM training and updating of their online courses.

Six faculty members volunteered, two for each level of QM training [self (provided with QM Rubric only), short (attended informal QM training session, no follow-up support), and long (attended QM training and follow-up support made available)].

Following “training”, each course was self-reviewed by study participant, informally peer reviewed, course updated to reflect recommendations, and officially QM reviewed. Participants were asked to comment on their experiences.

• An increase in rubric scores from peer to QM reviews was seen as evidence that participants were able to improve their courses.
• Taking QM training helped faculty improve their courses but how to change the course to meet the standards was sometimes unclear to participants, and the help of instructional designers in course improvement was valued.
• There was a trend toward more agreement of the self- and official-reviews for faculty who participated in the longer training and for the faculty that had more experience teaching online.
• Participants reported that they found QM principles and the rubric to be a valuable approach to improving their online courses.
• Feedback from the faculty peer reviewers was seen as helpful, as was feedback from the official QM review teams.
• Participants lamented the lack of time they had to learn how to use the rubric and to participate in the study.

Despite their ability to make improvements, instructors wanted to have help from experts when they were updating their courses so they could get things right the first time they worked on the course. The experiences of the faculty were positive overall. They liked the rubric and whatever training they had. The pervasive theme in faculty comments was lack of time to assess and revise their courses, much less to participate in the study, which implies that new approaches or processes may need to be devised to support faculty efforts to improve their courses.

These findings support administrative efforts to develop and support faulty peer review programs, perhaps starting with self-review, and to provide expert help with instructional design for faculty who are seeking to improve the quality of their online courses.

4) Impact of Quality Matters Recognition on Course Attrition Rates, Salvatore Diomede, Principle Investigator, Cleveland State University.

Findings:

Attrition in distance higher education is a common problem. Although recent studies have pointed to more and more colleges improving their online retention rates (Parry, 2010). There are many factors that go into determining whether or not a student is likely to continue with his or her studies. Diaz and Cartnal (2006) identified four broad categories of factors that help explain and predict attrition in distance education: student situations, student dispositions, institutional systems, and course content.

Course design, which QM attends to, falls under the category of institutional systems factors. Although course design is important, this prediction model shows that other factors, both internal and external to the student, play a role in explaining attrition, too.

This project was an attempt by the research team to organize a baseline study examining whether course design has an impact on course attrition.

In this study 11 courses were reviewed and improved during one semester (level of improvement was not detailed) and offered by same instructor the next semester.  While there is no statistical evidence that QM impacts students’ online experience and/or withdrawal rates, students’ responses to open-ended questions about their online experience do suggest QM standards are important to their positive learning experience. Student comments reflect three major themes:

• In order to reduce students’ frustration, courses should have clear course/unit level objectives, organization, and instructor expectations; additionally, clear instruction on how to proceed and where to find various course components should be provided. Many participants (especially those participants from Non-QM courses) expressed concern about organizational and navigational issues.
• To promote students’ positive experience in an online course, sufficient interaction and engaging activities or materials are also necessary.  Non responsiveness of instructors was noted as a frustration.  Unfortunately it appears that some instructors believe online teaching is about presenting reading materials/notes, and then testing what students have learned. From both QM standards and students’ perspectives, more interactive activities and materials are necessary, such as synchronous online interaction, case studies, videos, etc.
• While QM standards might facilitate students’ positive online learning experience, without an instructor’s positive presence, students are very likely left with a negative online experience. Initially, we (researchers) were surprised to see quite a few very negative comments about online courses (even some QM recognized courses). After more detailed examination of students’ feedback, we found that students were talking about instructor presence, which is more of a delivery related aspect, rather than course design itself.  Therefore, even if a course is well designed, it does not mean students will automatically glean a positive experience from the course.  A well-designed course still needs an instructor’s creative and diligent delivery.

Discussion:  Although this study as completed in a short two-semester "turn-around" did not find statically evidence either supporting or refuting QM’s effect on withdrawal rates, there are ways to expand upon the research to tell a greater piece of the story of attrition.

One important factor in future research would be to control for the delivery variable - meaning instructor level of interaction with students. According to our survey, instructor presence seems to have a direct effect on students’ perceptions about their online learning experience. This very likely impacts decisions students are making when deciding to persist or withdraw from a course. Therefore, to further understand the effects of QM recognition on attrition, a more accurate control for variables is necessary.

Additionally, a longitudinal study may provide a more complete set of data for comparison. This study had only one semester of data available for the QM recognized courses, which provided for a limited comparison to the three years of previous data for the Pre-QM courses. Further studies could examine multiple semesters of post-QM recognition in comparison to an equal length of time of Pre-QM offerings. 

Questions
Contact Kay Shattuck, D.Ed., Director of Research, Quality Matters, shattuck@qualitymatters.org, 717-387-2741 (mobile) or kay.shattuck (Skype).

 

Research:

About QM Knowledge Generation Initiative | Rubric Committee Open Forums | The Interaction Summit

Research Grants: FY10 | FY11 | Research HOME