Logic models: A tool for effective program planning, collaboration, and monitoring

Wendy Kekahio Brian Lawton Mid-continent Research for Education and Learning University of Hawaii at Manoa

Louis Cicchinelli Paul R. Brandon Mid-continent Research for Education and Learning University of Hawaii at Manoa

This guide, an installment in a four-part series on logic models, describes the role of

logic models in effective program planning, collaboration, and monitoring. It defines the

four components of these models—resources, activities, outputs, and outcomes—and

explains how they connect. Using logic models can help practitioners and evaluators

better understand a program’s mechanics and structure and chart a course toward

improved policy and practice.

A logic model is a visual representation of the assumptions and theory of action that underlie the struc- ture of an education program. A program can be a strategy for instruction in a classroom, a training session for a group of teachers, a grade-level curriculum, a building-level intervention, or a district- or statewide initiative.

Developing a logic model at the beginning of program planning gives you a framework for charting the links between your program’s resources, activities, and outputs and its intended outcomes. It enables you to evaluate your program once it begins. And it helps you communicate to your stakeholders what you want to accomplish, how you intend to reach your goals, and how you will track your progress.

Planning a program requires time, effort, participation, and collaboration of stakeholders, including stu- dents, teachers, administrators, community members, policymakers, and program evaluators. Using logic models in this process can help practitioners and evaluators better understand a program’s mechanics and structure and chart a course toward improved policy and practice.

This guide is an installment in a four-part series on logic models and program planning and monitoring.1 The four guides introduce logic models to educators new to the concept and provide a refresher for edu- cators who are familiar with logic models but who have not recently developed one.

U . S . D e p a r t m e n t o f E d u c a t i o n At Mid-continent Research for Education and Learning

2

Defining the logic model components

Logic models connect your work to your expected outcomes—often expressed in changes in student perfor- mance and achievement. Included in most logic models are four components: resources, activities, outputs, and outcomes.

• Resources are the raw materials needed to create the program, implement its activities, and attain the desired outputs and outcomes. Sometimes called inputs, resources include both material items (such as curricula, instruction materials, facilities, and funding) and nonmaterial items (such as time, community support, and specialized knowledge and skills).

• Activities are the processes, actions, and events through which the program resources achieve the intended outcomes; they are the steps in implementing a program. Examples include collaborating with partners, developing training or curriculum materials, conducting training sessions or work- shops, and collecting and analyzing student performance data.

• Outputs are tangible, often process-oriented results or products typically expressed in numbers, such as number of students tested, number of teachers trained, and number of books read. While outputs provide information derived from the completion of program activities, they cannot indi- cate whether a change has occurred. For example, an output can tell you how many teachers attended training but not whether the training increased the teachers’ knowledge of the training topic.

• Short- and mid-term outcomes are the changes in program participants’ knowledge, beliefs, and behavior due to their involvement in the program. Outcomes can also be quick adjustments in organizational practices or system design. Short-term outcomes are observable almost immediately after participation; mid-term outcomes can take months or years to emerge and typically build toward long-term outcomes.

• Long-term outcomes, sometimes called impacts, are a program’s lasting influences. Like short- and mid-term outcomes, long-term outcomes can be changes in knowledge, beliefs, and behavior. Examples include higher student achievement scores, increased high school graduation rates, and greater college acceptance rates.

Outcomes are required to demonstrate a program’s success. Which outcomes are included in a logic model depends on the program’s goals. Some programs have only short-term goals and outcomes, while others have only long-term ones. And not all programs differentiate among short, mid-, and long term. For example, a logic model for a program that offers class field trips to local institutions, such as art galleries or museums, might have the outcome of increasing student awareness of and interest in local cultural, artistic, or histor- ical sites, without specifying whether the outcome will occur during, on completion of, or sometime after the field trip.

Developing your program’s logic model components will take practice and collaboration. Some programs will involve only a few resources, activities, outputs, and outcomes; others will involve many. The number of each should match your program’s complexity, and the specificity of each can vary. Important resources, activities, outputs, and outcomes might be identified separately (for example, for a program to conduct a workshop), and less important or similar ones might be clustered and displayed together (for example, “conduct interviews” and “administer a survey” might be grouped as “collect data on program satisfaction”).

Connecting the logic model components

Logic models show the connections linking a program’s resources, activities, outputs, and outcomes (figure 1). These connections depict how the program is expected to operate—how each component will influence another to attain the intended outcomes. The connections can guide your program from start

3

Figure 1. Sample logic model for a teacher training program on alternative reading strategies

Resources

Develop and provide teaching

guides and sample lessons

Conduct teacher

workshops

Number and type of guides and sample

lessons for each grade level

Number of participants per workshop and

total hours each participant

attended the workshop

Increased teacher

knowledge of multiple

instruction strategies to teach reading

Increased teacher

knowledge of reading content

Increased teacher use of

alternative strategies for

presenting reading content

Increased positive student attitudes toward

learning

Increased student

understanding of reading content

Increased student reading

test scores

Research-based guidance on

reading strategies

Curriculum coordinators

Elementary school teachers

Activities Outputs Short-term outcomes

Mid-term outcomes

Long-term outcomes

Source: Authors.

to finish. For example, an activity can influence an output, and that output can influence a short-term outcome. Making these connections can help you anticipate your program’s flow and can illustrate the interdependencies across components.

In figure 1, research-based guidance, curriculum coordinators, and elementary school teachers are the resources required to implement a teacher training program on alternative reading strategies. Research- based guidance and curriculum coordinators are necessary for developing and providing teaching guides and sample lessons. Curriculum coordinators and elementary school teachers are needed to conduct the teacher workshops. The outputs of these two activities are the number and type of teaching materials distributed for each grade level and the number of participants per workshop and the number of hours each participant attended the workshops. These outputs connect to two short-term outcomes: increased teacher knowledge of instruction strategies to teach reading and increased teacher knowledge of reading content. The short-term outcomes support the mid- and long-term outcomes: increased teacher use of alternative strategies for presenting reading content, increased positive student attitudes toward learning, increased student understanding of reading content, and finally increased student reading test scores.

When properly constructed, logic models reflect the elements of a program and convey the underlying theory of action—how the program’s resources, activities, and outputs lead to desired outcomes.

Note

1. This guide serves as a companion to the Education Logic Model computer-based application developed by Regional Educational Laboratory (REL) Pacific administered by Mid-continent Research for Educa- tion and Learning. The application is an interactive, computer-based tool designed to help educators develop logic models.

This guide, and others on logic models and program planning, are available for download from the REL Pacific website (http://relpacific.mcrel.org) or by emailing REL Pacific at [email protected]. The Education Logic Model application can be accessed from the REL Pacific website.

REL 2014–025

The National Center for Education Evaluation and Regional Assistance (NCEE) conducts unbi- ased large-scale evaluations of education programs and practices supported by federal funds; provides research-based technical assistance to educators and policymakers; and supports the synthesis and the widespread dissemination of the results of research and evaluation through- out the United States.

March 2014

This report was prepared for the Institute of Education Sciences (IES) under Contract ED-IES-C-12-0010 by Regional Educational Laboratory Pacific administered by Mid-continent Research for Education and Learning. The content of the publication does not necessarily reflect the views or policies of IES or the U.S. Department of Education nor does mention of trade names, commercial products, or organizations imply endorsement by the U.S. Government.

This REL report is in the public domain. While permission to reprint this publication is not neces- sary, it should be cited as:

Kekahio, W., Cicchinelli, L., Lawton, B., & Brandon, P. R. (2014). Logic models: A tool for effective program planning, collaboration, and monitoring. (REL 2014–025). Washington, DC: U.S. Depart- ment of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory Pacific. Retrieved from http://ies.ed.gov/ ncee/edlabs.

This report is available on the Regional Educational Laboratory website at http://ies.ed.gov/ ncee/edlabs.

The Journal of Educators Online-JEO July 2016 ISSN 1547-500X Vol 13 Number 2

135

Logic Models as a Way to Support Online Students and Their Projects

Jesse Strycker, East Carolina University, Greenville, North Carolina, USA

Abstract

As online enrollment continues to grow, students may need additional pedagogical

supports to increase their likelihood of success in online environments that don’t offer the same

supports as those found in face to face classrooms. Logic models are a way to provide such

support to students by helping to model project expectations, allowing students to demonstrate

their current and changing understanding of relationships for different projects and instructors to

review and comment on those understandings, and as a vehicle for reflecting on the outcomes of

a project.

Keywords: logic models, distance education, instructional scaffolding, reflective pedagogy

INTRODUCTION

Students taking online classes do so for a variety of reasons. Full-time work schedules

aren’t always compatible with campus-based programs featuring classes that meet during regular

work hours. Night classes will work for some students, but other mays need flexibility to have

different nights be when they work on their assignments. Physical distance may also make

campus attendance impractical. In these different instances, a distance education class or

program may be the only viable option.

The Journal of Educators Online-JEO July 2016 ISSN 1547-500X Vol 13 Number 2 136

While enrollment numbers have slowed over recent years, online enrollment continues to

grow (Allen and Seaman, 2015). Despite the continued growth in online courses, they aren’t

without their concerns, such as dropout rates that are higher than traditional classes (Roby, Ashe,

Singh, & Clark, 2013) and the possibility of outcomes being negatively affected (Allen &

Seaman, 2014). A challenge to online courses and the learners who take them can be the lack of

the kinds of support, instructional scaffolding that occur in face to face classrooms, as well as the

need for greater self-regulated learning by students online (Delen, Liew, & Wilson, 2014). One

way to provide online students with support and scaffolding, and one utilized by the author, are

logic models. The author’s use of logic models with online students draws conceptually from

schema theory and metacognition. Schemas are the prior knowledge that individuals have

accumulated over time, how that knowledge is organized, and have also been identified as the

plans that they can follow when approaching different situations (Wiseman, 2008). A logic

model provides a way to illustrate a student’s current understanding or schema of how their

project will be designed and executed. This physical representation also provides a vehicle for

their metacognition. Often summarized as thinking about thinking, metacognition “refers to the

processes used to plan, monitor, and assess one’s understanding and performance…includes a

critical awareness of a) one’s thinking and learning and b) oneself as thinker and learner” (Chick,

n.d. para. 1). Students’ metacognition provides a basis for illustrating their existing schema in the

form of a logic model for how they will carry out each project, assess their current plan, and later

reflect on their execution of the plan and how it may have varied from their model and what they

have learned as a result.

The Journal of Educators Online-JEO July 2016 ISSN 1547-500X Vol 13 Number 2 137

What is a logic model?

The W.K. Kellogg Foundation (2004) defines a logic model as a “systematic way to

present and share your understanding of the relationship among the resources you have to

operate your program, the activities you plan, and the changes or results you hope to achieve”

(p.1). You may also know logic models under different names since “common synonyms for

logic models include idea maps, frameworks, rich pictures, action, results or strategy maps, and

mental models” (Knowlton & Phillips, 2013, p. 4). While the Kellogg Foundation uses program

logic models as part of their grant funding decisions, logic models can be used with other kinds

of projects as well. Some of the projects that logic models have previously been used with

include public health programs (De-Regil, Pena-Rosas, Flores-Ayala, & del Socorro Jefferds,

2014), as a basis for evaluating change in vocational rehabilitation programs (Groomers, Jones,

& Lewis, 2014), developing program theory for a father support program (Gervais, Lacharité, &

Dubeau, 2015), and assessing effective practices of out of school programs (Wilkerson & Haden,

2014), just to name a few. Regardless of what they are used for, at a basic level a logic model is

used to identify relationships between inputs, outputs, and outcomes (W. K. Kellogg

Foundation). Such models can vary in level of detail from the basic (see Figure 1) to the more

advanced (see Figure 2)

Figure 1: Basic logic model with three elements and an informing context.

Figure 2: Advanced logic model with five elements and informing context.

The Journal of Educators Online-JEO July 2016 ISSN 1547-500X Vol 13 Number 2 138

Why use a logic model?

The use of a logic model requires students to think through an entire project or program

and how the different pieces fit together in a meaningful way. This holistic approach may lead to

discoveries of inputs that have been overlooked or identifying limited consideration of outputs,

outcomes, or impacts, if not in the short term, then in the intermediate and longer terms. These

kind of models are unique to each individual and are always changings as learners develop new

understandings and modify their old ones (McNeil, 2015). This evolutionary aspect of logic

models increases their utility for use with online class projects.

In an online course where the instructor cannot always be present to meet with students

for regular mentoring and scaffolding, logic models can be a way to help outline expectations

and demonstrate understanding. From the instructor perspective, a logic model provides structure

to students to know exactly which elements in a project they need to think about when planning

and designing a project. From a student perspective, they are able to demonstrate their

understanding in several ways. This back and forth interaction using the logic models as a basis

for discussion is an example of an instructional scaffold (Delen et al., 2014). At a basic level

students are able to fill in each portion of the model to meet the requirements, but on a more

advanced level they can demonstrate their understanding of a situation. This can be

accomplished by having a model that acknowledges their unique context and how it informs the

project, the different kinds of inputs they have available to them to work with, and the different

types of results they can expect and hope for in the short, intermediate, and longer terms. The

logic model can also serve as a map to help keep students on task when they are outside of class

as they have constructed a map of their project, identifying both starting and ending points.

The Journal of Educators Online-JEO July 2016 ISSN 1547-500X Vol 13 Number 2 139

The end of a project does not signify the end of a logic model’s usefulness. The use of

logic models also provides a built in basis for comparison when assessing a project and reflecting

on it. As noted earlier, these kinds of models evolve as students develop new knowledge

(McNeil, 2015). The reflecting on the logic models helps students to make more meaningful

learning from the experiences (Guthrie & McCracken, 2010) of each project that the logic

models informed. A student is able to review the logic model of how they intended for the

project to be carried out and its results versus how the project proceeded and its results. Through

reflection students may realize they actually had greater access to inputs then they were initially

aware. For example, many students will overlook the instructor as an input that has provides both

feedback and continue guidance and may also not consider colleagues on who they can rely at

their internship sites. They can ask themselves further questions, such as, if they overlooked

certain inputs, were unaware of certain inputs, or if they didn’t use the inputs as well as they

could have. Students can also consider how well the outputs, outcomes, and impacts matched

those they had identified. If they are unable to identify a fit, this time for reflection could also

lead to a restructuring of the logic model to allow the results to be more likely or to better inform

future logic models so that they are more accurate. This reflective practice can aid instructors in

assessing student learning in classes they may otherwise have limited interactions with through

an online class setting or with students with whom they may be unable to visit at sites where

their projects were executed.

How to use a logic model?

Using a logic model begins by considering its different parts. The context or situation that

is informing the model should first be identified as this will inform what the model is addressing.

Next the inputs should be considered, which can be broken down into activities and other types

The Journal of Educators Online-JEO July 2016 ISSN 1547-500X Vol 13 Number 2 140

of inputs depending on how it is being approached. General inputs are the resources available,

individuals contributing support in some way, and may also include the activities being utilized

to garner results. One consideration for thinking about activities separately is that only certain

inputs may apply for given activities. For example, one activity might use only certain resources

and individuals who do not cross over to support other activities.

Outputs are the results that are yielded by the activities and inputs, such as artifacts which

are generated and exercises or trainings completed. Outcomes can sometimes be less tangible as

there may not be immediate results. Instead, outcomes might be increased skill or confidence.

Impacts can be more nebulous as they are based on the outcomes. For example, a participant in a

professional development activity may need to produce an artifact to demonstrate what was

learned as part of the session. The artifact of that session could be a lesson plan that incorporates

a technology the participant knew little about before the session. The outcome for that participant

is she or he is now more confident in incorporating the technology learned about and will do it

more often (short term impact), which potentially results in an impact of the participant

becoming a teacher that uses more technology in her or his classroom and having students who

are more engaged (intermediate or longer term impacts).

While logic models can be prepared offline by students and feedback offered by

instructors using the grading functionality within a learning management system, another

technology can better support the evolving nature and scaffolding opportunities of logic models.

Through the use of a collaborate technology, such as GoogleDocs, both student and instructor

can pose questions to each other via inserted comments or using the built-in chat feature.

Collaborative editing also provides students and instructors with additional real time support

options. Finally, instructors are able to view changes to the models over time via the version

The Journal of Educators Online-JEO July 2016 ISSN 1547-500X Vol 13 Number 2 141

control options within GoogleDocs. The use of such a Web 2.0 tool can also be a way to increase

instructional presence (Tunks, 2012), helping to add more of the support that would more

regularly be found in a face to face class.

How might logic models be used in a specific online course?

In an online K-12 technology leadership internship class taught by the author, students

must identify a mentor or mentors and have them approved before discussing what projects they

might pursue and how they’ll go about them. Once they have this initial brainstorming session

completed they must put together a brief proposal and logic model for each of their projects. In

doing so they must not only consider the activities they wish to pursue, but also the inputs they

have going into the projects, the activities they’ll be undertaking, the outcomes they’re hoping

for, and the impacts that they plan for and hope will result from those outcomes.

In this particular class students must construct logic models for a minimum of one needs

assessment, one professional development session, and one community outreach project. They

are given the freedom to brainstorm with their mentor(s) on the focus of each of these projects,

but they then must be able to explain their reasoning within their proposal and logic models to

gain approval from the instructor. While the proposal language will include what the students are

hoping to accomplish, the logic models show the instructor a map of how the students plan to

reach their goals. The logic models also provide an opportunity for the instructor to identify

missing components, understand students’ current thinking, and identify areas that students may

not have considered or understand at that point. Student are also encouraged to regularly revisit

their logic models as changes occur at their internship sites so that they can continue to see how

changes affect not only the logic models, but also the different relationships and outcomes of the

projects.

The Journal of Educators Online-JEO July 2016 ISSN 1547-500X Vol 13 Number 2 142

Before students get to this point, they often must be introduced to what a logic model is

and how to construct one. Each semester the author has taught this internship class there have

only ever been two or three students who had heard of logic models and fewer still who had used

them. Students typically listed To-Do lists as their organizational method for such projects in the

past if they weren’t just planning everything out in their heads and following a simple timeline to

stay relatively on track. In a face to face class or one where the instructor is able to visit

internship sites on a regular basis, the site visits may help students to avoid simple To-Do lists or

spur of the moment thinking and allow discussion with students about their process and steps,

reviewing any notes that may exist as well. These casual, non-systematic approaches by students,

such as the planning