Combine logic models with the framework method to streamline your next evaluation
04 September 2015 - IN evaluation, for, framework, logic, method, methods, model, NVivo, Windows
As an evaluation practitioner, you probably know your way around a logic model:
Basically, a logic model is a systematic and visual way to present and share your understanding of the relationships among the resources you have to operate your program, the activities you plan, and the changes or results you hope to achieve. (W.K. Kellogg Foundation Logic Model Development Guide 2004 page 1)
But you may be less sure about how you can use a logic model (and framework matrix) to structure your qualitative data in NVivo.
Late in 2013, Gareth Morrell from NatCen presented this invaluable eSeminar - Improving mixed-method evaluations by incorporating logic models into NVivo.
It describes how logic models combined with the framework method can help you to deliver robust, transparent and systematic evaluations.
Think of this blog post as a 'cheat-sheet' for working through the process.
Creating the logic model in NVivo
To start with, lets look at how you might create the logic model.
On the Explore tab of the ribbon, click New Model - name the model and click OK. Now, just add the shapes you need (select them from the ribbon) and double-click a shape to label it.
Your model might start out looking something like this:
Then, in collaboration with stakeholders, practitioners and policy makers you can add shapes and flesh-out the model:
Going forward, the logic model can inform your data collection strategy as well as providing a common reference point for planning and implementation.
Once you've gathered the qualitative data (interviews, observations, open-ended survey questions and so on), you can use your logic model to develop a framework for analyzing it.
In NVivo, the framework is a hierarchy of descriptive labels called nodes.
Convert model shapes into nodes
To speed things up, you can convert your model shapes into nodes - just select the shapes and then click Convert to, and then click Convert to New Project Item.
This creates a flat list of nodes but you can easily drag and drop to organize them in a hierarchy. Or you can be super efficient and convert the top-level shapes first (resources, activities, outputs, outcomes, impacts) and then convert the child shapes and save them under the parent node.
Now, you should have a node hierarchy that reflects your logic model:
Of course, the hierarchy of themes is not set in stone - but it's a useful starting point. Your node hierarchy can evolve with your understanding of the data.
"It's not defining what we find but it's defining the areas in which we find things..." Gareth Morrell
Code and/or summarize your qualitative data
As you read through the qualitative material (that you've imported into NVivo) you can choose to code (or tag) the content at your theme nodes. If you're pressed for time or have large sets of data - you can use auto coding features to speed this up.
An added bonus of coding is that you can continue to refer back to the model for a bird's eye view and then drill down to see all the material about a given topic:
If you want to summarize your qualitative data - without coding it first - that's fine too.
Creating cases for participants and stakeholders
So, you have the descriptive framework for analyzing your qualitative data, but you'll also want to look at your themes through the eyes of particular cases or stakeholders.
A framework matrix gives you an effective and condensed way to do this.
First, you'll need to make a case node for each participant - this is how you gather everything a particular person has said and stamp it with demographic info like age, gender, position and so on (check this video for details).
Creating a framework matrix
Once you have case nodes and theme nodes in place, you can create a new framework matrix:
On the Create tab of the ribbon, click Framework Matrix. Give it a name and then specify the rows and columns. Put your cases in the rows, and your descriptive theme nodes in the columns.
When you click OK, the framework matrix is opened in Detail View. You can read through each case and type your summaries under the theme headings.
If you've already coded the content - you can use the auto summarize feature to populate the cells.
Your framework matrix might start to look something like this:
Once you've summarized the data, you can easily scan the rows and columns to see all the content by a particular case, or all the content about a particular theme.
For example, what are the impacts around community involvement? Did it lead to a more engaged society as suggested by the logic model? Don't suffer in suspense, find out in Gareth's eSeminar!
As a practitioner, you want to deliver robust evaluations and actionable recommendations for programme improvement - using a logic model alongside the framework method can help you get there.
We'd love to hear about your experiences with logic models and/or the framework method - because I suspect this post is just the tip of the iceberg.