Using NVivo to enhance transparency in a realist evaluation
Sonia Dalkin, Natalie Forster, Philip Hodgson, Monique Lhussier and Sue Carr are researchers at Northumbria University, Faculty of Health and Life Sciences, Department of Public Health and Wellbeing. They recently collaborated on a project supported by The Centre of Translational Research in Public Health, FUSE, to evaluate how activities performed by the Citizen's Advice Bureau impact public health. The research is funded by The National Institute for Health's (NIHR) School for Public Health Research.
In this post, Sonia Dalkin and Natalie Forster describe the team's approach and assess the benefits of using NVivo to keep an audit trail of theory development.
Firstly, what is realist evaluation?
Realist evaluation is gaining momentum in the evaluation of complex social interventions (1).
It focuses on ‘what works, how, in which conditions and for whom’ using context, mechanism and outcome configurations as opposed to asking whether an intervention ‘works’ (2).
‘Hunches’ or ‘initial programme theories’ about how the intervention works are developed which are tested and refined using data. To our knowledge, there is no published literature focusing solely on the use of NVivo in realist evaluation, although there are some brief descriptions (e.g. 3, 4, 5).
Why use a computer assisted qualitative data analysis software like NVivo in a realist evaluation?
Computer assisted qualitative data analysis software such as NVivo have been used as an aid to data analysis in qualitative research in several methodological fields (6-8).
Researchers have found using the programme challenging but valuable in terms of advancing the robustness of qualitative research (8). There have been calls for more transparency in realist methods which the use of NVivo could address:
“Computer assisted qualitative data analysis software (CAQDAS) has been seen as aiding the researcher in the search for an accurate and transparent picture of the data whilst also providing an audit of the data analysis process as a whole—something which has often been missing in accounts of qualitative research” (9)
How have we used NVivo?
Here's how we approached analysis:
- We created a node for each ‘hunch’ we had about how the intervention worked.
- We created a linked memo to each node. In this memo we developed the hunch into an initial programme theory (a hunch about how the intervention worked, expressed in terms of context, mechanism and outcome).
- Under each node we created four child nodes: literature, interviews to develop theory, staff interviews to test theory, client interviews to test theory
- We imported literature and interview data and coded it to the ‘literature’ and ‘interviews to develop theory’ nodes. Once we were satisfied with our initial programme theories, we began to test them using interview data, which was coded to the appropriate child node (staff/client interviews to test theory).
- We selected the ‘aggregate coding from child nodes’ function which meant that the main node (for each initial programme theory) now stored information from the child nodes (literature, interviews to develop theory and client/staff interviews to test theory), giving us the option of examining all information for a theory, or only that from specific sources (e.g. client versus staff interviews).
The infographic below provides more information about the project and illustrates the coding strategy:
What have the advantages been?
Use of NVivo in a realist evaluation has:
- Aided transparency in decision making
- Facilitated team working
- Organised literature and data
What were the challenges?
A couple of challenges we faced included:
- The time required to become familiar with NVivo
- Not having shared access across the team to the NVivo file (due to institutional regulations).
Thoughts for future use of NVivo with realist evaluation
As our research continues, we’re developing our understanding of how NVivo can be used in a realist evaluation. For example, how should we integrate more formal, abstract theory? Also, since realist evaluation often requires mixed methods, how should we integrate quantitative work into NVivo? We are still learning but so far NVivo has proven to be a useful tool which we would encourage others to engage with.
1. Dalkin S, Greenhalgh G, Jones D, Cunningham B, Lhussier M. What's in a mechanism? Development of a key concept,in realist evaluation Implementation Science 2015;10(49).
2. Pawson R, Tilley N. Realistic Evaluation. London Sage; 1997.
3. Marchal B, McDamien D, Kegels G. A realist evaluation of the management of a wellperforming regional hospital in Ghana. BMC Health Services Research 2010;10(24).
4. Maluka S, Kamuzora P, SanSebastián M, Byskov J, Ndawi B, Olsen E, et al. Implementing accountability for reasonableness framework at district level in Tanzania: a realist evaluation. Implementation Science. 2011;6(11).
5. Douglas F, Gray D, van Teijlingen E. Using a realist approach to evaluate smoking cessation interventions targeting pregnant women and young people. BMC Health Services Research. 2010;10(49).
6. Bringer J, Johnston L, Brackenridge C. Maximizing transparency in a doctoral thesis1: the complexities of writing about the use of QSR*NVIVO within a grounded theory study. Qualitative Research. 2004;4(2):247-65.
7. Flowers P, McGregor Davis M, Larkin M, Church S, Marriott C. Understanding the impact of HIV diagnosis amongst gay men in Scotland: An interpretative phenomenological analysis. Psychology & Health. 2011;26(10):1378-91.
8. Bergin M. NVivo 8 and consistency in data analysis: reflecting on the use of a qualitative data analysis program. Nurse Researcher. 2011;18(3):6-12.
9. Welsh E. Dealing with Data: Using NVivo in the Qualitative Data Analysis Process. Forum: Qualitative Social Research. 2002;2(26).
Sonia Dalkin is a Lecturer in Public Health and Wellbeing in the Faculty of Health & Life Sciences at the University of Northumbria, England. She is also a member of Fuse (the centre for translational research in public health). Sonia has specific interests in palliative care, health inequalities and complex interventions. These subjects require a diverse methodology with an emphasis on mixed methods, realist and soft systems approaches.