International Journal of Health Policy and Management

International Journal of Health Policy and Management

Developing a Framework for a Program Theory-Based Approach to Evaluating Policy Processes and Outcomes: Health in All Policies in South Australia

Document Type : Original Article

Authors
Angela Lawless 1
Fran Baum 2
Toni Delany-Crowe 2
Colin MacDougall 3
Carmel Williams 4
Dennis McDermott 5
Helen van Eyk 2
1 College of Nursing and Health Sciences, Flinders University, Adelaide, SA, Australia
2 Southgate Institute for Health Society and Equity, Flinders University, Adelaide, SA, Australia
3 College of Medicine and Public Health, Flinders University, Adelaide, SA, Australia
4 Department of Health and Ageing, Adelaide, SA, Australia
5 The Poche Centre for Indigenous Health and Well-being, Flinders University, Adelaide, SA, Australia
10.15171/ijhpm.2017.121
Abstract
Background
The importance of evaluating policy processes to achieve health equity is well recognised but such evaluation encounters methodological, theoretical and political challenges. This paper describes how a program theorybased evaluation framework can be developed and tested, using the example of an evaluation of the South Australian Health in All Policies (HiAP) initiative.

 
Methods
A framework of the theorised components and relationships of the HiAP initiative was produced to guide evaluation. The framework was the product of a collaborative, iterative process underpinned by a policy-research partnership and drew on social and political science theory and relevant policy literature.

 
Results
The process engaged key stakeholders to capture both HiAP specific and broader bureaucratic knowledge and was informed by a number of social and political science theories. The framework provides a basis for exploring the interactions between framework components and how they shape policy-making and public policy. It also enables an assessment of HiAP’s success in integrating health and equity considerations in policies, thereby laying a foundation for predicting the impacts of resulting policies.

 
Conclusion
The use of a program theory-based evaluation framework developed through a consultative process and informed by social and political science theory has accommodated the complexity of public policy-making. The framework allows for examination of HiAP processes and impacts, and for the tracking of contribution towards distal outcomes through the explicit articulation of the underpinning program theory.

Highlights

Commentaries Published on this Paper

  • From Mid-Level Policy Analysis to Macro-Level Political Economy; Comment on “Developing a Framework for a Program Theory-Based Approach to Evaluating Policy Processes and Outcomes: Health in All Policies in South Australia”

          Abstract | PDF

  • Ideas for Extending the Approach to Evaluating Health in All Policies in South Australia; Comment on “Developing a Framework for a Program Theory-Based Approach to Evaluating Policy Processes and Outcomes: Health in All Policies in South Australia”

          Abstract | PDF

  • How Do We Evaluate Health in All Policies?; Comment on “Developing a Framework for a Program Theory-Based Approach to Evaluating Policy Processes and Outcomes: Health in All Policies in South Australia”

          Abstract | PDF

  • Evaluating Health in All Policies; Comment on “Developing a Framework for a Program Theory-Based Approach to Evaluating Policy Processes and Outcomes: Health in All Policies in South Australia”

          Abstract | PDF

  • Policy, Theory, and Evaluation: Stop Mixing the Fruit Salad; Comment on “Developing a Framework for a Program Theory-Based Approach to Evaluating Policy Processes and Outcomes: Health in All Policies in South Australia”

          Abstract | PDF

  • Researching Healthy Public Policy: Navigating the ‘Black Box’ Means Thinking More About Power; Comment on “Developing a Framework for a Program Theory-Based Approach to Evaluating Policy Processes and Outcomes: Health in All Policies in South Australia”

Abstract | PDF

 

Authors' Response to the Commentaries

  • Power, Process and Context in Theory Based Evaluation of Policy Implementation: A Response to Recent Commentaries

          Abstract | PDF

Keywords
Healthy Public Policy
Evaluation
Inter-Sectoral Action
Health Equity
Social Determinants
Subjects
  1. Walt G, Shiffman J, Schneider H, Murray SF, Brugha R, Gilson L. 'Doing' health policy analysis: methodological and conceptual reflections and challenges. Health Policy Plan. 2008;23(5):308-317. doi:10.1093/heapol/czn024
  2. Sabatier P. The Need for Better Theories. In: Sabatier P, ed. Theories of the Policy Process. Colorado: Westview Press; 2007:3-20.
  3. de Leeuw E, Clavier C, Breton E. Health policy--why research it and how: health political science. Health Res Policy Syst. 2014;12:55. doi:10.1186/1478-4505-12-55
  4. Colebatch H, Hoppe R, Noordegraaf M. Understanding Policy Work. In: Colebatch H, Hoppe R, Noordegraaf M, ed. Working for Policy. Amsterdam: Amsterdam University Press; 2010:11-25.
  5. Carey G. Re-Conceptualising Public Health Interventions in Government: A Response to Recent Commentaries. Int J Health Policy Manag. 2016;5(9):569-570. doi:10.15171/ijhpm.2016.91
  6. Colebatch H, Hoppe R, Noordegraaf M. The Lessons for Policy Work. Working for Policy. Amsterdam: Amsterdam University Press; 2010:227-243.
  7. Kokkinen L, Shankardass K, O'Campo P, Muntaner C. Taking health into account in all policies: raising and keeping health equity high on the political agenda. J Epidemiol Community Health. 2017;71(8):745-746. doi:10.1136/jech-2016-207736
  8. Farquharson K. A Different Kind of Snowball: Identifying Key Policymakers. Int J Soc Res Methodol. 2005;8(4):345-353. doi:10.1080/1364557042000203116
  9. Signal LN, Bowers SG, Edwards R, et al. Process, pitfalls and profits: lessons from interviewing New Zealand policy-makers. Health Promot Int. 2016. doi:10.1093/heapro/daw065
  10. Donaldson SI. Program Theory-Driven Evaluation Science: Strategies and Applications. New York: Psychology Press, Taylor and Francis Group; 2007.
  11. Mayne J. Contribution analysis: An approach to exploring cause and effect. ILAC Brief. http://www.betterevaluation.org/sites/default/files/ILAC_Brief16_Contribution_Analysis.pdf.  Published2008.
  12. Poland B, Frohlich K, Cargo M. Context as a Fundamental Dimension of Health Promotion Program Evaluation. Health Promotion Practices in the Americas. New York: Springer Science+Business Media; 2008:299-317.
  13. Blamey A, Mackenzie M. Theories of change and realistic evaluation peas in a pod or apples and oranges? Evaluation. 2007;13(4):439-455. doi:10.1177/1356389007082129
  14. Carey G, Friel S. Understanding the role of public administration in implementing action on the social determinants of health and health inequities. Int J Health Policy Manag. 2015;4(12):795-798. doi:10.15171/ijhpm.2015.185
  15. Freiler A, Muntaner C, Shankardass K, et al. Glossary for the implementation of Health in All Policies (HiAP). J Epidemiol Community Health. 2013;67(12):1068-1072. doi:10.1136/jech-2013-202731
  16. Hendriks AM, Kremers SP, Gubbels JS, Raat H, de Vries NK, Jansen MW. Towards health in all policies for childhood obesity prevention. J Obes. 2013;2013:632540. doi:10.1155/2013/632540
  17. Weiss CH. How can theory-based evaluation make greater headway? Eval Rev. 1997;21(4):501-524. doi:10.1177/0193841x9702100405
  18. Mayne J. Contribution analysis: coming of age? Evaluation. 2012;18(3):270-280. doi:10.1177/1356389012451663
  19. Delahais T, Toulemonde J. Applying contribution analysis: Lessons from five years of practice. Evaluation. 2012;18(3):281-293. doi:10.1177/1356389012450810
  20. Funnell S, Rogers P. Purposeful Program Theory: Effective use of the theories of change and logic models. San Francisco: John Wiley Sons; 2011.
  21. Hunt J, Shackley S. Reconceiving science and policy: academic, fiducial and bureaucratic knowledge. Minerva. 1999;37(2):141-164. doi:10.1023/A:1004696104081
  22. Considine M. Making Public Policy: Institutions, Actors, Strategies. Cambridge: Polity Press; 2005.
  23. Government of South Australia. South Australia’s Strategic Plan, Creating opportunity. Adelaide: Government of South Australia; 2010.
  24. Lawless AP, Williams C, Hurley C, Wildgoose D, Sawford A, Kickbusch I. Health in All Policies: evaluating the South Australian approach to intersectoral action for health. Can J Public Health. 2012;103(7 Suppl 1):eS15-eS19.
  25. Government of South Australia. The South Australian approach to Health in All Policies: background and practical guide, Version 2. http://www.sahealth.sa.gov.au/wps/wcm/connect/cb6fa18043aece9fb510fded1a914d95/HiAPBackgroundPracticalGuide-v2.pdf?MOD=AJPERES&CACHEID=cb6fa18043aece9fb510fded1a914d95.  Accessed  January 9, 2013. Published 2011.
  26. World Health Organization. Health in all policies: Helsinki statement. Framework for country action. Geneva: WHO; 2014.
  27. Ståhl T, Wismar M, Ollila E, Lahtinen E, Leppo K. Health in All Policies: Prospects and Potentials. Finland: European Observatory on Health Systems and Policy; 2006.
  28. Baum F, Lawless A, Delany T, et al. Evaluation of Health in All Policies: concept, theory and application. Health Promot Int. 2014;29 Suppl 1:i130-142. doi:10.1093/heapro/dau032
  29. Carey G, Crammond B, Keast R. Creating change in government to address the social determinants of health: how can efforts be improved? BMC Public Health. 2014;14:1087. doi:10.1186/1471-2458-14-1087
  30. Donaldson SI, Lipsey MW. Roles for theory in contemporary evaluation practice: Developing practical knowledge. In: Shaw IF, Greene JC, Mark MM. The SAGE Handbook of Evaluation.SAJE;2006.
  31. Judge K, Bauld L. Strong theory, flexible methods: Evaluating complex community-based initiatives. Crit Public Health. 2001;11(1):19-38. doi:10.1080/09581590010028237
  32. Stame N. Theory-based evaluation and types of complexity. Evaluation. 2004;10(1):58-76. doi:10.1177/1356389004043135
  33. Leeuw F, Vaessen J. Mind the gap: perspectives on policy evaluation and the social sciences. Paper presented at: Transaction Publishers; 2009.
  34. Rogers PJ. Using Programme Theory to Evaluate Complicated and Complex Aspects of Interventions. Evaluation. 2008;14(1):29-48. doi:10.1177/1356389007084674
  35. Rogers PJ, Weiss CH. Theory‐based evaluation: Reflections ten years on: Theory‐based evaluation: Past, present, and future. New Directions for Evaluation. 2007;(114):63-81. doi:10.1002/ev.225
  36. Weiss CH. Which links in which theories shall we evaluate? New Directions for Evaluation. 2000;(87):35-45. doi:10.1002/ev.1180
  37. Lawless A, Freeman T, Bentley M, Baum F, Jolley G. Developing a good practice model to evaluate the effectiveness of comprehensive primary health care in local communities. BMC Fam Pract. 2014;15:99. doi:10.1186/1471-2296-15-99
  38. Birckmayer JD, Weiss CH. Theory-based evaluation in practice. What do we learn? Eval Rev. 2000;24(4):407-431. doi:10.1177/0193841x0002400404
  39. Quigley R. Role of Health Impact Assessment in Health in All Policies  In: Kickbusch I, Buckett K. Implementing Health in All Polcies. Adelaide: Government of South Australia; 2010:101-107.
  40. Delany T, Harris P, Williams C, et al. Health Impact Assessment in New South Wales & Health in All Policies in South Australia: differences, similarities and connections. BMC Public Health. 2014;14:699. doi:10.1186/1471-2458-14-699
  41. Mannheimer LN, Lehto J, Ostlin P. Window of opportunity for intersectoral health policy in Sweden--open, half-open or half-shut? Health Promot Int. 2007;22(4):307-315. doi:10.1093/heapro/dam028
  42. Ollila E. Health in All Policies: from rhetoric to action. Scand J Public Health. 2011;39(6 Suppl):11-18. doi:10.1177/1403494810379895
  43. Exworthy M. Policy to tackle the social determinants of health: using conceptual models to understand the policy process. Health Policy Plan. 2008;23(5):318-327. doi:10.1093/heapol/czn022
  44. Ezzy D. Qualitative analysis: Practice and Innovation. Crows Nest, NSW: Allen Unwin; 2002.
  45. Pollitt C. Context: What kind of missing link? In: Pollitt C, ed. Context in Public Policy and Management - The missing link? Cheltenham: Edward Elgar; 2013:415-422.
  46. Clark J. Contexts: forms of agency and action. In: Pollitt C, ed. Context in Public Policy and Management: The Missing Link? Cheltenham: Edward Elgar; 2013:22-34.
  47. Williams C, Lawless A, Parkes H. The South Australian Health in All Policies model: The developmental phase. Public Health Bulletin South Australia. 2008;5(1):30-34.
  48. Sabatier PA. An advocacy coalition framework of policy change and the role of policy-oriented learning therein. Policy Sci. 1988;21(2-3):129-168. doi:10.1007/bf00136406
  49. Newman L, Ludford I, Williams C, Herriot M. Applying Health in All Policies to obesity in South Australia. Health Promot Int. 2016;31(1):44-58. doi:10.1093/heapro/dau064
  50. Noordegraaf M. Academic Accounts of Policy Experience. In: Colebatch H, Hoppe R, Noordegraaf M, eds. Working for Policy. Amsterdam: Amsterdam University Press; 2010:45-67.
  51. Sabatier P, Jenkins-Smith HC. Policy Change and Learning: An Advocacy Coalition Approach. Bolder, CO: Westview Press; 1993.
  52. Jenkins-Smith HC, Nohrstedt D, Weible CM, Sabatier PA. The advocacy coalition framework: Foundations, evolution, and ongoing research. In: Sabatier PA. Theories of the policy process. Boulder, CO: Westview Press;2014:3.
  53. Howlett M, Ramesh M, Perl A. Studying Public Policy: Policy Cycles and Policy Subsystems. Oxford: Oxford University Press; 2009.
  54. Lowndes V, Roberts M. Why Institutions Matter - The new Institutionalism in Policitical Science. New York: Palgrave Macmillan; 2013.
  55. Scott WR. Institutions and organizations: Ideas, interests, and identities. Los Angeles: SAGE Publications; 2013.
  56. Marsh D. Comparing policy networks. Buckingham, UK: Open University Press; 1998.
  57. Kingdon J. Agendas, Alternatives and Public Policies. 2nd ed. Boston: Longman; 2011.
  58. Giddens A. The constitution of society: Outline of the theory of structuration. University of California Press; 1984.
  59. Giddens A. Modernity and self-identity: Self and society in the late modern age. Stanford University Press; 1991.
  60. Solar O, Irwin A. A conceptual framework for action on the social determinants of health. Geneva: World Health Organization; 2010.
  61. Delany T, Lawless A, Baum F, et al. Health in All Policies in South Australia: what has supported early implementation? Health Promot Int. 2016;31(4):888-898. doi:10.1093/heapro/dav084
  62. Dwyer J, Silburn K, Wilson G. National Strategies for Improving Indigenous Health and Health Care. Canberra: Office for Aboriginal and Torres Strait Islander Health; 2004.
  63. Greenhalgh T, Humphrey C, Hughes J, Macfarlane F, Butler C, Pawson R. How do you modernize a health service? A realist evaluation of whole-scale transformation in london. Milbank Q. 2009;87(2):391-416. doi:10.1111/j.1468-0009.2009.00562.x
  64. Pawson R, Greenhalgh T, Harvey G, Walshe K. Realist synthesis: an introduction. ESRC Research Methods Programme. https://pdfs.semanticscholar.org/4351/46e6e6617491ff1c4b32b76e0a534c86d6c7.pdf?_ga=2.34585889.1692440165.1505184365-112133799.1505184365.  Published 2004.
  65. Stake R. The Case Study Method in Social Inquiry. In: Gomm R, Hammersley M, Foster P, eds. Case Study Method: Key Issues, Key Texts. London: SAGE Publications; 2000:19-26.
  66. Lincoln YS, Guba EG. The Only Generalization is: There is No Generalization. In: Gomm R, Hammersley M, Foster P, eds. Case Study Method: Key Issues, Key Texts. London: SAGE publications; 2000:27-44.
International Journal of Health Policy and Management
Volume 7, Issue 6
2018
Pages 510-521

  • Received Date 01 December 2016
  • Revised Date 05 October 2017
  • Accepted Date 07 October 2017
  • First Published Date 01 June 2018
  • Published Date 01 June 2018