Canadian Evaluation Society Conference – Day 3
Today is the last day of the Canadian Evaluation Society conference in Victoria. And, as I did yesterday and the day before, I’m using my blog as a dumping ground for my conference notes. For my regular blog readers, again please feel free to ignore these postings and I really, really promise to blog about whatever shiny thing happens to catch my attention again soon.
Addressing challenges in tobacco control strategy evaluation
- complexity
- multiplicity of goals, multiplicity of partners, multiplicity of interventions
- interactions among interventions
- expectations of synergies
- nonlinearity and feedback loops
- tipping points (e.g., if we got the % of the population who smoke below a certain point, it will change the climate – e.g., may make it possible to implement policies that couldn’t be implemented before
- challenges:
- program evaluators usually trained in evaluating a single program intervention
- determining population level outcomes (paucity of good data)
- obtaining data on resources, inputs and outcomes
- biggest challenge: attribution of population level outcomes to micro-level interventions
- classic approaches to complex strategy evaluation include things like comparing communities (e.g., using RCTs or quasi-experimental designs) or comparing regions/states/countries
- critiques of classic approaches:
- black box on final outcomes
- lack of attention to synergies (what mixes? in what sequence?)
- lack of attention to feedback loops
- lack of attention to multiplier effects
- little information to inform strategies
- approaches that help:
- thematic evaluation
- cluster evaluation
- contribution analysis
- complex evaluation strategies are needed to evaluate complex strategies
- complex evaluation strategy:
- evaluate each of micro, meso and macro levels
- need knowledge exchange (KE) that includes all stakeholders
- takes time and money
- path logic model:
- an innovative technique for helping understand if and how complex strategies are meeting their goals
- helps “tell the story” to policymakers
- useful for identifying the needs for further evaluative information
- stages in comprehensive evaluation
- identify high level macro outcomes
- determine key evidence-based paths to achieving outcomes
- identify relevant interventions
- assess the expected contributions of each path through literature synthesis
- assess the actual contributions of each path through evaluative information synthesis/contribution analysis
- assess interactions and synergies
- can look at which path(s) a given intervention links to
- can also look at which intervention(s) a given path links to
- e.g., if very few lines coming from a given path, it suggests there is a gap in interventions that address that path; if there are lots of line, it suggests redundancy (which may or may not be a good thing)
- all of this takes a long time
- also, strategies can evolve during the intervening period
Action Items From the Conference
- read up on “contribution analysis” by John Mayne [note: http://www.oag-bvg.gc.ca/internet/docs/99dp1_e.pdf]
- check out the dates of the AEA conference [note: November 10-13 in San Antonio, Texas] and European Evaluation Society [note: October 6-8 in Prague]
- get a copy of the Treasury Board’s 2009 policy on evaluation
- write my personal philosophy of evaluation statement (like a teaching philosophy). [This wasn’t talked about specifically at the conference, but I did a lot of thinking about how I approach evaluation and got to thinking that I should write a statement)
Comments |0|
This site uses Akismet to reduce spam. Learn how your comment data is processed.