The IMPACT Center approaches optimizing evidence-based intervention implementation as a three-stage process. Within each stage, multiple methods are used to move the intervention through to the next stage of optimization. Below, the limitations of methods currently used by the field of implementation science are outlined and the new methods that IMPACT will employ are detailed. Each new method will be applied across IMPACT-funded studies, refined each year, and built into widely available trainings and toolkits.

Provide researchers and implementers with robust, efficient methods for determinant identification and prioritization enables the precise targeting of high-priority problems.

Use causal pathway diagrams to show relationships between strategies, mechanisms, moderators, and outcomes, clarify how strategies function, facilitate effective matching to determinants, and identify conditions affecting strategy success.

Support rapid testing in analog (artificially generated experimental conditions) or real-world conditions by testing causal pathways to maximize the accumulation and use of knowledge across projects.

This stage uses methods to identify determinants of implementation success that are active in the specific implementation setting of concern. Strategies not well-matched to high-priority determinants operating in the implementation setting are unlikely to be effective. However, the existing methods for identifying and prioritizing determinants have at least four limitations:

1. They typically do not consider relevant determinants identified in the literature;
2. They are subject to issues of recall, bias, and social desirability;
3. They do not sufficiently engage the end user in the EBI prior to assessment;
4. Determinant prioritization typically relies on collaborator ratings of perceived qualities like feasibility to address determinants, which may have little to do with impact or being essential for success.

Rapid evidence reviews are used to summarize and synthesize research literature on known determinants for implementing evidence-based interventions in settings of interest. Unlike traditional systematic reviews, which can take more than a year, rapid reviews can be completed in three months or fewer.

Rapid evidence reviews are increasingly used to guide implementation in healthcare settings, but they are not often employed to identify implementation determinants.

Research Program Core experts collaborate with project leads and the practice partners to clarify the question and scope of each review. Data abstraction focuses on identified determinants and any information about timing (i.e., implementation phase), modifiability, frequency, duration, and prevalence.

This process results in a list of determinants organized by consumer, provider, team, organization, system, or policy level that is used to inform observational checklists and interview guid es for rapid ethnographic assessment.

Rapid ethnographic assessment is used to efficiently gather data about determinants by seeking to understand the people, tasks, and environments involved from community collaborator perspectives. This is achieved primarily by engaging these collaborators as active participants and applying user-centered approaches to efficiently elicit information.

Ethnographic observation includes semi-structured observations, including shadowing intended or actual evidence-based intervention users, which generates evidence that can be used to offset self-report biases. Through combined written and audio-recorded field notes, researchers document activities, interactions, and events (including duration, time, and location); note the setting’s physical layout; and map flows of people, work, and communication.

For a range of experiences, ethnographic interviews are informal during observation and yet formal through scheduled interactions with key informants. Interviews are unstructured, descriptive, and ask task-related questions. Researchers document the occurrence or presence of barriers, noting the duration, time, location, and affected persons or processes.

This method captures new and different perspectives from those captured during observations and interviews. Design probes are user-centered research toolkits that utilize items such as disposable cameras, albums, and illustrated cards. End users are prompted to take pictures, make diary entries, draw maps, or make collages in response to tasks such as “Describe a typical day” or “Describe using [the evidence-based intervention]”.

Participants have one week to observe, reflect on, and report experiences to generate insights, reveal ideas, and illuminate their lived experiences as they relate to implementing the evidence-based intervention (e.g., feelings, attitudes). In follow-up interviews, participants reflect on their engagement with the task.

Through memo writing, research team members analyze the data generated from design probes and interviews to identify new determinants, to corroborate determinants discovered via rapid evidence assessment, and to describe the meaning and importance of determinants to end users.

This method helps overcome the limitation of assessing collaborator perceptions in a vacuum that can happen when only interview and observation data is considered.

Criticality: How a determinant affects or likely affects an implementation outcome. Some determinants are prerequisites for outcomes (e.g., awareness of the evidence-based intervention). The influence of other determinants on outcomes depends on their potency (e.g., strength of negative attitudes).

Chronicity: How frequently a determinant event occurs (e.g., shortages of critical supplies) or how long a determinant state persists (e.g., unsupportive leadership).

Ubiquity: How pervasive a determinant is.

For each identified determinant, granular data is generated by the rapid evidence review, rapid ethnographic assessment, and design probes and organized in a table by these three criteria (criticality , chronicity, and ubiquity). The data is then independently rated by three researchers using a 4-point Likert scale from 0 to 3 (e.g., not at all critical, somewhat critical, critical, necessary) and three practice partners or community members. Priority scores and inter-rater agreement are calculated. The outcome is a list of determinants ordered by priority scores.

Drawing on Agile Science, the IMPACT Center is developing methods to create causal pathway diagrams that represent the best available evidence and hypotheses about mechanisms by which implementation strategies impact target determinants and downstream implementation outcomes. IMPACTS’s Research Program Core supports study leads to develop causal pathway diagrams, which will serve as an organizing structure of our relational database for accumulating knowledge.

A causal pathway diagram includes several key factors:

• The implementation strategy intended to influence the target determinant 
• A mechanism by which the strategy is hypothesized to affect the determinant
• The prioritized target determinant
• The observable proximal outcomes for testing mechanism activation and any precursors to implementation outcomes
• Preconditions for the mechanism to be activated and to affect outcome(s)
• Moderators (intrapersonal, interpersonal, organizational, etc.) that are hypothesized to impede strategy impact
• Any implementation outcomes that should be altered by determinant changes

Drawing on MOST and other underused experimental methods, we are developing guidelines for selecting experimental designs that can efficiently answer key questions at different stages of implementation research and obtain the right level of evidence needed for the primary research question. We prioritize signal testing of individual strategies to identify most promising forms and studies for optimizing blended strategies before testing them in a full-scale confirmatory RCT. Drawing on user-centered design, we are refining methods for ideation and low-fidelity prototyping to help researchers consider a broader range of alternatives for how an implementation strategy can be operationalized, which enables efficient testing of multiple versions and selection of the version that is most likely to balance effectiveness and burden or cost.

While randomized controlled trials provide robust evidence for strategy effectiveness, they do not provide a way to efficiently and rigorously test implementation strategy components. Faced with a similar problem in behavioral intervention science, MOST was developed to help behavioral scientists use a broader range of experimental designs to optimize interventions. The IMPACT Center is leveraging these designs to optimize strategies, including: factorial experiments, microrandomized trials, sequential multiple assignment randomized trials (SMARTs), and single case experimental designs. These designs, are highly efficient, requiring far fewer participants to test strategy components than a traditional RCT, enabling a range of research questions to be answered in less time and with fewer resources.

Ideation is the process of creating many versions of an idea. Ideation strategies include sketching, brainstorming, and timed idea generation in response to a prompt. Low-fidelity prototyping is rapid prototyping of design elements for early feedback from users, so designers can iteratively refine designs. Evidence from human-computer interaction shows that for some tasks, like strategy optimization, parallel prototyping—creating multiple designs in parallel vs. starting with a single design and iteratively refining it—is more efficient and results in higher quality designs. Methods for ideation and low-fidelity prototyping can help practice partners and researchers consider a broad range of alternatives for operationalizing a strategy.

Also known as RAMs, rapid analog methods create artificially generated situations for efficient, economical testing of strategies and mechanisms in conditions comparable to the real world. Analog studies involve an artificial representation of stimuli or occur in comparable, manufactured settings. Because you can isolate pathways of effects and limit burden on community partners, RAMs are ideal when testing a new strategy. These methods provide a signal of whether the implementation strategy is operating via the hypothesized mechanisms.

This type of design is best for optimizing complex strategies because they efficiently screen multiple components for an effect on target outcomes. Each component is a “factor” that can take several “levels” (e.g., yes vs. no; delivery source). Participants are randomized to cells corresponding to different combinations of levels of each factor allowing for analysis of main effects and interactions with fewer participants compared to randomized controlled trials.

MRTs evaluate implementation strategy components that are delivered repeatedly (e.g., automated reminders). Each time (called a “decision point”) that a component can be delivered (e.g., patient visit), provision or non-provision of the component is randomized, allowing for multiple components to be randomized concurrently. Microrandomized trials are a highly efficient design that takes advantage of both within-subject and between-subject comparisons in order to estimate marginal main effects, changes in component effect over time, and moderating effects.

SMARTs optimize adaptive strategies and help researchers determine decision rules for delivering a sequence of strategies that satisfy a set of optimization criteria, usually effectiveness and cost. Participants are initially randomized to two strategies that differ in intensity or cost and at predetermined times nonresponders are re-randomized to another set of strategy options; this can occur multiple times. SMARTs are highly efficient because analyses can use different sample subsets to answer different research questions e.g., differences between strategies and the optimal way to support nonresponders.

SCEDs gather evidence about strategy effects by observing changes in outcomes of interest for each participant (or unit, e.g., clinic). SCEDs are inherently within-subject designs with participants acting as their own controls. This is achieved through sequencing strategy exposures and comparing outcomes for periods when a participant was exposed to those when no strategy was provided. Several SCED designs exist, such as ABAB and multiple baseline. SCEDs require as few as six participants to provide information about effects, making it highly efficient with the low participant requirement. These features are particularly useful for preliminary implementation studies in a single clinic.