We first began to develop our definitions of Implementation Research in the assessment of NIH-funded research between 2008 and 2013. There were further refined through our systematic review seeking to categorize the primary objective of every research study focused on the CHAT+. These definitions may continue to evolve as this review is completed and begins to include secondary objectives.  More information about the specific milestones (e.g., from Clinical to Implementation  Research, and from Implementation to Other Applied Research) and the other core components of a Research Roadmap are available elsewhere.  A bibliography of all CHAT studies referenced below is available for download. An overview of the relation of research to other elements of the integrated network is available elsewhere on this site, while a more complete review if included in my 2013 book.

What is implementation research?

All applied research involves the use of a clinical practice by a professional. Which ones would we therefore consider to be implementation research? Implementation research seeks to demonstrate the use of practices in a large scale, by programs of professionals that are community-based.

The critical differences here lie in the scale and context of the utilization of the practice. In contrast to the final phase of clinical research, in which we explore different ways a practice might be adapted for more typical use by an individual professional, implementation research seeks to document the use of the practice across a whole program of practitioners. To begin to achieve utilization on a scale that can potentially impact outcomes on a population level, implementation research must extend to the full range of eligible, community-based programs - to the home, the school, the hospital, and the workplace, as appropriate and as needed.  And implementation research must achieve this for a broad range of representative populations, regardless of their background or the presence of other conditions.

As discussed elsewhere, the transition into implementation research creates new demands on researchers.  This includes demands on their training and experience, as well as greater flexibility with respect to the kinds of research and scholarship required.  Attaining the milestone marking the transition from implementation to other applied research also creates a wide range of exciting new opportunities for researchers to engage with other elements of the integrated network.

Implementation Research versus Implementation Science: The CHAT+

Those familiar with the emerging field of Implementation Science (IS) will note differences with the specific approach to understanding implementation research adopted here. Given that IS typically considers how many different factors operating at many different levels influence when and how practices are utilized, how are we able to achieve such a specific focus in our review?  There are several reasons.

• Our goal was to capture the kinds of empirical research already undertaken around a very specific practice, rather than imagine the entire spectrum of research that could potentially be undertaken.
• We specifically selected the CHAT+ for study because the nature of the practice eliminates some factors that might typically merit more intensive examination and that complicate a research roadmap.  For example, the CHAT+ itself is relatively simple to administer, limiting concerns about training and fidelity that dog other practices. The CHAT+ is also simple to interpret, and results in a very specific outcome - determination of risk for ASD, subsequently either confirmed or dis-confirmed by diagnostic evaluation.  The CHAT+ itself requires relatively little time and personnel to administer, reducing barriers linked to resources.  Given long-standing advocacy and research documenting the increased impact of early intervention, the benefits of ASD screening are not in dispute.  This mitigates concerns that misunderstandings about the practice have limited its adoption.  We would fully expect some or all of these factors to be at play in implementation research involving other practices.  And of course, if the implementation research failed to result in widespread use of the CHAT+, these factors would certainly merit re-examination.
• Whereas IS studies a broad range of other system-level variables influencing implementation, we have focused on a narrow range of research findings more specific to a practice. Instead of describing potential research to explore a wide range of other system-level variables, we offer a specific framework to capture how these specific findings can influence outcome at the system level - i.e., via our integrated network. Stated differently, we do not think we should begin by subjecting this framework to scientific study, but should begin by using it to guide program and policy management, and test whether it can immediately help to improve practice.
• In general, we also believe that IS must begin by establishing the effectiveness of a specific practice and then identifying operational barriers to its implementation, before focusing on other factors related to individual attitudes or institutional culture.  In our experience leading implementation of new practices in community-based programs, we find that professionals and programs are eager to implement proven practices once these operational barriers are addressed. We wonder how often these other barriers ascribed to individual "attitudes" or institutional "cultures" simply reflect other factors.  Perhaps the evidence of the practice's impact is underwhelming? Perhaps researchers have failed to recognize the limitations of the practice? Perhaps researchers have failed to identify and address practical barriers to implementation.

Phases of implementation research

Clinical ResearchImplementation Research1. Assess delivery and gaps in access2. Improve delivery and close gaps3. Demonstrate regional deliveryTrainingServicesPopulation OutcomesPolicy and AdvocacyOther Applied ResearchOur initial review suggested clear differences between research intended to assess gaps and barriers to implementation, and research intended to demonstrate how to close these gaps or overcome these barriers. Whereas the latter required some active intervention to change practice, the former could rely on surveys, file reviews, or other methods that were relatively less intensive. Our reviews therefore distinguished between an assessment and an improvement phase within implementation research.

As noted above above, the use of CHAT+ itself is also relatively straightforward. As a result, the range of factors impacting its utilization is more limited.  In fact, only a small proportion of research utilizing the CHAT+ involved a primary objective focused on focused on implementation.  We restricted our discussion of the range of possible questions to the examples available through CHAT+ research, and simply grouped all of these factors contributing to gaps and barriers into the first two phases of implementation research. We expect that the range of possible factors will increase once we extend our review to include secondary objectives.

Note that, unlike Clinical Research, we did not require that all Implementation Research studies result in a clinical outcome (i.e., a diagnostic evaluation). While this standard certainly seems reasonable for some studies (e.g., seeking to demonstrate that practitioners can be trained effectively), it seemed unnecessary for studies exploring other factors (e.g., attitudes towards  the practice). We did require, however, that the study in question specifically use the CHAT+.

1. Assess barriers to delivery and gaps in access

We found only 5 studies with a primary objective involving the assessment of gaps and barriers to implementation, with most focused on gaps associated with the background of the child and family. For example, Herlihy and her colleagues explored whether there were racial/ethnic or socioeconomic factors that impacted age of screening and diagnosis (Herlihy et al., 2014). Lynch and his colleagues evaluated the feasibility of implementing the M-CHAT by mail, with follow-up provided by a registered nurse acting as a developmental screening coordinator (Lynch et al., 2015).

2. Improve delivery and close gaps

We found only 2 studies primarily focused on closing gaps and overcoming barriers to implementation. Brooks and her colleagues introduced web-based screening to improve access to ASD screening in an urban population (Brooks, Haynes, Smith, McFadden, & Robins, 2016). Roux and her colleagues explored the provision of ASD screening over the telephone to an underserved urban population (Roux et al., 2012).

3. Demonstrate effective and comprehensive regional delivery

In a final step towards demonstrating impact for a practice at a population level, we must demonstrate that we can close gaps and overcome barriers at the scale required.   While actual implementation at the state or national level remains the responsibility for public health and education, we believe that there is a role for researchers to confirm that this is possible at a smaller scale - say, across at least a city, a county, or a region. This may include additional clinical research to adapt and validate the practice as needed to each of those languages, cultures, and subpopulations (e.g., with co-occurring conditions) that constitute a meaningful proportion of the population. This may also include other clinical research to adapt and validate the practice for different settings (e.g., school, hospital, community, etc.), or professionals as needed.

We found only 1 study with a primary objective that can be viewed as seeking to improve delivery at the regional level. Rotholz and his colleagues evaluated the impact of a new policy that immediately connected children who failed a 2-tiered screening process (including the M-CHAT) to ASD-specific early intervention services. This recommendation was based on the assumption that a subsequent diagnostic evaluation would frequently confirm an ASD diagnosis (Rotholz, Kinsman, Lacy, & Charles, 2017).