|
|
||||||||||||||
|
|
|||||||||||||||
DEVELOPING RESEARCH AND PRACTICE |
1 Institute for Healthcare Improvement, 143 Garden Street, Wethersfield, CT 06109, USA
2 Health Care Improvement Leadership Development, Dartmouth Medical School, 7251 Strasenburgh Hall, Hanover, NH 03755, USA
Correspondence to:
Dr F Davidoff
143 Garden Street, Wethersfield, CT 06109, USA; fdavidoff{at}cox.net
10 August 2005
ABSTRACT
In contrast with the primary goals of science, which are to discover and disseminate new knowledge, the primary goal of improvement is to change performance. Unfortunately, scholarly accounts of the methods, experiences, and results of most medical quality improvement work are not published, either in print or electronic form. In our view this failure to publish is a serious deficiency: it limits the available evidence on efficacy, prevents critical scrutiny, deprives staff of the opportunity and incentive to clarify thinking, slows dissemination of established improvements, inhibits discovery of innovations, and compromises the ethical obligation to return valuable information to the public.The reasons for this failure are many: competing service responsibilities of and lack of academic rewards for improvement staff; editors and peer reviewers unfamiliarity with improvement goals and methods; and lack of publication guidelines that are appropriate for rigorous, scholarly improvement work. We propose here a draft set of guidelines designed to help with writing, reviewing, editing, interpreting, and using such reports. We envisage this draft as the starting point for collaborative development of more definitive guidelines. We suggest that medical quality improvement will not reach its full potential unless accurate and transparent reports of improvement work are published frequently and widely.
Keywords: quality improvement; evidence; publication; guidelines
A great deal of creative effort currently goes into making medical care safer and more reliable. The results of that improvement work are sometimes shared informally among those in the field and are occasionally published, primarily in the administrative and management literature and in a small number of journals devoted to the subject. Although much of that work is both useful and rigorous, most of it is unfortunately never made publicly available in either print or electronic form, and the reports that do appear vary considerably in accuracy, completeness, and transparency. The lack of such reports is particularly noticeable in the clinical literature, which is regrettable since, despite the crucial role they play, clinicians are notably reluctant to become involved in improvement efforts.
We have suggested elsewhere that both scientific discovery and experiential learning are required for improvement in medical care to flourish (F Davidoff, P Batalden, submitted for publication). The principal goal of science is to discover and disseminate new knowledge, a process that can be briefly summarized as "Plan-Do-Study-Publish" (T Nolan, personal communication). In contrast, the principal goal of experiential learning is to enhance performance. As a consequence, neither the action cycle used to characterize informal experiential learning ("Experience-Question-Conceptualize-Retry"1) nor the formal version of that cycle ("Plan-Do-Study-Act") that is now a central component of medical quality improvement2 include a "Publish" step. As discussed below, however, we suggest that, just as the growth of scientific knowledge would be unthinkable without publication, the improvement process will not realize its full potential unless the experiential learning that makes up much of quality improvement is also widely shared through publication.
WHY IS PUBLICATION ESSENTIAL IN QUALITY IMPROVEMENT?
The canon of science absolutely requires that scientific work must be captured in written or graphic form. Indeed, the biologist E O Wilson has gone so far as to state that "One of the strictures of the scientific ethos is that a discovery does not exist until it is safely reviewed and in print".3 It is true that publication in science does have an unfortunate tendency to be overvalued (the "publish or perish" phenomenon), since the principal instrument of social control in the scientific community is the exchange of information for professional and social status, funding, and power.4 Nonetheless, publication remains essential in science for many reasons:
Failure to publish new knowledge about improvement work also has a number of serious consequences:
THE IMPROVEMENT-PUBLICATION PARADOX
Unfortunately, the strengthening of quality improvement evidence through publication has recently become entangled in the following bureaucratic paradox, with potentially serious consequences. The Common Rule that governs the conduct of federally supported human subject research in the US defines research as a "systematic investigation, including research development, testing and evaluation, designed to develop or contribute to generalizable knowledge".18 Since the most widely used and respected criterion for the generalizability of knowledge is whether it has been published, quality improvement projects that have been (or even may be) published are now being considered "research" under the Common Rule definition. To complicate matters, since quality improvement in medicine virtually always involves human participants, quality improvement work that is published is now frequently considered to be a form of human subject research.19 Framed in those terms, virtually all quality improvement immediately becomes subject to the regulatory mechanisms that govern clinical researchmost importantly, protection of human participants through ethics committee or Institutional Review Board (IRB) review.
Protection of participants in quality improvement is, of course, essential. But a conceptual shift of medical quality improvement from an intrinsic professional responsibility20,21 to a research activity seems both illogical and counterproductive on several grounds. First, the transitivity of the Common Rules logic is itself open to question. Although all research strives to be generalizable (hence publishable), it does not automatically follow that all investigations that are published (hence generalizable) are researchfor example, case reports and case series, reviews, analytical studies, commentary and opinion pieces, many of which contain important, original, and generalizable knowledge, are not generally considered "research," or at least not pre-planned "systematic investigations" in the usual sense. In fact, the Belmont Report itself, on which the Common Rule is based, recognizes that valuable generalizable knowledge can and does flow from the experience of health care delivery per se, not just from research; in its words:
"Even when a procedure applied in practice may benefit some other person, it remains an intervention designed to enhance the well-being of a particular individual or groups of individuals; thus, it is a practice and need not be reviewed as research".22
Thus, since quality improvement is fundamentally "a procedure applied in practice", designed to enhance the well being of particular individuals or groups rather than to produce generalizable knowledge, there is no reason for it to be considered a priori as "research". Of course, such work can and should be seen as research if the initial improvement plan contains formal elements designed specifically to generate new generalizable knowledge, over and above its intended immediate benefit to its local participants.
Secondly, when protection of participants is at stake, it does not matter whether quality improvement is characterized as "research" or as some other kind of activity (such as experiential learning); protection is necessary in any case. Indeed, it can be argued, ironically, that patients need more protection in medical care systems that are not actively engaged in quality improvement than in systems that are. Finally, most IRBs are overburdened, understaffed, and underfunded;23 formal IRB review is generally slow and cumbersome;23 IRB judgments are often inconsistent;23,24 and most IRBs have little familiarity with the nature and methodologies of quality improvement. Requiring all quality improvement efforts to undergo such review could therefore have the paradoxical and damaging result of actually discouraging improvements in care.
For all of these reasons, we suggest that no one should be deterred from working to improve the care of individual patients and groups simply because of concerns that the results of that work may ultimately turn out to warrant publication. Similarly, no one who has already done improvement work should be reluctant to publish their results if they recognize, after the fact, that what they have learned is generalizable. The critical issue here is not whether they are doing research; it is whether staff engaged in improvement have taken the appropriate steps to protect those people who participate in their efforts to improve care.
WHY ISNT QUALITY IMPROVEMENT WORK PUBLISHED MORE OFTEN?
If publication of quality improvement evidence is so crucial, what explains the "publication gap"? To some extent it has to do with the nature of the people who do the work. Most are busy "front line" healthcare professionalsmanagers, administrators, planners, clinicianswith heavy competing service responsibilities. Many are neither oriented to, nor experienced in, academic work, including writing and publication; they also generally do not work in academic environments where they would "perish" if they failed to publish. And writing itself, particularly writing well, is hard.25 (A widely quoted saying among writers is: "Writing is easy. Just sit down at your desk and open a vein.")
But other powerful intellectual and cultural forces are also at work here. Editors, peer reviewers, and the academic medical community generally control both biomedical and clinical publication, and all of those stakeholders are deeply immersed in the culture of scientific discovery whose primary purpose is generation of new knowledge. As a consequence, they may be unfamiliar with the goals and methodologies of experiential learning, which is the principal approach used in solving the complex non-linear problems of quality improvement. The editors of, and reviewers for, biomedical journals may therefore have difficulty recognizing the nature, importance, or even the existence of many of those problems, which consequently can interfere with publication of reports of quality improvement work, even when that work is clinically important and methodologically sound. Moreover, until recently, little guidance has been available for authors, editors, and reviewers on how best to write, review, and edit complete and precise accounts of quality improvement.
PUBLICATION GUIDELINES FOR IMPROVING QUALITY IMPROVEMENT EVIDENCE: A DRAFT PROPOSAL
With these considerations in mind, we offer here a draft set of guidelines in the form of a checklist, designed to increase the completeness, accuracy, and transparency of original reports of quality improvement work (table 1
). The guidelines proposed here are intended primarily to support publication of the strongest and most definitive evidence on quality improvement in the permanent peer reviewed journal literature. They may also be useful in preparing reports of quality improvement work, much of it still preliminary or in progress, that are presented in the many important but more transient venues used for disseminating that information such as meetings, white papers, and media reports.
|
They have been developed with informal input from people with extensive experience in quality improvement, medical ethics, clinical research, and medical editing, and have been modified in response to feedback from people who have used them in writing and critiquing reports of quality improvement projects. They generally conform to the principles used in creating guidelines for reporting randomized clinical trials,27 and for studies that use other designs2833 and studies in specific content areas.34,35 Importantly, the consistency and completeness of reporting has been found to improve in journals that have endorsed and used such guidelines.36
The guidelines in table 1
differ in a number of significant ways from the earlier set.26 For example, the new draft guidelines are organized according to the IMRaD (Introduction, Methods, Results, and Discussion) format. The earlier guidelines explicitly rejected the IMRaD format on the grounds that, unlike the invariant protocols of clinical research, the initial plans used in improvement projects are frequently (and intentionally) altered during the course of the projects, which was seen as making them intrinsically incompatible with that format.26 In our view, the IMRaD structure is generic, reflecting the flow of thinking that underlies all learning and discovery, which is why it is widely used to report study designs of all types. More specifically, we would argue that, far from violating the logic of discovery, the shifts in improvement plans are, in fact, among the more important outcomes of the experiential learning (improvement) process, and therefore fit comfortably into the Results section of the IMRaD structure.
Secondly, the number of items in these draft guidelines has been expanded from eight in the earlier set26 to 16, to accommodate several important additional topics including: prior information available on the problem area; failures, risks or harms encountered; assessment of the projects limitations; evaluation of the projects internal and external validity; and specific plans for assessing maintenance of the improvement. Although 16 items is a substantial number, it should be manageable. Authors, editors, and peer reviewers have not had difficulty using 2025 items in other publication guidelines.
NEXT STEPS
We view the draft reporting guidelines offered here as a reflection of the rapidly developing science of quality improvement, and hence as a further step in the evolution of publication standards, not as a finished product. It would be helpful if readers would send the editors of this journal their comments and criticisms regarding this version of the guidelines. Feedback on their strengths and limitations from people who use them to write and critique reports of quality improvement would be particularly valuable.
We also propose that, at the earliest opportunity, a group representing the many stakeholders in quality improvementclinicians, administrators, health services researchers, social scientists, editors, ethicists, statisticians, patients, and othersshould be convened to assume stewardship of the guidelines. Diffusion of an innovation such as reporting guidelines is a complex and intensely social process.37,38 Drawing on the experience gained in creating other publication guidelines27,28,30 and similar standards documents,39 this group would undertake a systematic critique of the completeness, clarity, and appropriateness of specific guideline items. It should also consider a number of general questions including:
We recognize that the use of reporting guidelines is not justified unless they improve the quality of reporting. We therefore urge, that, once a more definitive version of the guidelines has been formulated, their impact on quality improvement reports be formally and carefully assessed. For example, editors and peer reviewers could be asked for subjective judgments of their value in making editorial judgments; authors could judge their value in organizing and writing reports; and readers could judge their value in understanding and applying published papers. The impact of the guidelines could be assessed objectively and quantitatively by comparing papers prepared with and without them with regard to completeness, accuracy, and precision of reporting,40 and the suitability of such papers for inclusion in systematic reviews.
Finally, we recognize that, by itself, accurate and transparent reporting is of little value if the work being reported is fundamentally flawed. On the other hand, we would argue that reporting standards can actually play a role in improving improvement work. Thus, although the primary purpose of such guidelines is to improve the reporting rather than the planning and conduct of improvement work, they could very well have a positive secondary (or "backwash") effect on the quality of the work itself by providing an explicit consistent framework for its design and execution. In that sense, the guidelines could serve an important educational purpose, particularly in conjunction with an "elaboration and explanation" document such as the one created to support the CONSORT guidelines.41 Systematic consideration of specific background elements and criteria related to each guideline item, such as those listed in table 2
, is an example of such an educational function.
|
The improvement-publication gap in part reflects the reality that most people who do the work of quality improvement are more interested in actually improving care than in writing about what they do. But widespread misunderstandings about the nature of experiential learning and experimental discoverythe perceptions, for example, that experiential learning deals with problems that are intractably "messy;" that the evidence for the efficacy of most experiential learning is intrinsically weak; and that "applied" disciplines are of less intellectual and social value than "pure" onesalso appear to play important roles in discouraging publication, particularly in the clinical literature. Moreover, because the current editors and peer reviewers of biomedical journals are relatively unfamiliar with the elements that are most worthwhile in making improvements, they have not encouraged authors to use an inclusive typology of those elements, thus possibly contributing to the lack of progress toward a "science" of improvement.
We therefore strongly encourage the widest possible reporting of quality improvement work in print and electronic form. In support of that goal, we urge the further development, adoption, and widespread use of publication standards such as those proposed here, that contain a systematic and comprehensive set of the elements of medical quality improvement.
ACKNOWLEDGEMENTS
Donald Berwick, Ann Davidoff, Joanne Lynn, Lloyd Provost, Jane Roessner, Mark Rzeszotarski, Mark Splaine, and David Stevens all contributed importantly to the development of ideas in this paper.
FOOTNOTES
Financial sponsors: None
REFERENCES
Related Articles
This article has been cited by other articles:
![]() |
D. M. Berwick The Science of Improvement JAMA, March 12, 2008; 299(10): 1182 - 1184. [Full Text] [PDF] |
||||
![]() |
T. F. Boat, S. M. Chao, and P. H. O'Neill From Waste to Value in Health Care JAMA, February 6, 2008; 299(5): 568 - 571. [Full Text] [PDF] |
||||
![]() |
R. Grol, D. M Berwick, and M. Wensing On the trail of quality and safety in health care BMJ, January 12, 2008; 336(7635): 74 - 76. [Full Text] [PDF] |
||||
![]() |
S. H. Woolf The Meaning of Translational Research and Why It Matters JAMA, January 9, 2008; 299(2): 211 - 213. [Full Text] [PDF] |
||||
![]() |
D P Stevens and B. Andersson-Gare A proposal: publication guidelines for healthcare improvement education reports Qual. Saf. Health Care, December 1, 2007; 16(6): 402 - 402. [Full Text] [PDF] |
||||
![]() |
B. V Watts, B. Shiner, A. Pomerantz, P. Stender, and W. B Weeks Outcomes of a quality improvement project integrating mental health into primary care Qual. Saf. Health Care, October 1, 2007; 16(5): 378 - 381. [Abstract] [Full Text] [PDF] |
||||
![]() |
J. Thor, J. Lundberg, J. Ask, J. Olsson, C. Carli, K. P. Harenstam, and M. Brommels Application of statistical process control in healthcare improvement: systematic review Qual. Saf. Health Care, October 1, 2007; 16(5): 387 - 399. [Abstract] [Full Text] [PDF] |
||||
![]() |
P. Batalden and F. Davidoff Teaching Quality Improvement: The Devil Is in the Details JAMA, September 5, 2007; 298(9): 1059 - 1061. [Full Text] [PDF] |
||||
![]() |
P. E. Morris and K. Dracup Quality Improvement or Research? The Ethics of Hospital Project Oversight Am. J. Crit. Care., September 1, 2007; 16(5): 424 - 426. [Full Text] [PDF] |
||||
![]() |
D. P Stevens If you haven't published your work, it's time to start Qual. Saf. Health Care, August 1, 2007; 16(4): 242 - 243. [Full Text] [PDF] |
||||
![]() |
C. Grady Quality Improvement and Ethical Oversight Ann Intern Med, May 1, 2007; 146(9): 680 - 681. [Full Text] [PDF] |
||||
![]() |
J. Lynn, M. A. Baily, M. Bottrell, B. Jennings, R. J. Levine, F. Davidoff, D. Casarett, J. Corrigan, E. Fox, M. K. Wynia, et al. The Ethics of Using Quality Improvement Methods in Health Care Ann Intern Med, May 1, 2007; 146(9): 666 - 673. [Abstract] [Full Text] [PDF] |
||||
![]() |
B. E. Landon, L. S. Hicks, A. J. O'Malley, T. A. Lieu, T. Keegan, B. J. McNeil, and E. Guadagnoli Improving the Management of Chronic Disease at Community Health Centers N. Engl. J. Med., March 1, 2007; 356(9): 921 - 934. [Abstract] [Full Text] [PDF] |
||||
![]() |
K. G. Shojania, S. R. Ranji, K. M. McDonald, J. M. Grimshaw, V. Sundaram, R. J. Rushakoff, and D. K. Owens Effects of quality improvement strategies for type 2 diabetes on glycemic control: a meta-regression analysis. JAMA, July 26, 2006; 296(4): 427 - 440. [Abstract] [Full Text] [PDF] |
||||
![]() |
G R Baker Strengthening the contribution of quality improvement research to evidence based health care. Qual. Saf. Health Care, June 1, 2006; 15(3): 150 - 151. [Full Text] [PDF] |
||||
![]() |
P Pronovost and R Wachter Proposed standards for quality improvement research and publication: one step forward and two steps back. Qual. Saf. Health Care, June 1, 2006; 15(3): 152 - 153. [Full Text] [PDF] |
||||
![]() |
F. Godlee Improving on improvement BMJ, May 6, 2006; 332(7549): . [Full Text] [PDF] |
||||
![]() |
M H Chin and A T Chien Reducing racial and ethnic disparities in health care: an integral part of quality improvement scholarship. Qual. Saf. Health Care, April 1, 2006; 15(2): 79 - 80. [Full Text] [PDF] |
||||
![]() |
R. Niederman and J. Leitch "Know what" and "know how": knowledge creation in clinical practice. J. Dent. Res., April 1, 2006; 85(4): 296 - 297. [Full Text] [PDF] |
||||
![]() |
T Greenhalgh, J Russell, and D Swinglehurst Narrative methods in quality improvement research Qual. Saf. Health Care, December 1, 2005; 14(6): 443 - 449. [Abstract] [Full Text] [PDF] |
||||
![]() |
D P Stevens Why new guidelines for reporting improvement research? And why now? Qual. Saf. Health Care, October 1, 2005; 14(5): 314 - 314. [Full Text] [PDF] |
||||
![]() |
D M Berwick Broadening the view of evidence-based medicine Qual. Saf. Health Care, October 1, 2005; 14(5): 315 - 316. [Full Text] [PDF] |
||||
![]() |
R G Thomson Consensus publication guidelines: the next step in the science of quality improvement? Qual. Saf. Health Care, October 1, 2005; 14(5): 317 - 318. [Full Text] [PDF] |
||||
Read all eLetters
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP |