Home Print this page Email this page Small font size Default font size Increase font size
Users Online: 3126
Home About us Editorial board Search Ahead of print Current issue Archives Submit article Instructions Subscribe Contacts Login 

 Table of Contents 
Year : 2018  |  Volume : 7  |  Issue : 2  |  Page : 394-400  

Development of a quality scoring tool to assess quality of discharge summaries

1 Primary Health Care, Nova Scotia Health Authority; Dalhousie University, Halifax, Nova Scotia, Canada
2 Nova Scotia Health Authority, Halifax, Nova Scotia, Canada
3 Primary Health Care, Nova Scotia Health Authority, Halifax, Nova Scotia, Canada
4 Dalhousie University; Nova Scotia Health Authority, Halifax, Nova Scotia, Canada

Date of Web Publication11-Jul-2018

Correspondence Address:
Dr. Tara Sampalli
Primary Health Care, Suite 0265, 6960 Mumford Road, Halifax, NS B3L 1P4
Login to access the Email id

Source of Support: None, Conflict of Interest: None

DOI: 10.4103/jfmpc.jfmpc_407_16

Rights and Permissions

Introduction: Timely, precise, and relevant communication between hospital-based clinicians and primary care physicians post-discharge (DC) ensures quality transitions, thereby reducing patient safety incidents and preventing readmission. At the present time there is limited knowledge of elements of quality or methods to score the quality criteria in the context of DC summaries. The Nova Scotia Health Authority, a provincial health system responsible for the delivery of services in a small Canadian province, embarked on a system-level approach to the standardization of DC summaries in an effort to improve quality and safety at care transitions from hospital to primary care. Materials and Methods: A comprehensive literature review to retrieve items relevant to quality in DC summaries, retrospective audit of charts, a consensus development process, and, finally, validation of a scoring tool were conducted in order to develop a quality scoring tool for DC summaries. Results: Relevant items were identified through the literature review and consensus development process. Corresponding definitions that were established assisted the development of the quality criteria, which were subsequently used to score the quality of DC summaries in our organization. Conclusion: The scoring tool developed through this work will be applied to help us gain a more in-depth understanding of quality in DC summaries and support the development of suitable education and quality processes in the health authority that can best support safe care transitions for patients.

Keywords: Accreditation standards, quality criteria in discharge summaries, safe care transitions, scoring tool

How to cite this article:
Savvopoulos S, Sampalli T, Harding R, Blackmore G, Janes S, Kumanan K, Gibson R, MacKnight C. Development of a quality scoring tool to assess quality of discharge summaries. J Family Med Prim Care 2018;7:394-400

How to cite this URL:
Savvopoulos S, Sampalli T, Harding R, Blackmore G, Janes S, Kumanan K, Gibson R, MacKnight C. Development of a quality scoring tool to assess quality of discharge summaries. J Family Med Prim Care [serial online] 2018 [cited 2021 Sep 23];7:394-400. Available from: https://www.jfmpc.com/text.asp?2018/7/2/394/236428

  Introduction Top

Safe and effective care transitions from hospital to home require accurate and efficient transfer of relevant patient information. Timely, precise, and relevant communication between hospital-based clinicians and primary care physicians (PCPs) postdischarge (DC) ensures quality transitions, thereby reducing patient safety incidents and preventing readmissions.[1] Furthermore, evidence suggests that appropriate engagement of patients and families in DC planning can help make care transitions safe and effective.[2] The DC summary can support quality care transitions and thus is of relevance to clinicians and organizations alike.[3]

While the DC summary is an important component of quality care transitions, it has been shown that it frequently lacks critical data, is often not received in an appropriate timeframe, or may not even reach the PCP at all, which then results in clinicians being unaware of pending test results and evaluations to be scheduled after DC.[4] Delays in follow-up have been associated with increased hospital readmissions for the same condition and a trend toward a longer length of stay, exacerbating the challenge of providing safe and effective care.[5],[6] Furthermore, quality transitions from hospital to community remain out of reach due to the lack of practical and standardized approaches to assess patients who are at a higher risk for hospital readmissions, patient safety incidents, or even death in the immediate post-DC period. Conversely, by standardizing DC summaries, it has been demonstrated that uptake and sustainability of process and use are high among end users and timeliness is improved.[7]

Computers may be viewed as a solution to delayed or incomplete information transfer of DC summaries through automation, thereby bringing an increased focus on early DC.[8] O'Leary et al. reported reduced delays in the delivery of important information after implementing an automated process, recommending computers to improve overall quality, improve patient satisfaction, and reduce preventable adverse outcomes.[2] In a systematic review of 12 major studies, Motamedi et al. assessed the efficacy of computer-based DC communication, finding evidence for comparatively better quality, completeness, timeliness, and satisfaction of physicians and patients/families when electronic systems are used during the DC process.[9] No significant difference for mortality and readmission was reported. Furthermore, limited information about post-DC patient outcomes was found in the review.

However, while automation in the DC process addresses some of the drawbacks of paper-based systems, clinicians do not completely embrace electronic DC (eDC) systems due to issues of system reliability, incompatibility, and period duplication of data. Enguidanos and Brumley (2005) reported less uptake of eDC by physicians who preferred previous practices over the new system, suggesting resistance to change.[10],[11] Callen et al. reported that incompleteness in eDCs is possibly a result of insufficient training and education of care providers, the lack of understanding of the importance of accuracy and completeness of eDC, inadequate user interface, deficiencies in computer literacy, and insufficient integration with work processes.[12]

Kusnadi concluded that the problem of data quality and delayed delivery of DC summaries may not be solved simply by implementing an eDC summary system and is best targeted by more global strategies, suggesting a more comprehensive approach to facilitate better communication between hospital care and PCPs.[13] Kusnadi posits the “entrenched custom and practice of uni-professional orientation of DC summaries, attitude of senior doctors, usage of short forms and abbreviations, and little accountability for quality control.” (pg. 5) He recommends the gap in quality control must be counteracted by training of junior doctors, regulation of use of shortened forms, and improving features of data entry systems. Moreover, he suggests structuring the clinical coding of data, introducing systems to ensure greater organizational accountability for effective DC communication, inter-disciplinary contribution toward building DC summaries, and integrating them into the care pathways. A multipronged approach is supported by Schabetsberger et al. when transitioning from paper-based DC communication to help address change management and organizational change challenges.[14] The propensity to automate is validated by the advantages enumerated above; nevertheless, it is prudent for organizations to carefully consider what determines quality in a DC summary. However, a paucity of specific criteria to determine quality of a DC summary in the existing literature has resulted in a lack of consensus on what specifically defines quality of a DC summary and how it can support quality and safe care transition process.[9] This study aims to define the elements of quality in the context of the DC summary and to propose a framework and a scoring tool for organizations to use to determine the quality of existing or proposed DC summaries.

  Materials and Methods Top

Recognizing the value of implementing eDCs, a pilot project in one inpatient department (implemented in April 2012) was subsequently extended across the organization. Based on positive feedback from clinicians, the Nova Scotia Health Authority (NSHA), a provincial health system responsible for the delivery of services in a small Canadian province, embarked on a system-level approach to the standardization of DC summaries in an effort to improve quality and safety at care transitions from hospital to PCPs. The current work contributes to this organizational initiative.

Study design

The methodology for developing the quality tool involved three key steps, outlined in [Figure 1]. Ethics approval was approved from NSHA's ethics board.
Figure 1: Study methodology

Click here to view

The literature was reviewed to elucidate the meaning of the “quality of discharge summary” and second, to identify specific quality items to be included in the assessment tool. Articles published between 1998 and 2014 were selected which included case studies, surveys, interviews, mixed linear models, and retrospective, prospective, quasi-experimental, systematic reviews, and randomized controlled trials. An initial search conducted in PubMed yielded more than 100 articles, based on relevance, from WorldCat.org, ScienceDirect, and a manual search of article bibliographies. Specifically, Medical Subject Heading (MeSH) terms and the keyword “electronic” as a string were used to access articles. The term “patient discharge” was found to have two meanings in MeSH: “An administrative process” and “summaries that serve as the primary documents.” The following search algorithm yielded 48 articles:

In addition, other key words in various combinations were used as strings to find nonindexed articles. The key word terms included: Quality of discharge summary; hospital discharge; quality of eDC; e-discharge; quality of e-discharge summary; quality improvement of discharge summary; quality and timeliness discharge summary; and quality and timeliness hospital discharge.

The literature review was used to identify elements of quality in eDCs. A quality scoring tool was developed based on the evidence review to support the understanding and advancement of quality in DC summaries. The content quality evaluation tool developed as part of this study is intended to be generalizable to other institutions. In addition to examining the status of DC summaries and the care transition process, based on the wide variation of what is deemed to be quality found in the existing research, the criteria identified in the first step of our methodology were further reviewed to validate items through a second step of consensus development. A diverse group of relevant members from the health authority were recruited to review the content and validate the scoring criteria in the scoring tool. These members brought diverse perspectives based on their background and included policymakers, quality leads, family physicians, department chiefs, and researchers. The scoring tool was validated by coders. Inter-rater reliability was calculated to determine the level of agreement among reviewers using Cohen's Kappa.[15]

  Results Top

A key finding from the evidence review showed that the quality of a DC summary is determined by understanding the criteria used for assessment and how these criteria are defined. These searches yielded an additional 42 articles [Figure 2].
Figure 2: Literature search

Click here to view

Our focus was to retrieve articles comparing electronic and dictated DC summaries and to evaluate the criteria on which the quality of eDC was determined. If abstracts failed the inclusion criteria, a cursory full-text review was performed. Articles were divided into two categories for further analysis. The first category included 51 articles comparing electronic versus handwritten or dictated DC summaries. The second category included 39 articles about quality improvement initiatives, aspects of DC summary quality, family physician surveys about quality aspects of summaries, and quality improvement experiences of organizations.

Horwitz et al. assessed the quality of DC summaries in three domains: timeliness, transmission, and content.[3] They defined timeliness as the number of days elapsed between the date of DC and the date of dictation, median timeliness (the most common timeline required to complete post-DC dictation), and the percentage of summaries completed on the DC day. The evaluation of the transmission aspect has two components: first, a proportion of summaries sent out successfully and second, median number of physicians per DC summary scheduled for patient follow-up without receiving a copy of the summary. Content is defined as presence of 21 predetermined items. In addition, compliance with Joint Commission (JC) and Transitions of Care Consensus Conference (TOCCC) recommendations on individual items was also assessed. In contrast, Russell et al. assessed DC summary quality on four areas of common error: presentation, relevance, accuracy, and clarity.[16] Presentation is defined as the use of subheadings to compartmentalize information and appropriate prioritization of problem list. Accuracy is the verification of significant abnormalities, such as laboratory or physical findings by comparing them to results obtained from computer databases. Clarity is inclusion of all the significant findings without excluding important details. Relevance is defined as whether the listed principal and secondary diagnoses are relevant to current admission and the discussion in the clinical synopsis.

Other studies have assessed DC summaries based on completeness, pertinence, organization, and global rating, and inclusion (of predetermined data items), clarity (whether lucid, understandable, hard to read, or unintelligible), exclusion (percentage of useless material in summary), and consistency (level of consistency of dictation with principal diagnosis).[17],[18] It is important to note that most of these concepts are open to deliberation and subjective interpretation and therefore, must be carefully defined and validated, which is the approach taken in the current work. However, our literature review and consensus development process consistently highlighted particular aspects of a DC as being critical [Figure 3].
Figure 3: Quality criteria

Click here to view

[Table 1] shows a list of the most relevant items identified from literature review and the corresponding definitions that were established to assist in our development of the quality criteria.
Table 1: Scoring tool content

Click here to view

Thirteen items suggested by the JC and TOCCC and an initial ten items ranked according to hospitalist and PCP preference were adapted for our purposes.[2],[4],[19] Consensus definitions of JC items were consulted to specify items in our scoring tool, followed by assigning appropriate weights to assess quality based on each item's individual contribution.[20] An item would get the defined maximum score when content is ideal, sufficient, accurate, and clear.

Scoring of individual items was a two-step process: in the first step, an item was assigned a base score up to a maximum of two points based on the assessment of information content. One point was deducted when information was deficient or excessive/irrelevant to the DC. No points were assigned if the information was absent. In the second step, base scores were multiplied with preassigned weights.

Quality scoring tool

A total of eight key stakeholders participated in the consensus development process. The team developed required elements of assessment, standardized definitions, scoring criteria, and methods that are described in the following sections.

The tool is based on six components proposed in the JC Standard IM.6.10, EP 7 as shown in [Table 2]: (1) Reason for hospitalization, (2) significant findings, (3) procedures and treatment provided, (4) patient's DC condition, (5) patient and family instructions (as appropriate), and (6) attending physician's signature.[19],[20],[21] It is also based on seven TOCCC recommendations: (1) Principal diagnosis, (2) problem list, (3) medication list, (4) transferring physician name and contact information, (5) cognitive status of the patient, (6) test results, and (7) pending test results for quality DC summary.[22]
Table 2: Scoring criteria

Click here to view

Validation of scoring tool

The scoring tool was then validated by four coders who rated eDC quality using the tool. The coders were two members of the research team (a physician team member and research associate) and two community physicians. This process helped to further refine the scoring strategy used in the tool. Each member applied the scoring tool to rate the quality of five eDC summaries randomly selected from a set of recently completed summaries by another member of the research team. We completed two iterations of validation with two sets of five charts and modifications to the scoring strategy to shift the kappa scores from a fair level of agreement (0.05) to a moderate level of agreement (0.58) among reviewers in the coder rating of the quality (0.01–0.20 as none to slight, 0.21–0.40 as fair, 0.41–0.60 as moderate, 0.61–0.80 as substantial, and 0.81–1.00 as almost perfect agreement).[15]

  Discussion Top

The primary objective of this initiative was to examine quality, patient safety, and best practices in care transitions while we move toward adopting user-friendly tools for standardized patient care documentation. The eDC provides an important transition step for processes, behaviors, and system changes that are required to support an electronic documentation environment as shown in [Table 2]. However, based on our review of the current status and evidence, it was identified that there was a clear gap in established definitions and criteria to ascertain quality in DC summaries. These standards are equally important in determining the benefits and merits of using eDC summaries.

As described in this article, a comprehensive literature review helped develop quality scoring tool which will be used to assess the current state of quality in DC summaries leading to some future state considerations for quality in DC summaries and care transition processes.

  Conclusion Top

In this paper, we have described the development and application of a quality scoring tool to improve the quality of care transitions for patients being discharged from hospital care to the care of their primary care physicians. Next steps of this work will involve using the quality scoring tool to encourage education, training and other supports to further enhance the use of eDCs at the point of care and in follow up care to better meet the needs of patients and providers and towards the broader goal of improving safety of care in care transitions. Quality assurance in eDC can impact transitions to PCPs, an important consideration as increasing numbers of patients are cared for in community settings.

Financial support and sponsorship


Conflicts of interest

There are no conflicts of interest.

  References Top

van Walraven C, Seth R, Laupacis A. Dissemination of discharge summaries. Not reaching follow-up physicians. Can Fam Physician 2002;48:737-42.  Back to cited text no. 1
O'Leary KJ, Liebovitz DM, Feinglass J, Liss DT, Baker DW. Outpatient physicians' satisfaction with discharge summaries and perceived need for an electronic discharge summary. J Hosp Med 2006;1:317-20.  Back to cited text no. 2
Horwitz LI, Jenq GY, Brewster UC, Chen C, Kanade S, Van Ness PH, et al. Comprehensive quality of discharge summaries at an academic medical center. J Hosp Med 2013;8:436-43.  Back to cited text no. 3
van Walraven C, Rokosh E. What is necessary for high-quality discharge summaries? Am J Med Qual 1999;14:160-9.  Back to cited text no. 4
Li JY, Yong TY, Hakendorf P, Ben-Tovim D, Thompson CH. Timeliness in discharge summary dissemination is associated with patients' clinical outcomes. J Eval Clin Pract 2013;19:76-9.  Back to cited text no. 5
Misky GJ, Wald HL, Coleman EA. Post-hospitalization transitions: Examining the effects of timing of primary care provider follow-up. J Hosp Med 2010;5:392-7.  Back to cited text no. 6
Wood SK, Campbell AK, Marden JD, Schmidtman L, Blundell GH, Sheerin NJ, et al. Inpatient care to community care: Improving clinical handover in the private mental health setting. Med J Aust 2009;190:S144-9.  Back to cited text no. 7
Hübner U, Giehoff C. Why continuity of care needs computing – Results of a quantitative document analysis. Stud Health Technol Inform 2002;90:483-7.  Back to cited text no. 8
Motamedi SM, Posadas-Calleja J, Straus S, Bates DW, Lorenzetti DL, Baylis B, et al. The efficacy of computer-enabled discharge communication interventions: A systematic review. BMJ Qual Saf 2011;20:403-15.  Back to cited text no. 9
Pagliari C, Donnan P, Morrison J, Ricketts I, Gregor P, Sullivan F, et al. Adoption and perception of electronic clinical communications in Scotland. Inform Prim Care 2005;13:97-104.  Back to cited text no. 10
Enguidanos SM, Brumley RD. Risk of medication errors at hospital discharge and barriers to problem resolution. Home Health Care Serv Q 2005;24:123-35.  Back to cited text no. 11
Callen JL, Alderton M, McIntosh J. Evaluation of electronic discharge summaries: A comparison of documentation in electronic and handwritten discharge summaries. Int J Med Inform 2008;77:613-20.  Back to cited text no. 12
Kusnadi K. Moving from Paper Based to Electronic Hospital Discharge Summaries: A Mixed Methods Investigation. University of Huddersfield; 01 March, 2012. Available from: http://www.eprints.hud.ac.uk/13223/1/kusnadifinalthesis.pdf. [Last accessed on 2015 Sep 15].  Back to cited text no. 13
Schabetsberger T, Ammenwerth E, Andreatta S, Gratl G, Haux R, Lechleitner G, et al. From a paper-based transmission of discharge summaries to electronic communication in health care regions. Int J Med Inform 2006;75:209-15.  Back to cited text no. 14
Viera AJ, Garrett JM. Understanding interobserver agreement: The kappa statistic. Fam Med 2005;37:360-3.  Back to cited text no. 15
Russell P, Hewage U, Thompson C. Method for improving the quality of discharge summaries written by a general medical team. Intern Med J 2014;44:298-301.  Back to cited text no. 16
Santana MJ, Holroyd-Leduc J, Flemons WW, O'Beirne M, White D, Clayden N, et al. The seamless transfer of care: A pilot study assessing the usability of an electronic transfer of care communication tool. Am J Med Qual 2014;29:476-83.  Back to cited text no. 17
Rao P, Andrei A, Fried A, Gonzalez D, Shine D. Assessing quality and efficiency of discharge summaries. Am J Med Qual 2005;20:337-43.  Back to cited text no. 18
Joint Commission. About Our Standards|Joint Commission; 2014. Available from: http://www.jointcommission.org/standards_information/standards.aspx. [Last retrieved on 2014 May 05].  Back to cited text no. 19
Reilly JB, Marcotte LM, Berns JS, Shea JA. Handoff communication between hospital and outpatient dialysis units at patient discharge: A qualitative study. Jt Comm J Qual Patient Saf 2013;39:70-6.  Back to cited text no. 20
Kind AJH, Smith MA. Documentation of mandated discharge summary components in transitions from acute to subacute care. In: Henriksen K, Battles JB, Keyes MA, Grady ML, editors. Advances in patient safety: new directions and alternative approaches (Vol 2: Culture and Redesign). Rockville, MD: Agency for Healthcare Quality and Research; 2008. p. 1-10. Available from: http://www.mendeley.com/catalog/documentation-mandated-discharge-summary-components-transitions-acute-subacute-care/. [Last accessed on 2015 Nov 10].  Back to cited text no. 21
Snow V, Beck D, Budnitz T, Miller DC, Potter J, Wears RL, et al. Transitions of care consensus policy statement American College of Physicians-Society of General Internal Medicine-Society of hospital medicine-American Geriatrics Society-American College of Emergency Physicians-Society of academic emergency medicine. J Gen Intern Med 2009;24:971-6.  Back to cited text no. 22


  [Figure 1], [Figure 2], [Figure 3]

  [Table 1], [Table 2]

This article has been cited by
1 Structure, content, unsafe abbreviations, and completeness of discharge summaries: A retrospective analysis in a University Hospital in Austria
Christine Maria Schwarz,Magdalena Hoffmann,Christian Smolle,Michael Eiber,Bianca Stoiser,Gudrun Pregartner,Lars-Peter Kamolz,Gerald Sendlhofer
Journal of Evaluation in Clinical Practice. 2021;
[Pubmed] | [DOI]
2 Usability of Electronic Health Record-generated Discharge Summaries: Heuristic Evaluation (Preprint)
Patrice Tremoulet,Priyanka D Shah,Alisha A Acosta,Christian W Grant,Jon Tyler Kurtz,Peter Mounas,Michael Kirchoff,Elizabeth Wade
Journal of Medical Internet Research. 2020;
[Pubmed] | [DOI]
3 An Evaluation of the Content of Pediatric Discharge Summaries
Divya Lakhaney,Sumeet L. Banker
Hospital Pediatrics. 2020; 10(11): 949
[Pubmed] | [DOI]


Similar in PUBMED
   Search Pubmed for
   Search in Google Scholar for
 Related articles
Access Statistics
Email Alert *
Add to My List *
* Registration required (free)

  In this article
   Materials and Me...
   Article Figures
   Article Tables

 Article Access Statistics
    PDF Downloaded182    
    Comments [Add]    
    Cited by others 3    

Recommend this journal