Quality of out-of-hours telephone triage by general practitioners and nurses: development and testing of the AQTT – an assessment tool measuring communication, patient safety and efficiency

Abstract Objective: To develop a valid and reliable assessment tool able to measure quality of communication, patient safety and efficiency in out-of-hours (OOH) telephone triage conducted by both general practitioners (GP) and nurses. Design: The Dutch KERNset tool was translated into Danish and supplemented with items from other existing tools. Face validity, content validity and applicability in OOH telephone triage (OOH-TT) were secured through a two-round Delphi process involving relevant stakeholders. Forty-eight OOH patient contacts were assessed by 24 assessors in test-retest and inter-rater designs. Setting: OOH-TT services in Denmark conducted by GPs, nurses or doctors with varying medical specialisation. Patients: Audio-recorded OOH patient contacts. Main outcome measures: Test-retest and inter-rater reliability were analysed using ICCagreement, Fleiss’ kappa and percent agreement. Results: Major adaptations during the Delphi process were made. The 24-item assessment tool (Assessment of Quality in Telephone Triage – AQTT) measured communicative quality, health-related quality and four overall quality aspects. The test-retest ICCagreement reliability was good for the overall quality of communication (0.85), health-related quality (0.83), patient safety (0.81) and efficiency (0.77) and satisfactory when assessing specific aspects. Inter-rater reliability revealed reduced reliability in ICCagreement and in Fleiss’ kappa. Percent agreement revealed satisfactory agreements when differentiating between ‘poor’ and ‘sufficient’ quality). Conclusion: The AQTT demonstrated high face, content and construct validity, satisfactory test-retest reliability, reduced inter-rater reliability, but satisfactory percent agreement when differentiating between ‘poor’ and ‘sufficient’ quality. The AQTT was found feasible and clinically relevant for assessing the quality of GP- and nurse-led OOH-TT. KEYPOINTS Comparative knowledge is sparse regarding quality of out-of-hours telephone triage conducted by general practitioners and nurses. The assessment tool (AQTT) enables assessment of quality in OOH telephone triage conducted by nurses and general practitioners AQTT is feasible and clinically relevant for assessment of communication, patient safety and efficiency. AQTT can be used to identify areas for improvement in telephone triag