Article Text
PDF
Abstract
Background Analgesics represent one of the most frequently administered drugs during hospital stay. It is estimated that 40–80% of inpatients suffer from intermittent pain during their stay. In the current literature, indicators are considered an efficient, targeted method for the assessment of drug therapy quality. However, little is known about useful methods of identifying indicators for the quality of pharmacological therapy for pain.
Purpose The aim of this work was to design an indicator development method and subsequent indicator set for the assessment of pharmacological pain management quality with a primary focus on safety, and secondarily efficacy, in the inpatient setting.
Indicator development process The following strategies were used: (i) a systematic literature review (PubMed, Embase and Scopus); (ii) analysis of a critical incident reporting system, encompassing 1557 reports from 37 hospitals; and (iii) semistructured expert interviews. A list of 172 pain related indicators was compiled. The 56 most pain specific indicators were subsequently evaluated in a Delphi panel. The final set consisted of 15 indicators: nine were chosen based on the risk priority number, seven according to expert rating of their overall relevance. The three highest rated indicators were: (1) patients treated with non-steroidal anti-inflammatory drugs AND creatinine clearance <60 ml/min; (2) standard use of a visual analogue scale for pain assessment; and (3) paracetamol >4 g/day.
Conclusion A combination of literature search, analysis of a critical incident reporting system and expert interviews was useful for the indicator development. Although indicators could be well rated based on risk priority number, process and structure indicators were better assessed based on their attributed relevance.
Statistics from Altmetric.com
Background
Drug related problems (DRPs) represent a major threat to patient safety in healthcare institutions.1 The actual literature states that DRPs cause approximately 5–7% of all hospital admissions and affect 7.5–20% of all inpatients, prolonging hospital stay significantly.2,–,5 It is furthermore estimated that 47–76% of DRPs are preventable.6,–,10 Medication errors most frequently happen during the prescribing (39%) and administration (38%) process, followed by documentation (12%) and dispensing (11%).11 Certain drug classes are more prone to DRPs or cause more severe consequences than others. Organisations such as the Institute for Safe Medication Practices recommend focusing on specific drug classes considered high alert medications, among them opioids for pain therapy.12
Pain is a serious concern among inpatients, and analgesics represent one of the most frequently administered drugs during hospital stay. It is estimated that 40–80% of inpatients suffer from intermittent pain, and that up to 30% experience serious pain during their stay.13,–,15 Among patients in surgery and oncology, it is estimated that some 85% are treated with analgesics, many of them inadequately.13
In order to improve medication safety in an individual institution, meaningful baseline data are necessary. Different methods to quantify and qualify DRPs have been discussed in the literature. While medical chart reviews might provide the best insight into individual cases, this approach is very expensive in terms of personnel required. Severe events are best detected by the analysis of critical incident reporting systems (CIRS). However, the indicator technology (also referred to as ‘trigger tool’) is considered more efficient, and nevertheless effective.16 While indicators do not offer comprehensive quality insight, even when they are highly valid, they allow the facilitation of complex quality assessment issues.17
Little guidance is available in the literature on an efficient indicator development process for medication related indicators. While several articles recommend lists of indicators to assess medication safety and offer guidelines for their implementation, only five studies were identified in Pubmed, Embase and Scopus databases describing the indicator development process for drug related quality aspects.18,–,22 While the Delphi expert panel process was ubiquitously used in all studies for the ranking of the indicators provided, the choice of the indicators submitted for expert opinion were based on a wide variety of selection criteria, ranging from ranking in the literature to relevance in clinical outcomes.
In this paper, we describe a practical strategy for the development of a set of 15 medication safety indicators for the inpatient setting, using the assessment of pain management as an example.
Indicator development process
An overview of the indicator development process is displayed in figure 1.
Systematic literature review and indicator development strategies.
Identification of pain related indicators
The following three strategies were used for the compilation of a list of pain therapy related indicators
Literature review
A literature review was undertaken in order to identify recommended indicators assessing medication safety. PubMed (1966–April 2011), EMBASE (1972–April 2011) and Scopus databases (1823–April 2009) were searched for the following free text terms:
▶ Indicator AND (drug safety OR medication safety OR drug related problem OR adverse drug event OR adverse drug reaction OR medication error)
▶ Trigger AND (drug safety OR medication safety OR drug related problem OR adverse drug event OR adverse drug reaction OR medication error)
No language restrictions were applied. References from the retrieved articles were also reviewed to search for additional strategic reports.
The inclusion of 24 references allowed the compilation of a list of 333 different medication safety indicators.18 ,20 ,21 ,23,–,43
Subsequently, among the 333 medication safety indicators, 134 pain therapy related indicators were identified and extracted.
Analysis of a critical incident reporting system
The CIRRNET database of the Swiss Patient Safety Foundation (www.cirrnet.ch), encompassing 1557 critical incident reports from 37 Swiss hospitals (1998–29 March 2011) was systematically searched for pain therapy related incidents.
The retrieved incident reports were categorised for drugs involved and processes (prescribing, preparing, dispensing/administration, monitoring, documentation, logistics) affected by errors. Subsequently, they were prioritised for the severity of the outcome. The most severe incidents were analysed for causes and translated into indicators. Duplicates with indicators identified in the literature review were eliminated, yielding a total of six new indicators.
Semistructured interviews with experts in pain management
Semistructured interviews with experts in pain management (one intensive care anaesthesiologist, one chronic pain anaesthesiologist, one geriatric specialist, one pain nurse) were conducted. The following questions represented the core questions of the interviews:
▶ Where do you see the most critical aspects of pain therapy?
▶ Where do the most errors related to pain therapy occur?
▶ Which analgesic drugs are especially prone to drug related problems?
▶ Which drug combinations in pain therapy do you consider the most risky?
▶ Which aspects should be standardised by protocols because they are critical process steps for a safe analgesic therapy?
▶ Which indicators to assess the quality of pain therapy management would you personally implement (for example, based on guidelines implemented in the inpatient setting)?
Analysis of the interviews yielded additional critical process aspects, being translated into 32 indicators not yet mentioned in the literature.
The indicators resulting from the three strategies applied were compiled into a list and assigned to the five following indicator categories for organisational purposes, following the systematic of dimensions proposed by Donabedian: processes, institutional structure and outcome/indication quality split into the subcategories clinical event, biochemical/biophysical data and pharmaco-/epidemiology.44
These indicators were subsequently reviewed by a focus group of two pharmacists (one clinical pharmacy expert, one medication safety expert) and one pharmacy student, and prioritised according to the following criteria45,–,49:
▶ Relevance for the assessment of the quality of medicinal pain therapy in the inpatient setting
▶ Prevalence of the problem assessed
▶ Inherent capacity for improvement
▶ Suitability for the assessment of good quality
▶ Unambiguity for assessment by experts in a Delphi process
Indicators were considered relevant if they had the ability to assess one or several of the following quality aspects of medicinal pain therapy: accessibility of care, therapy appropriateness, continuity of care, effectiveness, efficacy, efficiency, safety and/or timeliness.50
In favour of innovation, the availability of already existing routine data for the measurement of the indicator was not considered a priority.
Fifty-six indicators were chosen by the focus group for the preliminary indicator set submitted to an expert panel for assessment; six indicators addressed clinical events (two new indicators—ie, not derived from the literature), 11 biochemical/biophysical parameters (three new), 16 pharmaco-/epidemiological aspects (four new), nine process elements (seven new) and 14 assessing the institutional structure (four new).
Delphi-like process
The 56 remaining indicators were evaluated in a Delphi-like process among a panel of Swiss pain experts. For the first round, 13 experts were invited to participate in the Delphi-like process.
Experts were chosen according to the following criteria:
▶ Experts in pain therapy in their field of medical specialty and/or
▶ Experts in clinical pharmacy and/or
▶ Experts in medication use evaluation and/or
▶ Experts in the field of drug therapy related indicator development/application and
▶ Willingness to participate in an interdisciplinary quality improvement project, and
▶ Availability during the timeframe envisioned for the Delphi process
Participation was requested via email 2 months ahead of time. The experts were chosen based on their expertise in various fields of pain management.
The Delphi form in an Excel table format was sent to the experts with a 2 week turnaround time. Completion of the questionnaire required approximately 1.5 h. An excerpt of the Delphi form, encompassing indicator suggestions, is displayed in figure 2.
Delphi form (excerpt) (adapted from Matlow et al19).
Eight experts returned the Delphi form (response rate 62%): five clinical pharmacists (one with vast experience in indicator development, testing and use), one paediatric pharmacist, one anaesthesiologist (specialised in chronic pain management) and one orthopaedic surgeon. The following specialists did not return the Delphi form: one specialist in geriatric medicine, one rheumatologist, one oncologist, one pain nurse and one anaesthesiologist.
Statistical evaluation of the results was based on calculation of a risk priority number (RPN; multiplication of values attributed for frequency, risk, detectability) adopted from a recent publication by Matlow et al.19 Frequency (potential for improvement in clinical practices) could be attributed with values ranging from 1 (very rare) to 5 (very common), risk (impact on patient outcomes) with values ranging from 1 (not at all serious) to 5 (very serious) and detectability (feasibility of abstracting the data) with values ranging from 1 (not at all easy) to 5 (very easy).
The method was complemented by the introduction of an additional characteristic to be judged by the experts: the relevance of the indicator (ability of assessing qualitatively important aspects of pain therapy) with values ranging from 1 (not at all relevant) to 5 (very relevant).
The indicators were ranked based on (1) their attributed RPN and (2) on their attributed relevance.
(1) Indicators were directly included in the final set of 15 if their mean RPN was higher than the mean RPN of all indicators plus the mean standard deviation (SD) of all indicators. Indicators were dropped if their mean RPN was lower than the mean RPN of all indicators minus the SD of all indicators. Seven indicators were directly included into the final set based on this selection criteria; six indicators were dropped. In order to complete the set of 15 indicators, the remaining indicators ranking in the middle were additionally assessed for their relevance.
(2) The second round of Delphi was conducted in the setting of a focus group meeting. The focus group, consisting of two pharmacists and one pharmacy student, assessed the remaining indicators based on the ranking of their relevance by the expert panel; eight more indicators were included.
The indicators included in the final set of 15 are displayed in table 1.
Final set of 15 indicators
Discussion
The indicator development process is a multidimensional process that needs thorough longitudinal planning. A combination of scientific research approaches (specifically for the literature review and the conceptual approach to the assessment of a specific drug class) and project management are required.
While the literature offers detailed guidance on the implementation of predefined indicators, sources recommending indicator development processes are scarce and there is no gold standard available. So far, the influence of different development strategies on indicator quality is unknown.17
We therefore designed our own indicator development process, consisting of two subprocesses: (1) compilation of an indicator set based on a literature review, analysis of critical incident reports and semistructured expert interviews and (2) execution of a two tiered Delphi-like process, consisting of the first round of Delphi in an expert panel and the second round in a focus group, in order to evaluate the indicators proposed.
While indicator sets proposed in the current literature are often focused on specific medical specialties or patient groups, no comprehensive set for the overall assessment of the quality of pain therapy was available.
Remarkably, very few of the 134 pain related indicators recommended in the literature were effectively tested for positive predictive value, sensitivity and specificity. Indicators were often derived from treatment guidelines; cut off values for biochemical and biophysical indicators were rarely defined. Correlation to actual outcomes was infrequently provided in a numerical format or clearly evidence based. This limits the information available to the experts for the evidence based ranking of indicator relevance in a Delphi process. In addition, this complicates the effective use of indicators recommended in the literature because every institution needs to go through additional consensus processes among experts for detailed indicator definitions which are often purely based on face validity.
The interviews helped identify important aspects of pain therapy and error prone processes. The interviewed specialists were often very focused on their specialty fields, supporting the approach of indicators addressing specific medical specialties or patient groups seen in the literature.
The systematic analysis of a CIRS yielded additional indicators, based on prioritisation of error prone drugs and process steps. The drug most often causing DRPs was fentanyl, followed by morphine in the opioid group and paracetamol (acetaminophen) among non-opioid analgesics. This might be explained by the fact that most reports were derived from anaesthesiology departments. Processes most often affected by errors were administration (45.3%) and prescribing (31.2%), reflecting the current status reported in the literature.11
Overall, this combination of methods to identify established indicators and to develop new ones yielded a list of indicators with very little overlap.
Current expert opinion was collected in the format of a two tiered Delphi-like process. There is general agreement in the literature that the Delphi method might be a useful consensus mechanism for practice oriented questions in the absence of sufficient scientific data.51,–,53 The systematic combination of evidence based scientific literature with expert opinion can create knowledge.54 However, the results depend largely on the composition of the expert panel and the knowledge based preparatory work provided to the experts.54 Detailed planning and definition of the Delphi process in a protocol is key to guarantee a valid outcome.51 Interestingly, indicators belonging to all organisational categories were represented in the final indicator set, although structure and process indicators are less represented in the current literature.
A limiting factor regarding the quality of indicators chosen for the final pain assessment indicator set might be the familiarity of the experts with the indicator technology itself. In a survey conducted among hospital pharmacists in 2010, only 33% of hospital pharmacy directors considered the indicator technology for use, and 38% were not familiar with the indicator technology.55 This finding might also be applicable to the expert panel and influence the judgement of indicator suitability. In addition, due to the lack of data on indicator validity, face validity might not be confirmed in effective clinical testing and application of the indicators proposed.
A second full round of Delphi instead of a focus group discussion should be considered if expert consensus instead of simple ranking is required. In the first round of this Delphi process, inter-rater agreement was limited and RPN encompassed a broad numerical spectrum. Consequently, several expert rounds and a broader timeframe might be necessary to achieve consensus. As an alternate approach to expert consensus, the RAND/UCLA appropriateness method or a more general nominal group technique might be considered.56 ,57
Conclusion
To our knowledge, this is the first study describing the process of developing quality of drug therapy related indicators in its entirety. In addition, no specific indicator set for the assessment of the quality of pain management has been suggested in the literature to date.
The two tiered process, involving indicator development based on a literature review, CIRS analysis and semistructured expert interviews, combined with a Delphi process for the ranking of the indicators suggested in this manuscript, allowed efficient completion of the project within 6 months. As a next step, the set of 15 indicators identified should be tested and validated in a clinical setting.
Key messages
Indicators are an efficient method of proactively addressing medication safety. A combination of literature search, analysis of a CIRS and expert interviews was useful for the development of indicators assessing pharmacological pain management.
Acknowledgments
The authors would like to thank Dr Anne G Matlow and Virginia Flintoft, MSc, BN, University of Toronto, Canada, for generously sharing their method and expertise; the experts participating in the Delphi process; and the steering committee of the critical incident reporting system CIRRNET, operated by the Swiss Patient Foundation, for their approval of the use of the database for research purposes.
References
Footnotes
-
Competing interests None.
-
Provenance and peer review Not commissioned; externally peer reviewed.