Article Text
PDF
Abstract
Objectives Clinical pharmacists’ interventions (PIs) are an important element in ensuring good pharmaceutical care. We aimed to develop and validate a comprehensive multidimensional tool for assessing the potential impact of PIs for daily practice of medication review.
Methods Experts of the French Society of Clinical Pharmacy (SFPC) developed the CLinical, Economic and Organisational (CLEO) tool, consisting of three independent dimensions concerning clinical, economic and organisational impact. They were asked to analyse 30 scenarios of PIs, and re-rated 10 PIs with a washout of 1 month (internal validation). Then, seven external experts not involved in the development of the tool rated 60 scenarios collected when using the CLEO in daily practice. Inter- and intra-rater reliabilities were determined by calculation of the intra-class correlation (ICCA,1). Users’ satisfaction and acceptability of the tool were assessed on a 7-level Likert scale with a 17-item questionnaire.
Results For internal reliability, the inter-rater reliability for the CLEO tool was good for clinical dimensions (ICCA,1=0.693), excellent for economic dimensions (ICCA,1=0.815) and fair for organisational dimensions (ICCA,1=0.421); and the intra-rater reliability was good for clinical dimensions (ICCA,1=0.822), excellent for economic dimensions (ICCA,1=0.918) and good for organisational dimensions (ICCA,1=0.738). For external reliability, the inter-rater reliability was good for clinical dimensions (ICCA,1=0.649), excellent for economic dimensions (ICCA,1=0.814) and fair for organisational dimensions (ICCA,1=0.500). CLEO was viewed as relevant (mean±SD 4.93±1.27), acceptable (4.81±1.78), practicable (5.56±1.45) and precise (5.38±1.47).
Conclusions CLEO is a comprehensive tool assessing clinical, economic and organisational impacts of PIs which has been developed, validated and was reliable and feasible for use in routine clinical practice.
- documentation
- pharmacy service
- hospital
- quality of health care
- safety
- professional competence
Data availability statement
Data are available upon reasonable request. All data relevant to the study are included in the article or uploaded as supplementary information. Data are available upon reasonable request.
Statistics from Altmetric.com
Introduction
Pharmaceutical care which aims to optimise medicine use and improve patients’ health outcomes is the main role of clinical pharmacists. Of these, a patient's medication review with major outputs in the form of pharmacist interventions (PIs) is an important activity.1 Any action by a pharmacist that directly results in a change in patient management or treatment is defined as a PI.2
Many methods have been reported for documentation of PIs, including process-related indicators and outcome-related indicators such as potential or actual impacts of drug-related problems (DRPs) or PIs.3 Pharmacists often evaluate the potential impact of a PI because of convenience. However, there is no comprehensive, reliable, pragmatic and validated tool for this approach. Some tools focus only on clinical impact and few tools have been tested for their reliability.3 The clinical dimension of the tool developed by Hatoum et al 4 is most commonly used. However, the original tool developed by Hatoum et al has not yet been tested for reliability and its adapted versions proposed in other studies were either not tested or were unreliable. We should assess clinical and also economic, humanistic (ie, patients’ health-related quality of life and satisfaction) and organisational impacts of IPs. According to a recent review,3 this type of tool should be multidimensional and comprehensive; well-structured, open, numeric, hierarchical, quick and easy-to-use; as well as validated and reliable in tests.
The French Society of Clinical Pharmacy (SFPC) provided a scale for documenting PIs in hospitals5 and then integrated it into a website called Act-IP.6 This tool has successfully encouraged pharmacists to document PIs in their daily activities, including patient information, description and classification of DRPs and PIs and physician’s acceptance. However, there is a growing need for evaluation of the impacts of PIs to justify these activities. We therefore aimed to design an optimal tool in order to assess the potential impacts of PIs and to validate this tool in use.
Methods
Development of the multidimensional tool
Seven experts (clinical pharmacists from six different hospitals in France, all with at least 12 years of clinical experience) who are members of the SFPC expert group discussed the approval of the composition of the tool. The development of the multidimensional tool was based on a review of previous models and tools for the assessment of PIs3 and the experience of clinical pharmacists. The ‘structure-process-outcome model’ proposed by Donabedian7 stated that the quality of healthcare interventions should be evaluated by three groups of indicators related to ‘structural features’ (the system in which healthcare is provided); ‘process of care’ (the way by which healthcare is provided); and ‘outcomes’ (the results of the healthcare). The model developed by Kozma et al 8 grouped the ‘outcomes’ of pharmacy activities into clinical, humanistic and economic outcomes. Furthermore, economic outcomes9 take account of cost savings (savings from the direct cost of the modified therapy), cost avoidance (avoidance of additional health resources to manage a potential adverse outcome such as an emergency admission) and costs of conducting a PI (eg, pharmacist wages). We recognise that an individual PI rarely has any effect on structure, but rather on process and outcome. To take into account of all these factors, we developed a new multidimensional tool consisting of three dimensions, as reflected in its name CLEO (CLinical, Economic and Organisational). The CLEO tool consisted of three dimensions (figure 1):
The CLinical dimension aimed to assess the impact of the PI on the patient’s health from the point of view of the patient. It was scored using six levels—negative, null, minor, moderate, major and avoiding a fatality— from −1 to 4, respectively, and coded as ‘score C’. We based its structure and content on two previous tools: (1) a 6-level structure from the most frequently adapted tool for evaluation of the significance of the PI by Hatoum et al 4; and (2) the content of levels from the National Coordinating Council on Medical Error Reporting and Prevention index,10 a well-known tool for evaluating the severity of medication errors.
The Economic dimension aimed to assess the impact of the PI on the direct cost of treatment from the point of view of the hospital. It had three levels—negative, null and positive—scored −1, 0 and 1, respectively, and coded as ‘score E’. We chose to evaluate only cost savings but not cost avoidance, since the tool needed to be simple to use and the clinical dimension already partially reflects cost avoidance. The economic dimension was similar to the tool developed by Briceland et al. 11
The Organisational dimension aimed to evaluate the effect on the process of care from the healthcare providers’ perspective, such as time saving, decrease in workload and improvement of workplace safety. It had three levels—negative, null and positive—scored −1, 0 and 1, respectively, and coded as ‘score O’. We used organisational indicators mentioned in various existing tools.3
The CLinical, Economic and Organisational (CLEO) tool. PI, pharmacist intervention.
Overall, the CLEO tool was characterised by the following properties: some terms/indicators were defined right under each dimension; each dimension consisted of 3–6 levels and had negative, zero and positive levels. These dimensions were grouped into a three-component code (C, E and O) describing the overall impact of the PI. Each dimension could be entered as ‘undetermined’ for cases in which the available information was insufficient to assess the impact, as in the tool of Dale et al. 12 Some examples of judgement of the effects of PIs using the tool are shown in figure 2.
Examples of assessment of impacts of pharmacist interventions using the CLinical, Economic and Organisational (CLEO) tool.
Validation of the multidimensional tool
We validated the new multidimensional tool in two parts: internal reliability and external reliability (table 1).
Process of development and validation of the CLinical, Economic and Organisational (CLEO) tool
Internal reliability
First, we tested inter- and intra-reliability by seven internal clinical pharmacists as experts involved in the development of the tool who had more than 12 years of experience. The inter-rater reliability of the tool was tested by asking each of the pharmacists to assign scoring codes to 30 scenarios collected from daily pharmacy practice. The intra-rater reliability of the tool was tested by randomly selecting 10 of the 30 PIs used in the second step to be rated a second time by the same experts 1 month later.
External reliability
We tested the inter-rater reliability between pharmacists who used the CLEO tool in daily practice and seven external experts with more than 10 years' experience who were not involved in the development of the tool. We invited the seven external experts to read 60 scenarios and to rate the impact of the tool. The inter-rater reliability was calculated from the concordance between the pharmacists and external experts. All raters were provided with the CLEO tool and some examples to form a judgement (figures 1 and 2).
Data collection
French clinical pharmacists collected and recorded PIs performed during routine clinical pharmacy services into the Act-IP website (http://www.actip.sfpc.eu/actip) according to a standard and validated form designed by the SFPC in 2006 (see online supplemental appendix 1).5 This form asked pharmacists to provide the patient characteristics (age, sex, weight), intervention-related drugs and a description of the context including medical information such as medical and medication histories, diagnosis and examination results, relevant data, the identified DRP, the pharmacist’s suggestion, classification of DRPs and PIs, and the physician's acceptance at the time of intervention. Over 700 000 PIs were submitted by 3300 pharmacists from 1100 hospitals in France during 2006–2020.
Supplemental material
The 30 scenarios and 60 scenarios were chosen to ensure a balance in the types of DRPs and PIs, and extracted from the Act-IP recorded from 1 July to 30 September 2014 and from 1 July to 30 September 2019, respectively. The CLEO has been integrated into the Act-IP website since 2016, so the latter 60 scenarios had pharmacists’ assessments of the impact of PIs based on the CLEO. To demonstrate interpretability and future applications, we performed descriptive statistics on the dataset obtained.
Appropriateness, acceptability, feasibility and precision
The appropriateness, acceptability, feasibility and precision of the CLEO tool was also evaluated during the external validation step. The questionnaire, which included 17 questions, and the 7-point Likert scale to evaluate the level of agreement (completely disagree=1, disagree=2, more or less disagree=3, undecided=4, more or less agree=5, agree=6, completely agree=7) were adapted from the questionnaire developed by AbuRuz et al. 13 We also asked for the average time for rating the majority of PIs.
Statistical analysis
To investigate the inter- and intra-rater reliabilities, the intra-class correlation (ICCA,1) of a single measurement, absolute agreement, two-way mixed-effects model was calculated. An ICCA,1 value was considered poor, fair, good and excellent with values of <0.04, 0.40–0.59, 0.60–0.74 and 0.75–1.00, respectively.14 For inter-rater reliability we calculated the sample size of scenarios based on the method of Walter et al. 15 Sample size calculation was derived from the formula of the ICC test using the PASS software. When alpha and power are fixed at 0.05 and <80%, respectively, a minimum sample size of 30 scenarios was needed to detect the value of the ICC with a number of raters of 7. Since this test-retest reliability will only involve two observations, the minimum number of samples required will be 10 for detecting ICC values of 0.7.16 Furthermore, we chose 10 scenarios in order to present 10 types of DRPs (according to the classification of the SFCP). Statistical analysis was conducted using the SPSS 22 software.
Results
We successfully developed a tool for comprehensively evaluating the impact of PIs. The reliability of the tool was then assessed by both internal and external processes with 30 and 60 scenarios, respectively.
Data interpretation
The characteristics of 30 and 60 scenarios and frequencies of all PI evaluations by the CLEO tool are shown in table 2. Scenarios covered mostly elderly patients (73.3% and 83.3%, respectively), female patients (73.3% and 58.3%, respectively) and were collected mainly in long-term care (56.7% and 71.7%, respectively). The 30 scenarios and 60 scenarios were chosen to ensure balance in the types of DRPs and PIs, so the frequencies of each type of DRP and PI ranged from 5.0% to 15.0% and from 10% to 20%, respectively.
Characteristics of clinical cases selected for assessment and its impact based on the CLinical, Economic and Organisational (CLEO) tool
Most PIs related to the 30 scenarios were evaluated as having minor clinical relevance (26.2%), whereas most PIs related to the 60 scenarios were evaluated as having a major clinical impact (39.8%). PIs which were evaluated to decrease costs (44.3% and 62.3%, respectively) were classified as stopping a treatment (44.0% and 33.5%, respectively), dose adjustment (25.9% and 32,3%, respectively), substitution/replacement (10.3% and 9.6%, respectively), optimisation of dosing modalities (7.7% and 10.5%, respectively) or others (12.1% and 14.1%, respectively). A minority of the PIs that were evaluated to increase costs (16.2% and 9.8%, respectively) were starting/restarting a treatment (48.5% and 41.2%, respectively), followed by treatment monitoring (15.0% and 23.4%, respectively), dose adjustment (22.9% and 23.7%, respectively) and others (13.6% and 11.7%, respectively). The PIs were judged to have a positive relevance (49.1% and 50.0%, respectively), a negative revelance (5.7% and 3.3%, respectively) or no relevance (45.2% and 46.7%, respectively) within the organisational dimension of the CLEO tool.
Internal and external reliability of the CLEO tool
The results of the internal and external reliability of the CLEO tool are summarised in table 3. For internal reliability, the inter-rater reliability for the CLEO was good for the clinical dimension (ICCA,1=0.693), excellent for the economic dimension (ICCA,1=0.815) and fair for the organisational dimension (ICCA,1=0.421); and the intra-rater reliability was good for the clinical dimension (ICCA,1=0.822), excellent for the economic dimension (ICCA,1=0.918) and good for the organisational dimension (ICCA,1=0.738).
Internal and external reliability of the CLinical, Economic and Organisational (CLEO) tool
For external reliability, the inter-rater reliability was good for the clinical dimension (ICCA,1=0.649), excellent for the economic dimension (ICCA,1=0.814) and fair for the organisational dimension (ICCA,1=0.500).
Appropriateness, acceptability, practicability and precision
All seven external experts answered the 17-item questionnaire. The CLEO was viewed as relevant (mean±SD 4.93±1.27), acceptable (4.81±1.78), practicable (5.56±1.45) and precise (5.38±1.47). All items had an average score above neutral (figure 3). Five of the seven external experts took less than 1 min to rate the majority of PIs.
Raters' judgements on appropriateness, acceptability, practicability and precision of the CLinical, Economic and Organisational (CLEO) tool with mean (SD) scores on the 7-point Likert scale (completely disagree=1, disagree=2, more or less disagree=3, undecided=4, more or less agree=5, agree=6, completely agree=7). Q01, the CLEO includes all dimensions needed to evaluate a PI; Q02, the CLEO is an appropriate instrument for evaluating Pis; Q03, I did not have issues to evaluate the PIs with the CLEO; Q04, I could imagine to use CLEO in my work in the future; Q05, in general I am satisfied with the CLEO; Q06, there are enough evaluation levels in the dimension ‘Clinical Impact’; Q07, there are enough evaluation levels in the dimension ‘Economic Impact’; Q08, there are enough evaluation levels in the dimension ‘Organisational Impact’; Q09, the levels in the dimension ‘Clinical Impact’ are well defined; Q10, the levels in the dimension ‘Economic Impact’ are well defined; Q11, the levels in the dimension ‘Organisational Impact’ are well defined; Q12, the different levels in the dimension ‘Clinical impact’ are clearly separated; Q13, the different levels in the dimension ‘Economic impact’ are clearly separated; Q14, the different levels in the dimension ‘Organisational impact’ are clearly separated; Q15, the CLEO is easy to use; Q16, the training with case examples is sufficient to use the CLEO; Q17, the time expenses for an evaluation of a PI with the CLEO are reasonable.
Discussion
Development of the CLEO tool
It took 1 year to develop the final version of the CLEO tool. One pharmacist was responsible for developing the new tool and then discussed it with six experts to get a consensus on its structure and content. The following questions arose when deciding the final version of the tool:
Which impacts of a PI would we include in the tool? So far, most of the proposed tools focus mainly on evaluating the clinical impact of a PI and fail to measure other effects.3 The new tool needs to integrate clinical, economic and organisational impacts and to provide a multidimensional comprehensive framework for assessment.
How to evaluate the economic benefit of a PI? In most studies the economic impact of the PI is measured by a combination of cost savings and cost avoidance, and implementation cost. In our ‘Economic impact’ dimension we assess cost saving related to drug and monitoring costs, while cost avoidance is not considered here but reflected in the ‘Clinical impact’ (eg, avoidance of intensive care, prolonging hospital stay) if the PI were implemented.17 This allows the CLEO tool to be used for screening to select PIs for economic studies. This approach was used by Zecchini et al 18 in the setting of a cancer department to select PIs that induced positive and negative cost savings as part of their estimation of the total amount of money saved as a result of PIs.
Whether the tool should consist of a probability of impacts? An individual PI has some different impacts with different probabilities of them happening. Some tools integrate the estimation of the probability of consequences into the assessment procedure.19 Nonetheless, it is usually difficult to judge. Hence, the clinical dimension of the CLEO tool is assessed as stated in the most probable scenarios and does not obligate an estimation of a separate probability of each consequence. This approach has been used in a similar manner in other tools.20
How to use the tool in practice and in research? The independence of three dimensions helps to interpret the scoring results flexibly. Mongaret et al 21 and Duwez et al 22 used only the clinical dimension of the CLEO tool in their study to determine clinically significant PIs, while Zecchini et al 23 used only the clinical and economic dimensions. Another tool could take advantage of all three dimensions to demonstrate the whole effects of a PI. However, there is a question as to the way in which one should combine the three different scores/codes into a single overall score/code for the PI. Should we consider one dimension (eg, the economic or organisational impact) as less important than the other impact (eg, clinical impact)? If so, how should we weight the different impacts and should the same weight be used for the same types of PIs? There are 54 possible three-component scores of the CLEO tool. It is easy to say that PIs with 3C, 1E, 1O assessment have a more powerful impact than PIs assessed as −1C, −1E, −0O. Nonetheless, other scores are difficult to weight. We also have to consider the implementation costs (eg, pharmacists’ time). After validation of the multidimensional CLEO tool, the next step is to find a method of estimating the overall impact score.
Validation of the CLEO tool
Outputs of validation of a tool rely on many features such as content and structure of a scale, assessment process and characteristics of the raters.3 Therefore, a tool for the evaluation of potential significance of PIs such as the CLEO tool has to be tested for both inter-rater and intra-rater reliability, both internal and external validation. In 2018 Stampfli et al 24 translated the CLEO into German (CLEOde) and evaluated its inter- and intra-reliability and feasibility for implementation in daily routine clinical pharmacy practice by 10 pharmacists in three Swiss hospitals.
In our study, inter- and intra-reliability was good to excellent with a range in ICCA,1 of 0.649–0.822 for the clinical dimension of the CLEO, which was confirmed by the results of CLEOde with a range in ICCA,1 of 0.63–0.76,24 and which was higher than most other tools.3 The inter-reliability obtained in most other studies was slight to fair (k=0.14–0.31).25–27 The highest level of inter-rater reliability was found in a study by Overhage and Lake (kw=0.76).28 However, this encouraging result was not repeated when other groups adopted this tool.25 27 Because reporting clinical impact is indispensable when assessing the impact of PIs, and the clinical impact of the CLEO had the highest number of ordinal levels (six levels), the results of the reliability of the clinical dimension is promising.
In our tool, the economic impact aspect had the best score of inter- and intra-rater reliability with an excellent ICC1,A of 0.814–0.918. In the study by Stampfli et al, 24 inter- and intra-rater reliability of the economic impact was lower but still good with ICC1,A of 0.65 and 0.85, respectively. The reason for the high score is that pharmacists can base easily on direct cost-related modified therapy compared with the original therapy to rate the economic effect (for example, adding a drug has a negative impact while changing to a cheaper drug or stopping a drug or laboratory test is positive).
Th organisational impact received a fair to good score (range of ICC1,A 0.421–0.738), which was worse than the clinical or economic impact. We propose several reasons for this. It is not easy to rate many organisational indicators (eg, time saving, improvement of safety) from the points of view of the various people (nurses, pharmacists, doctors) in one single score. For example, asking a nurse to lengthen the duration of infusion of meropenem, which can demand more nursing time for supervision, may improve satisfaction for pharmacists and physicians. Therefore, suggestions are that pharmacists can use it as a suggestive practical tool for consensus-based rating or more training should be provided for pharmacists before rating in order to obtain good reliability.
For the high external reliability achieved when using the CLEO, our study suggested that individual clinical pharmacists can code their own interventions as they make them in daily practice, and their coding is likely to be consistent with other clinical pharmacists as experts or supervisors. Therefore, we integrated the CLEO tool into the Act-IP website in order to allow French pharmacists to submit their own assessment. However, as Batty and Barber stated, hospital pharmacists play a large part in medical audit, working with clinicians to identify prescribing problems and to set standards and monitor practice, and they have to develop performance indicators of prescription monitoring.29 Assessment of PIs by the CLEO may be a performance indicator which allows comparison of pharmacists' opinions and PI’s impacts among pharmacists, healthcare providers and sites and may be a valuable tool in auditing them. It is useful to assess PIs with periodic assessment by others.
Appropriateness, acceptability, feasibility and precision
Most raters spent no more than 1 min when rating the impact of one PI, which demonstrated the feasibility of its implementation. User satisfaction and acceptability with the tool was high. Indeed, the CLEO tool was used easily in French hospital settings21–23 and received good feedback from three Swiss hospitals.24 However, pharmacists suggested that training with sample cases and regular discussion of real scenarios in daily practice is necessary to improve its reliability.
Limitations
This research has some limitations. Maturation bias occurred because, over time, evaluators rated more consistently as they gained experience/familiarity in using the tool. Limitations of internal validation were a small sample size and biased raters, which were resolved by external validation. As evaluators were aware of the comparison of ratings of different raters, a testing bias could have arisen. Therefore, the evaluators were asked to give their true opinions.
Conclusions
The CLEO tool is a comprehensive tool which assesses the clinical, economic and organisational impacts of PIs. It has been developed, validated and was reliable and feasible for use in routine clinical practice.
Data availability statement
Data are available upon reasonable request. All data relevant to the study are included in the article or uploaded as supplementary information. Data are available upon reasonable request.
Ethics statements
Ethics approval
This study, which evaluated the impacts of PIs, did not modify the healthcare process of patients so ethical permission was unnecessary. However, the study protocol was reviewed by the Grenoble Alpes University Board (2015GREAS034).
Acknowledgments
We thank Dr Alison Foote (Grenoble Alpes University Hospital) for scientific editing services.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
EAHP Statement 4: Clinical Pharmacy Services.
Contributors All co-authors fulfill the criteria required for authorship. HTV, BA and PB were responsible for study conception. HTV performed data collection. HTV and SC performed data analysis and interpretation and drafted the manuscript. PB, BC, SC and BA revised the drafting for important intellectual content. All authors contributed with reviews of the final draft.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.