Article Text
PDF
Abstract
Objectives To determine the face and content validity of items for measuring safe medication practices in Portuguese hospitals.
Methods 128 items were drafted from content analysis of existing questionnaires and the literature, employing preferred terms of the WHO International Classification for Patient Safety (Portuguese version). A two-round e-Delphi was convened, using a purposive multidisciplinary panel. Hospital-based experts were asked to rate the relevance of items on a 7-point Likert scale and to comment on their clarity and completeness.
Results The response rate was similar in both rounds (70.3% and 73.4%, respectively). In the first round 91/128 (71.1%) items reached the predefined level of positive consensus. In the second round 23 additional items reached positive consensus, as well as seven items newly derived by the panel.
Conclusions Most items have face and content validity, indicating relevance and clarity, and can be included in a future questionnaire for measuring safe medication practices in Portuguese hospitals.
- Delphi
- Portugal
- patient safety
- medication
- safe practices
Statistics from Altmetric.com
Introduction
A systematic review including a total of 74 485 patients showed that nearly one out of 10 patients had an adverse event (AE) during hospital stay; adverse drug events (ADEs) were the second most common type of event, after surgical AEs.1 Almost half of the AEs were deemed preventable.1 A Portuguese pilot study is in accordance with these findings.2
Prevention of ADEs is imperative in view of their clinical and economic impact and also other unwanted effects, such as patient dissatisfaction and loss of trust in health professionals.3
Safe medication practices aim to reduce the risk of ADEs. If embedded in the systems approach, by targeting “the person, the team, the task, the workplace and the institution as a whole”,4 they may decisively contribute to reduction of the ADE burden. Measurement of safe medication practices is an important step towards improvement.
Existing measurement instruments should be adapted to each country owing to differences in cultural and clinical practice.5 To our knowledge there is no instrument for this purpose adapted to the Portuguese situation. It has been suggested that the design of measurement instruments should include three stages: domain selection, item development and instrument construction.5 Items should exhibit face and content validity.5
The aim of this study is to determine the face and content validity of items to measure safe medication practices in Portuguese hospitals.
Methods
Instrument domains were identified based on a conceptual framework by Woloshynowych et al6: environment; tasks; organisation; staff; practices related to the patient and others.
The starting point for item development was content analysis of available instruments and a literature review. A total of 128 items were drafted employing the preferred terms of the WHO International Classification for Patient Safety (Portuguese version).7
Both face and content validity may be tested by review by an expert panel;5 in this study we chose the e-Delphi technique.8 An expert was defined as a health professional (physician, nurse, pharmacist and pharmacy technician) working in hospital and regularly dealing with medicines. Additional criteria were professional credentials and/or status within the profession. In view of the inclusion of four different professional groups, the need to ensure variability in geographical location within the panel, hospital affiliation (teaching/non-teaching) and clinical specialties, and the likely occurrence of attrition, we set a target of 120 experts. Recruitment of experts was carried out by the Portuguese directorate-general of health.
The first round of open-ended questions was replaced by previously derived items. We decided to restrict the number of rounds to two, owing to the length of the Delphi questionnaire. In addition to 128 items, the first round included two reliability items inserted towards the end of the questionnaire, to ascertain participants' fatigue: one was a repeated item and one an item designed not to represent safe medication practice. Experts were asked to rate the relevance of each item on a Likert scale, from 1 (definitely not relevant) to 7 (definitely relevant); there was also space for comments on the completeness of each item, its wording or the reasons underlying the rating. The final sections of the Delphi questionnaire consisted of space for new items and demographic data.
Consensus was defined at the outset as 75% or more experts scoring 6 or 7 (positive consensus) or 1 or 2 (negative consensus). Failure to meet these criteria meant that consensus was not achieved.
In round two, for each item the panel was supplied with numerical feedback (overall median score, the median score for the expert's professional group and his own rating) and qualitative feedback (summary of comments). Additionally, eight new items were added based on experts' suggestions.
Data were input onto a SPSS database and accurateness verified through a random check of questionnaires. Missing values were excluded from analysis. Intergroup comparisons were performed with Pearson's χ2 test for a significance level of α=0.05.
After the second round, experts' comments were collated and their content analysed to refine the wording of the items.
Results
A total of 135 experts were invited to participate. The first round was concluded by 95 panelists (70.4%), whereas 73.7% (70/95) completed the second round. Attrition rate was roughly similar in the four professional groups, ranging from 23% for pharmacists and 29% for physicians.
The panel who concluded the second round comprised 34% (24/70) of nurses, 24% (17/70) of pharmacists, 24% (17/70) of pharmacy technicians and 17% (12/70) of physicians; more panelists were female (46/70; 65.7%). The mean age of experts was 45 years (range 27–62) and mean professional experience was 21 years.
In round two 71% (91/128) of the items had a median score of 7 (definitely relevant) and 27% (35/128) achieved a median of 6. Likewise, the new items were considered relevant; the majority (6/8) obtained a median score of 7 and two had a median of 6.
As depicted in figure 1, 114 items (89%) obtained positive consensus and no item achieved negative consensus. In addition, seven of the eight items introduced in round two achieved consensus. The repeated reliability item obtained similar median scores as that presented at the start of the questionnaire while the item designed not to represent safe medication practices did not reach consensus.
Overview of the Delphi results.
Panelists' comments shed light on possible reasons for dissent. For example, on the item concerning the 24-hour availability of pharmaceutical services some experts considered that an ‘on-call’ pharmacist 24/7 would suffice. For standardisation of drug administration times, a view was expressed that such a procedure was not patient-centered, serving mainly the staff's convenience. On the use of bar coding for patient identification, comments illustrated the belief that effective patient identification can be achieved through other procedures.
In round two agreement and consensus differed between the four professional groups (p=0.000): physicians approved by consensus only 88 items, whilst pharmacists approved 114 items. Table 1 provides examples of consensus obtained in the different professional groups.
Examples of scores obtained by consensus-approved items per professional group
Subgroup analysis highlighted items considered definitely relevant by all panelists of each professional group. Pharmacists, nurses and pharmacy technicians unanimously assigned a score of 7 to three items. No item was awarded a unanimous score of 7 in the physicians' group.
Discussion
This study was devised to determine the face and content validity of items for measuring safe medication practices in Portuguese hospitals. Positive consensus was obtained on 114 items, which indicates relevance and clarity. Refinement of the wording of items in light of experts’ comments further enhanced their content validity. Endorsement of many safe medication practices accepted internationally by Portuguese experts is a new finding. Another strength of this study is the relatively large nationwide panel convened. For example, in a systematic review on the use of the Delphi method for selecting healthcare quality indicators, the median number of participants was 17.9 Although there are no definitive guidelines on panel size, larger panels may improve reliability.
One limitation of this study is the lack of sound evidence underpinning some items. Consensus reflects experts' opinion and should not be regarded as unconditional truth. For example, ‘tall-man’ letters were endorsed by panelists as a way of preventing mix-ups with look-alike, sound-alike medication. Although encouraged by WHO10 and the Institute for Safe Medication Practices,11 the evidence supporting this practice consists primarily of laboratory-based studies, using name recognition as an outcome measure (and not medication incidents or ADE prevention). A recent before-and-after study in 42 paediatric American hospitals on the frequency of these errors after ‘tall-man’ lettering was introduced found no statistically significant change in either their intercept or rate for 11 look-alike, sound-alike pairs.12 These findings call for further research in adult populations to determine for which drug pairs, if any, ‘tall-man’ letters are effective, and highlight the need for multimodal error prevention practices.
This study was not set up to construct an instrument to measure safe medication practices; given the number of potentially important items, a stepwise approach was adopted, focusing first on item selection. This is in line with existing approaches and should not be seen as a limitation.3
Subgroup analysis showed the existence of a statistically significant difference in the number of consensus-approved items between professions. This influence of panel composition has been described9 and supports our decision to use a heterogeneous panel.
Despite attrition, which is common in Delphi panels, we achieved a response rate >70% in both rounds. According to Walker and Selfe, this is the minimum threshold required to maintain rigour in data collection.13 The questionnaire length might have been an important factor in explaining attrition.
Conclusions
Most items subjected to scrutiny reached the predefined level of positive consensus, indicating relevance and clarity for measuring safe medication practices in Portuguese hospitals.
Future steps are needed to develop an instrument for this purpose with sound psychometric characteristics. However, the safe medication practices endorsed by this panel may be useful to identify and prioritise areas in need of improvement by hospitals.
Acknowledgments
Many people have contributed to this study; a particularly warm thank you goes to panel members.
Footnotes
Contributors MG initiated the idea, implemented it, and wrote the paper. MP collected data, analysed the results and revised the manuscript. AC, AG and MJG supervised the collection of the data and revised the paper. CB contributed to the revision of the questionnaire and the writing of the manuscript. SdOM supervised the execution of the project, analysed the data and wrote the manuscript.
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement The data is available on application to the corresponding author.