Article Text

other Versions


Original article
Strategy for development and pre-implementation validation of effective clinical decision support
  1. Anne-Marie J Scheepers-Hoeks1,
  2. Rene J Grouls1,
  3. Cees Neef2,
  4. Eric W Ackerman1,
  5. Erik H Korsten3
  1. 1Department of Pharmacy, Catharina Hospital, Eindhoven, The Netherlands
  2. 2Department of Clinical Pharmacy and Toxicology, Maastricht University Medical Center, CAPHRI, Maastricht, The Netherlands
  3. 3Department of Anaesthesiology, Intensive Care and Pain Relief, Catharina Hospital, Eindhoven, The Netherlands
  1. Correspondence to Anne-Marie J Scheepers-Hoeks, Department of Pharmacy, Catharina Hospital Eindhoven, Post office 1350, Eindhoven 5602 ZA, The Netherlands; anne-marie.scheepers{at}


Objective Well-designed clinical decision support systems (CDSS) can reduce the problem of alert fatigue by generating patient-specific alerts. This paper describes a strategy for the development and pre-implementation validation of specific and relevant clinical rules in order to reduce alert fatigue.

Methods A four-step development and validation strategy of clinical rules is presented. As an example, from March to September 2006 the ‘lithium therapy rule’ was developed with this strategy based on the Plan-Do-Check-Act cycle. 15 368 patients were retrospectively screened and 2503 patients were prospectively screened while the positive and negative predictive values (PPV/NPV) were continuously monitored. The first step is to confirm that the parameters used in the definitions are linked to the correct data in the electronic health record; the second step involves an expert team in the review process to assure that alerts generated are clinically relevant; in the third step the rule is adjusted to generate the right alerts in daily practice; and the fourth step ensures technical and therapeutic maintenance after implementation in practice.

Results From September 2006 to July 2010 nine other rules were developed following exactly the same strategy. The 10 clinical rules developed showed a progression during the development and all resulted in a final therapeutic PPV of ≥89% before implementation, based on expert opinion. NPV was determined for five clinical rules and was always 100%.

Conclusions The proposed strategy is effective for creating specific and reliable clinical rules that generate relevant recommendations. The inclusion of an expert team in the development process is an essential success factor. It is hoped that it will accelerate the widespread use of these promising decision support systems in practice.

Statistics from


Clinical decision support systems (CDSS) are considered powerful tools for improving patient safety and healthcare quality.1 ,2 They are defined as ‘computer-based information systems used to integrate clinical and patient information to provide support for decision-making in patient care’.3 CDSS are the next generation of medication safety systems and are able to narrow the gap between evidence-based medicine and practice.4 ,5 They generate patient-specific alerts using data stored in electronic health records (EHRs). Advanced CDSS, used in addition to basic CDSS, include, for example, checking contraindications (disease and drugs), individualised dosing support during renal impairment or guidance for medication-related laboratory testing.6 ,7

There is an urgent need for implementation of CDSS as medication errors occur frequently despite the use of medication safety systems.2 ,8 However, high rates of alert overrides (‘alert fatigue’) and low rates of alert adherence can hinder the success of well-designed CDSS.9–12 Alert fatigue compromises the intended increase in medication safety, with clinicians’ override rates varying from 49% to 96%.11 ,13 To prevent the pitfalls of alert fatigue, structured development and validation of the decision support algorithms (‘clinical rules’) are crucial before widespread dissemination into practice.9 ,14 To achieve fewer interruptive alerts, a solution is to select only alerts with the highest clinical importance.15 Few studies have concurrently examined the technical correctness of alerts and the corresponding clinical appropriateness of CDSS alerts.9 However, a need exists to share best practices in CDSS development and implementation.16 Two key components of validation strategy have been described in most studies: (1) the use of a multidisciplinary expert panel; and (2) off-line test and revision cycles.17–21 A framework recently published by McCoy et al describes a potentially effective method for assessing clinical appropriateness of medication alerts.9 A key attribute of this framework is that it determines appropriateness at the time of a triggered alert by applying expert knowledge. Weingart et al21 examined a subset of all displayed alerts to determine alert validity and expert agreement with overrides, although no measures of unintended adverse consequences were reported. Sucher et al mention factors that need to be tested, such as verification, validation and worst case testing, but these factors are not explained in detail.22 A practical validation approach is described by Osherhoff et al using cases and testing scenarios to validate clinical rules.23 However, the method has limited usefulness due to lack of a detailed description of the method and outcome values.

To prevent alert fatigue, CDSS implementers must monitor and identify situations that frequently trigger inappropriate alerts and take well-defined steps to improve alert suitability.9 Studies examining CDSS content validation often lack a complete and reproducible method that is demonstrably leading to appropriate alerts. We propose a strategy for the development and validation of clinical rules that generate patient-specific relevant alerts with a high predictive value.


Study site

This study was conducted in the Catharina Hospital in Eindhoven, The Netherlands, a 600-bed university-affiliated hospital. The hospital uses a hospital information system (EHR: CS-EZIS, Chipsoft BV, Amsterdam, The Netherlands) with integrated computerised physician order entry (CPOE). Most patient data (eg, medication, laboratory data, therapy, microbiology, diagnosis) are available in this system. However, the system also provides unstructured text notes describing the actual status of the patient, which cannot be used for decision support. The CPOE system includes basic drug-oriented decision support such as drug–drug interactions and drug dose checking, based on the nationally established electronic drug database (WinAp, G-standard, Den Haag, The Netherlands).12 Since 2004 the Department of Pharmacy of the Catharina Hospital has been involved in the development of a strategy for designing and validating clinical rules with an advanced clinical decision support system.

Decision support system

In this study the CDSS Gaston (Medecs BV, Eindhoven, The Netherlands) is used. This system, which is commercially available worldwide, was developed in 1998 at the Technical University Eindhoven in collaboration with our hospital. Technical assistance during our research was supplied by Medecs BV.

The CDSS Gaston is linked to our EHR, which allows the electronic data stored in the EHR to be used in clinical rules.24 ,25 The CDSS consists of two modules: (1) a guideline editor for development of electronic guidelines; and (2) a guideline execution engine. The editor is a user-friendly environment in which clinical rules are built as flowcharts. The steps in the flowchart contain the selection definitions based on the parameters that are available in the EHR. The engine is used for retrospective and prospective database research and prospective alerting.

Site set-up and participants

The strategy described was first applied in the study period from March 2006 to September 2006 for the clinical rule for lithium therapy. Lithium was chosen as it is a potentially hazardous drug with a narrow therapeutic window and many parameters need to be monitored.26 The method described in this paper will cover the development of this clinical rule. In the period from September 2006 to July 2010 nine other rules were developed following exactly the same strategy. These rules were all selected according to a national study that identified high-risk patients with medication-related problems leading to hospital admission.8

The development of the clinical rule was carried out by the research team, consisting of a pharmacist (in training) who built the clinical rule, a hospital pharmacist/clinical pharmacologist and a research pharmacist experienced in decision support. Time investment for this rule was 3 months full-time (spread over 6 months) for the pharmacist in training and 1 h a week for 6 months for the other two members of the research team. The clinical relevance was monitored by an expert team consisting of three specialised physicians (head of internal medicine, head of psychiatry and an anaesthesiologist specialised in computer technology) and an experienced hospital pharmacist, all of whom were experts in lithium therapy. The expert team invested 3 h for the plenary meetings with the research team and another 2 h for preparation and written consultation.

Outcome values

The applicability of the strategy was determined by measurement of the positive and negative predictive values (PPV/NPV) of the clinical rule (see table 1). PPV and NPV are used to express the quality of a clinical rule and to monitor the right balance between clinical relevance and over-alerting.15 ,27 In each step of the strategy the PPV and NPV were monitored. The research team decided whether an alert was technically (in)correct. The expert team decided whether an alert was therapeutically (in)correct, with the expert physician in psychiatry having the final decision. The outcome values measured in this study therefore represent expert opinion.

Table 1

Schematic representation of calculation of positive and negative predictive values

Positive validation

By testing the CDSS, alerts were generated which could either be correct (true positive) or incorrect (false positive). A PPV is calculated by dividing the number of true positives by the total number of alerts (table 1). The aim was to maintain a maximum NPV while minimising the amount of false positives to prevent alert fatigue.

Negative validation

We used our completely independent pharmacy system (Centrasys, Isoft BV, The Netherlands) to select the last 100 patients with a lithium prescription. The research team checked all these patients manually by reviewing their EHR data on all conditions of the clinical rule. For example, when a patient's lithium blood level needed to be monitored within 5 days after starting the drug, the EHR laboratory data were checked to confirm that this was performed. If so, this patient correctly did not receive an alert (true negative). If not, the research team checked that the CDSS generated an alert at that time. If no alert was generated, this would have been a false positive alert. The NPV was calculated by dividing the number of true negatives by the total number of patients without alerts (table 1). The NPV was added to the strategy at a later stage. It was therefore only measured for clinical rules that have been developed recently (clozapine, phenytoin, lithium, digoxin and aminoglycosides).


The different steps of the strategy were tested on the database of our EHR. The first two steps (retrospective) were tested on the historical database, including all 15 368 patients admitted in the 12 months prior to the start of the development period (March 2005 to March 2006). We felt that physicians’ acceptance would not be affected after implementation of the rule if no error was detected in the population of the preceding year. The third step (prospective) was tested on all 2503 patients admitted during the months of July and August 2006. The fourth step was performed on the same hospital database starting from the time of implementation in practice. These results are not included in this study.

Development and validation strategy for clinical rules: the Catharina Hospital approach

We developed a four-step strategy for the development and validation of clinical rules. To improve the quality of the clinical rule, the Plan-Do-Check-Act (PDCA) cycle for quality control is applied at least once in every step (figure 1). In steps 1–3 the reliability and quality of the rule are expressed as PPV and NPV.

Figure 1

Plan-Do-Check-Act (PDCA) cycle for the development and validation of clinical rules followed in each validation step of the described strategy to improve the positive and negative predictive values (PPV and NPV).

Step 1: Retrospective technical validation

The objective of the first step is to determine whether the parameters in the CDSS are linked to the correct parameters in the EHR creating technically valid definitions. For example, for the definition in the CDSS ‘The last lithium blood level is >1.2 mmol/l’, a check is made that the intended EHR data are transferred to the CDSS to correctly interpret the lithium blood level.

The first step starts by designing the clinical rule based on evidence-based guidelines (Plan). Protocols or literature were used if guidelines were unavailable. The guidelines were translated into a computer-interpretable format with measurable and specified parameters. The clinical rule was then tested on the retrospective database (Do). Results were analysed by the research team to determine the amount of true and false positives and negatives, and discussed in a plenary meeting with the expert team (Check). Here possible improvements were identified and later implemented by the research team (Act). If so, another pass through the PDCA cycle was required. The PPV and NPV were calculated with the final results. When the objectives were met (PPV>90% and NPV>95%), the second step was started.

Step 2: Retrospective therapeutic validation

The second step is to check whether all alerts are clinically relevant, actionable and judged to be useful by an expert team. Although alerts can be technically valid and based on evidence-based guidelines, physicians may not always consider them relevant or useful. Therapeutic validation is most important for gaining user acceptance.

This step starts with a meeting between the research team and the expert team to discuss the therapeutic value of the alerts and to determine the baseline therapeutic PPV and NPV (Check). The experts reviewed all alerts generated as being clinically relevant (true positive) or not (false positive). Differences between theory and practice were discussed and the expert team formulated modifications of the clinical rule (Act). After refinement (Plan), the rule was applied again to the same retrospective database as in step 1 (Do). In a second meeting, the research and expert teams evaluated the results of the adjusted clinical rule (Check). The updated therapeutic PPV and NPV were calculated to monitor the improvement in accuracy of the clinical rule during this development (figure 1). All proposed modifications were first technically validated according to step 1 before therapeutic validation in step 2 (Act). The results of the clinical rule were discussed with the expert team until the PPV and NPV were maximised (and above threshold) and the team indicated that no further adjustments were required. Two to three PDCA cycles were usually needed.

Step 3: Prospective pre-implementation validation

The third step is the preparation of the clinical rule for routine application in daily practice. The CDSS is now linked to the prospective (live) database of the EHR, allowing the generation of alerts of actual admitted patients. In step 3 the clinical rule is adapted to assure prospective timely alerting, integrated in the clinical workflow.

The expert team was consulted on the content of the message (eg, proposal, command), the recipient of the message (eg, nurse, physician, pharmacist), the frequency (eg, once daily, continuously) and the alerting method (eg, on-demand, automatic) (Check). When the rule was refined on these issues (Act, Plan), it was implemented into the operational EHR in a test setting (Do) for final validation. This validation is conducted prospectively using patients admitted to the hospital to provide a simulation of daily practice. In this step the rule may need some minor technical adjustments as the retrospective rule needs slightly different definitions to correctly select patients compared with the prospectively designed rule. This may influence PPV and NPV in a minor way, so they are calculated again. After a 2-month test period the results are again checked by the research team and discussed with the expert team to determine a final PPV. Maximising the PPV is the final step before implementing the clinical rule in practice. NPV was measured on one test patient ‘admitted’ to our hospital in whom a prescription of lithium and relevant laboratory values were added. The clinical rule with—its technical, therapeutic and prospective PPV and NPV—is then ready for implementation in daily practice.

Step 4: Prospective post-implementation validation

The fourth step, after implementation of the clinical rule in daily practice, is continuous maintenance. In this step the clinical rule is monitored while the rule is operational (in our case, in the pharmacy). It consists of technical and therapeutic maintenance to ensure continuous accuracy of the alerts. We found that, after implementation in practice, adjustments were needed for every clinical rule that were not foreseen during the development phase (steps 1–3).

First, technical adjustments may be necessary due to updates or new functionalities in the CDSS or EHR. These technical adjustments were developed, validated and implemented by the research team only (according to step 1). When the changes also had therapeutic consequences, the expert team was consulted.

Second, the content of the clinical rule should be updated regularly due to changes in the underlying evidence-based medicine or end users’ preferences. For example, when a new (monthly) update of the underlying drug database (G-standard) was available, clinical rules were checked on these changes and the proposed changes were applied to the strategy described. Clinical rules developed in this study were also checked in the pharmacy quality system, both the content of the rules and the strategy for development. This step finalises the strategy by continuously optimising the suitability of the rule in practice.


Using the strategy outlined above, we developed 10 clinical rules with a high PPV (table 2). All the clinical rules showed a progression during the development and resulted in a final therapeutic PPV of ≥ 89% before implementation (end of step 3).

Table 2

Ten clinical rules resulting from application of the strategy*

The clinical rules for clozapine, phenytoin, lithium, digoxin and aminoglycosides all had a NPV of 100%. This indicates that an alert was generated for all patients for whom an alert was expected and no patients or alerts were missing.


CDSS are increasingly used to improve patient safety, yet the appropriateness of displayed alerts and subsequent decision-making is not well understood.9 Two major categories of modulators for alert acceptance have recently been identified: alert content and alert presentation.10 Alert fatigue due to non-specific clinical rules may be an important causative factor.11 The application of our proposed strategy for the development and validation of CDSS content leads to clinical rules with a high predictive value (PPV/NPV). The development and validation of a clinical rule was shown to be labour-intensive as it took 4–6 months on average before implementation. The widespread use of CDSS could therefore be promoted by using national or international clinical rules. This strategy can facilitate the translation of evidence-based medicine into practical and relevant clinical rules.

A key attribute of the strategy is the involvement of an expert team, as already addressed by others.17–21 The expert team is consulted to overcome the problem of non-specific guidelines and the literature. In addition, the experts are able to determine clinically relevant conditions and refine the content.21 An additional advantage is that cooperation from physicians is assured by involving them in the development cycle. The dedication of the professionals and their time is required to realise these results.

A new feature of this strategy is the use of the PDCA cycle in combination with the calculation of the PPV and NPV during the complete development period. Baseline PPVs remained relatively low due to technical challenges and the fact that the rule was not adapted to end users’ wishes. Our strategy has overcome these problems. The PPV and NPV are suitable for determining technical and therapeutic reliability to indicate physicians’ acceptance of alerts generated by the CDSS. However, PPV and NPV values depend on the prevalence of the condition. The number of patients included in the retrospective database must therefore be large enough to detect possible errors.

An important parameter influencing the outcome values is the quality of the database. In our situation we have an extensive database which is suitable for the use of advanced CDSS. This is an important prerequisite, as it directly influences the PPV and NPV of the alerts generated. The results shown were corrected for 0–3% technical errors due to wrong input of data in the EHR by inexperienced users of the EHR. These are errors that are hard to prevent, especially in a hospital with many junior doctors.

By the use of our strategy, new applications of CDSS were developed and definitions refined, thereby raising the PPV of each clinical rule. A newly developed clinical rule always required new functionalities of the CDSS as well. The development of clinical rules made it necessary to add new parameters such as diagnoses, temperature and planning of surgical operations to our EHR. These examples show that the technical assistance of the developer of the CDSS was essential. The purchase of our CDSS was therefore accompanied by a service pack to support technical implementation in our hospital.

When the clinical rules were validated retrospectively, they were applied to a large database (step 2). The resulting alerts were used to predict the impact of the rule on medication safety. An estimate was obtained of the type(s) of medication errors that are prevented when the clinical rule is used in practice. The strategy used focused on pre-implementation validation and its results and therefore the results of step 4 are not included in this paper. Future publications will cover these results, but preliminary results show that a substantial increase in medication safety can be achieved.7

Limitations of the study

The strategy described has only been tested in a single hospital. Testing of the outlined strategy on a larger scale is needed. Due to technical barriers, initially only the PPV and not the NPV were measured in this study so only pre-implementation PPV values are shown in table 2. We have recently added the measurement of NPV to strengthen the validation. However, well-founded screening revealed no false negative patients, indicating that all patients were correctly identified by the CDSS. More research is needed to translate the PPV and NPV results into daily practice. Interpretation of the results of the post-implementation step is therefore necessary as we saw that, after implementation, a number of other factors influence acceptance of the alerts in practice. Various factors influence the effective application of decision support.10 ,28 For example, it may be influenced by the type of alert. Literature on this topic is limited, but indicates that active alerting is more effective than passive alerting.10 ,29 Comparative studies describing different alerting types have not yet been performed.

Another factor is the threshold setting for notification before implementing a clinical rule. When a PPV is low, alerts will more likely be overridden and implementation of the rule will not be useful because it may result in alert fatigue. The setting of the threshold will also depend on factors such as disease severity or alert frequency. A low PPV may be accepted (PPV of 30% or even 10%) if the alert can prevent severe complications or save a patient's life. Further research is needed to determine the setting of the minimum threshold to improve the beneficial effects of the CDSS.


A strategy is presented for the development and pre-implementation validation of clinical rules with a high PPV/NPV, determined by an expert team. The proposed strategy is effective for creating specific and reliable clinical rules that generate relevant recommendations. No other study has yet described a complete method for validation of clinical rules monitored with PPV and NPV during all phases of development. Ensuring appropriate alerting is a prerequisite for continued efficacy and acceptance of recommendations generated by CDSS. The crucial value of our approach lies in the consultation of an expert team in the development of the clinical rules, together with the continuous monitoring of the technical correctness and clinical relevance during the development.

The strategy was shown to be effective since the reliability of all the clinical rules developed ranged from a PPV of 89–100%, in consensus with an expert team and corrected for 0–3% incorrect technical alerts due to incorrect EHR data input. More research is needed to determine the setting of the notification threshold of clinical rules and the effects of different alert mechanisms on the acceptance of recommendations. It is hoped that this strategy will facilitate standardisation and acceleration of the effective and widespread use of these innovative and promising decision support systems in daily practice.

Key messages

  • A strategy is presented for the development and pre-implementation validation of clinical rules with a high PPV/NPV, determined by an expert team.

  • The proposed strategy is effective for creating specific and reliable clinical rules that generate relevant recommendations.

  • The crucial value of our approach lies in the consultation of an expert team in the development of the clinical rules, together with continuous monitoring of the technical correctness and clinical relevance during the development.


Medecs BV, Eindhoven, The Netherlands provided technical support regarding the CDSS during the study. We thank all experts for their extensive contribution to the development of the clinical rules in this study.

View Abstract


  • Contributors The study was designed by AJS, RJG, CN and EHK and performed by AJS and RJG. The manuscript was prepared by AJS and corrected by RJG, CN, EWA and EHK.

  • Competing interests None.

  • Provenance and peer review Not commissioned; externally peer reviewed.

Request permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.