Objective Well-designed clinical decision support systems (CDSS) can reduce the problem of alert fatigue by generating patient-specific alerts. This paper describes a strategy for the development and pre-implementation validation of specific and relevant clinical rules in order to reduce alert fatigue.

Methods A four-step development and validation strategy of clinical rules is presented. As an example, from March to September 2006 the ‘lithium therapy rule’ was developed with this strategy based on the Plan-Do-Check-Act cycle. 15 368 patients were retrospectively screened and 2503 patients were prospectively screened while the positive and negative predictive values (PPV/NPV) were continuously monitored. The first step is to confirm that the parameters used in the definitions are linked to the correct data in the electronic health record; the second step involves an expert team in the review process to assure that alerts generated are clinically relevant; in the third step the rule is adjusted to generate the right alerts in daily practice; and the fourth step ensures technical and therapeutic maintenance after implementation in practice.

Results From September 2006 to July 2010 nine other rules were developed following exactly the same strategy. The 10 clinical rules developed showed a progression during the development and all resulted in a final therapeutic PPV of ≥89% before implementation, based on expert opinion. NPV was determined for five clinical rules and was always 100%.

Conclusions The proposed strategy is effective for creating specific and reliable clinical rules that generate relevant recommendations. The inclusion of an expert team in the development process is an essential success factor. It is hoped that it will accelerate the widespread use of these promising decision support systems in practice.