Background A significant knowledge gap exists for the management of critically ill patients with coronavirus disease 2019 (COVID-19). This study aimed to systematically investigate the consistency of recommendations from the available clinical practice guidelines (CPGs) to those of the WHO on the management of critically ill COVID-19 patients.
Methods We examined CPGs and UpToDate point-of-care resources on the management of critically ill COVID-19 patients that had been published as of 30 April 2020 and compared them against the CPG by the WHO. The main outcome was the rate of consistency among CPGs for the management of critically ill COVID-19 patients. Sensitivity analyses were conducted by excluding recommendation statements that were described as insufficient evidence and by excluding single CPGs one at a time.
Results Thirteen reference recommendations derived from the CPG of the WHO were generated using discrete and unambiguous specifications of the population, intervention, and comparison states. Across CPGs, the rate of consistency in direction with the WHO is 7.7%. When insufficient evidence codings were excluded, the rate of consistency increased substantially to 61.5%. The results of a leave-one-out sensitivity analysis suggested that the UpToDate recommendation source could explain the inconsistency. Consistency in direction rates changed by an absolute 23.1% (from 1/13 (7.7%) to 4/13 (30.8%)) if UpToDate was removed.
Conclusions We observed inconsistencies between some recommendations of the CPGs and those of the WHO. These inconsistencies should best be addressed by consensus among the relevant bodies to avoid confusion in clinical practice while awaiting clinical trials to inform us of the best practice.
- intensive & critical care
- infectious diseases
- clinical pharmacy
- clinical governance
- clinical audit
This article is made freely available for use in accordance with BMJ’s website terms and conditions for the duration of the covid-19 pandemic or until otherwise determined by BMJ. You may use, download and print the article for any lawful, non-commercial purpose (including text and data mining) provided that all copyright notices and trade marks are retained.https://bmj.com/coronavirus/usage
Statistics from Altmetric.com
The unprecedented rapid spread of novel coronavirus disease 2019 (COVID-19) has created a significant knowledge gap for its management. The publication of an interim clinical practice guideline (CPG) on the clinical management of COVID-19 by the World Health Organization (WHO) is a much-needed initiative.1 Major infectious disease/intensive care societies and national organisations have since followed suit and issued their own CPGs. Nevertheless, variation in recommendations with those of the WHO in each CPG may lead to ambiguity in clinical practice. We aim to investigate the consistency of recommendations in the available CPGs to those of the WHO on the management of critically ill COVID-19 patients.
This cross-sectional study adapted methods from Alper et al2 to evaluate consistency in CPGs that had been published as of 30 April 2020. We searched four sources that listed links to national and international CPGs: DynaMed Plus,3 UpToDate,4 BMJ Best Practice,5 and Turning Research into Practice,6 to identify CPGs on the management of critically ill COVID-19 patients. From our initial search, we selected CPGs that were currently active, publicly available, published in (or has been translated to) English, likely to be used as the primary source of guidance for the management of critically ill COVID-19 patients, and published by societies/organisations that have a membership of at least 1000 or more healthcare practitioners.
The recommendations concerning therapeutic management of critically ill patients with COVID-19 in the WHO CPG1 was selected to be the reference standard for other CPGs to be compared against, since this was the first CPG issued on the management of critically ill COVID-19 patients, and it was intended to be the guide for a structured global response. There may be different interpretations among coders towards recommendations by the WHO in terms of patient population, intervention, and comparator, and thus to ensure consistency without ambiguity, we created a reference standard to compare recommendations against. A total of 13 reference recommendations derived from the CPG of the WHO1 were generated using a combined population-intervention-comparison (PIC) concept to provide a consistent framework and scope for comparison to ease the comparison process without any ambiguity. The population was for whom the recommendations from the CPG of the WHO were intended, and intervention was defined as the approach adopted in the CPG of the WHO,1 whereas comparator was defined as the approach in contrast to the CPG of the WHO.1
To minimise bias, for each reference recommendation derived from the WHO CPG, a coder (KCS) independently compared and coded the recommendation from comparator CPGs in three steps. First, the coder assessed whether each recommendation was addressed in each comparator CPG and, if not, the recommendation was labelled ‘out of scope’ for that CPG and excluded from further analysis; second, each in scope recommendation was compared for consistency to the reference standard; and finally, the rate of consistency of all recommendations was assessed.
The consistency of the recommnedations in the comparator CPG to the reference standard derived from WHO CPG was coded as:
‘for’ if the CPG recommended the intervention in favour of the comparison,
‘against’ if the CPG recommended the comparison in favour of the intervention,
‘insufficient’ if the CPG did not recommend ‘for’ or ‘against’ the intervention due to insufficient evidence from the CPG to recommend ‘for’ or ‘against’, but the PIC specification was within the scope to be addressed, or
‘different’ if the assertion from the CPG could not be categorised as ‘for’, ‘against’, or ‘insufficient’.
A code reviewer (SSH) independently checked the coding of the coder. An investigator with clinical experience in critical care (STRZ) reviewed the codings from both the coder and the reviewer, and any discrepancy identified was resolved by consensus.
While assessing the rate of consistency, we did not include CPG coded as ‘out of scope’ or ‘different’ for a reference recommendation in any of the analyses for consistency, as these code for an absence of recommendation rather than the similarity of a recommendation. We assessed the rate of consistency only if two or more CPGs provided a coding of ‘for’, ‘against’, or ‘insufficient’. For assessments of the rate of consistency, we regarded the reference recommendation to be consistent with the WHO CPG if all comparator CPG codings were ‘for’, or if they were all coded as ‘for’ or ‘insufficient’ but ≥60% were ‘for’. In contrast, we regarded the reference recommendation inconsistent with the WHO CPG if all comparator CPG codings were all either ‘against’ or ‘insufficient’, or if they were all coded as ‘for’ or ‘insufficient’ but <60% were ‘for’.
We conducted a sensitivity analysis that excluded insufficient codings from the analysis. We also performed a leave-one-out sensitivity analysis to assess rates of consistency with each CPG excluded one at a time.
The initial search yielded 300 CPGs (online supplementary figure S1). We excluded 293 of these based on our consensus (281 CPGs did not address management of critically ill COVID-19 patients, eight CPGs were published as review articles without endorsement from professional organisations, two CPGs addressed <3 reference recommendations, and two CPGs published in a language other than English and no English translation was available); the remaining seven CPGs came from the Australian and New Zealand Intensive Care Society (ANZ),7 Belgium (BEL),8 Canada (CAN),9 National Health Commission (NHC) of China,10 National Institutes of Health (NIH),11 Surviving Sepsis Campaign (SSC),12 a collaboration of Faculty of Intensive Care Medicine, Intensive Care Society, Association of Anaesthetists and Royal College of Anaesthetists of the UK (ICM),13 and UpToDate (UTD) point-of-care resources.14 Since many clinicians refer to electronic point-of-care resources, we included UpToDate, which is the most frequently used point-of-care clinical decision tool, and treated it functionally as an additional CPG. The reference recommendation derived from the CPG of the WHO1 is presented in online supplementary table S1.
Considering all eight CPGs, we found consistency with the WHO CPG for only one of the 13 reference recommendations (7.7%). This consistency was for recommendation #6 (prescribing thromboprophylaxis where there are no contraindications), although this recommendation was labelled out of scope for five comparator CPGs as it was not covered in each guideline. In each reference recommendation, there was at least one CPG with ‘insufficient’ coding, except reference recommendations #6 (prescribing thromboprophylaxis) and #13 (not prescribing systemic corticosteroids). Recommendation #3 (empiric antibiotics) received only two ‘for’ and one ‘against’ codings and for two CPGs it was out of scope (table 1).
The recommendation with the highest rate of ‘insufficient’ codings was observed in reference recommendation #1 (prescribing initial oxygen therapy) where the target oxygen saturations were not specified in most guidelines (7/8; 87.5%) (online supplementary table S2). When we removed ‘insufficient’ ratings from consideration as part of the sensitivity analyses, rates of consistency in direction with the WHO (8/13; 61.5%) became higher. Also, there were two reference recommendations—#2 (prescribing maintenance oxygen therapy) and #11 (prescribing vasopressor to a target mean arterial pressure (MAP) ≥60 mmHg)—with the available CPGs that had exclusively ‘against’ codings when ‘insufficient’ codings were removed from consideration. The results of a leave-one-out sensitivity analysis suggested that the UpToDate recommendation source could explain the inconsistency. Consistency in direction rates changed by an absolute 23.1% (from 1/13 (7.7%) to 4/13 (30.8%)) if UpToDate was removed. If insufficient ratings are excluded in leave-one-out sensitivity analyses, the consistency in direction rates ranged from no change to change by an absolute of 15.3%.
This cross-sectional study observed substantial inconsistency across CPGs for the management of critically ill COVID-19 patients, which could be explained by insufficient ratings, especially from UpToDate point-of-care resources. A high rate of insufficient ratings across each reference recommendation may be due to partial recommendations resulting from a lack of clinical studies addressing the specific area of recommendation in critically ill patients with COVID-19. For example, the WHO1 recommended a target oxygen saturation (SpO2) >94% on oxygen therapy during initial resuscitation for patients with severe acute respiratory infection (SARI) and respiratory distress, hypoxaemia or shock targets (reference recommendation #1), and though most CPGs (ANZ,7 BEL,8 NHC,10 NIH,11 SSC,12 ICM,13 UTD14) recommended the administration of oxygen therapy, the target SpO2 was not mentioned. Consequently, the inconsistency with the WHO CPG had resulted from only partial recommendations made in comparator CPGs on the use of oxygen therapy, and reflects the absence of evidence for optimal oxygenation requirements during resuscitation in critically ill COVID-19 patients. Indeed, the same can be applied to the high rate of out of scope ratings across each reference recommendation, where a lack of clinical studies focusing on COVID-19 patients addressing the specific area of recommendation may deter CPGs from giving their recommendations.
When independent societies/organisations have reviewed the same evidence with various factors including values and preferences being considered, and reach similar conclusions regarding a recommendation, the credibility of the recommendation may be higher. An analogy would be when findings from a research study are replicated by others, the credibility of the results is increased. However, when societies/organisations reach varying conclusions about a recommendation, the inconsistency can create confusion for clinical practice. Online supplementary table S3 presents some clinical implications for inconsistencies among comparator CPGs with the WHO CPG. The fact that the UpToDate CPG with one author was the biggest contributor to inconsistency scorings when compared with the WHO suggests a limitation with this resource in comparison with guidelines produced by consensus of national and international organisations. COVID-19 is exceptional in that, because there is such a significant knowledge gap, this emphasises the need for a collaborative approach to guide the production of CPGs based on evidence and expert consensus, and consequently requires a diverse number of specialist viewpoints to make reasonable recommendations for practice.
The most obvious example of inconsistencies would be the administration of corticosteroid therapy in severely ill patients with COVID-19. While the WHO1 discouraged routine administration of corticosteroid therapy outside of the clinical trial, some CPGs recommended its administration in COVID-19 patients with refractory shock (SSC),12 mechanically ventilated COVID-19 patients with acute respiratory distress syndrome (ARDS) (SSC),12 and COVID-19 patients with progressive deterioration of oxygenation indicators, rapid deterioration in imaging, and excessive activation of the body's inflammatory response (NHC).10 In fact, UpToDate14 has changed from being against the routine administration of corticosteroid therapy to recommending low-dose dexamethasone for intensive care unit (ICU) patients with COVID-19 who require oxygen supplementation and for selected patients with shock who is refractory to fluid resuscitation, upon the announcement of the results of the RECOVERY trial.15 The RECOVERY trial15 randomised patients either to dexamethasone (6 mg once per day enterally or parenterally) along with usual care (n=2104) or usual care alone (n=4321). Compared with patients randomised to usual care alone, dexamethasone reduced the overall 28 day mortality rate by 17% (relative risk (RR) 0.83, 95% CI 0.74 to 0.92; p=0.0007). Subgroup analysis revealed that dexamethasone reduced deaths by about one-third in ventilated patients (RR 0.65, 95% CI 0.48 to 0.88; p=0.0003) and by one fifth in patients receiving oxygen only (RR 0.80, 95% CI 0.67 to 0.96; p=0.0021), but there was no mortality benefit among patients who did not require respiratory support (RR 1.22, 95% CI 0.86 to 1.75; p=0.14). Therefore, CPGs which recommend against routine use of systemic corticosteroids, including that from the WHO, should be updated accordingly based on the results of RECOVERY trial, since this is the first and only randomised trial of treatment intended for COVID-19 patients at the time of writing which has demonstrated mortality benefits. This reflects an advantage of the UpToDate point-of-care resources14 in that it allows rapid updates to be added to their recommendation without protracted peer review and consultation.
Another area worth attention due to disagreement with the WHO’s recommendation1 was target SpO2 during maintenance oxygen therapy, where some CPGs (CAN,9 NIH,11 SSC,12 ICM,13 UTD14) recommended not to exceed an SpO2 of 96%, while the WHO1 did not recommend the maximum target SpO2. Those CPGs9 11–14 which recommended an upper limit of target SpO2 of 96% cited the systematic review and meta-analysis16 which included 25 randomised controlled trials with over 16 000 patients; it reported that a liberal oxygen strategy is associated with increased risk of hospital mortality (RR 1.21, 95% CI 1.03 to 1.43) in acutely ill patients, with meta-regression also demonstrating a linear association between risk of death and higher SpO2 targets. The upper limit of target SpO2 of 96% was recommended since the baseline median SpO2 was 96% in the liberal oxygen group across all trials included in the aforementioned systematic review and meta-analysis. Furthermore, the increased cost could be foreseen with liberal oxygen use in every patient requiring oxygen therapy during hospitalisation.
There was also disagreement in some CPGs with the WHO’s recommendation in the target MAP for patients who received a vasopressor for septic shock. Specifically, CAN and SSC allowed a lower target MAP up to 60 mmHg, instead of ≥65 mmHg recommended by the WHO. Two references have been cited by SSC to support their recommendation in which a 2017 individual patient-data meta-analysis17 of two randomised controlled trials, which randomised 894 adult patients with shock to either higher or lower MAP targets for vasopressor therapy, reported no significant difference in 28 day mortality, 90 day mortality, myocardial injury or limb ischaemia; however, the odds ratio (OR) for arrhythmias was increased among patients randomised to the higher MAP target (OR 2.50, 95% CI 1.35 to 4.77). Another more recent trial,18 not included in the above-mentioned meta-analysis, reported an absolute risk difference in mortality of 3% (RR 0.93, 95% CI 0.85 to 1.03) in favour of a MAP target of 60–65 mmHg compared with the higher MAP target.
In terms of neuromuscular blockade among patients with moderate-to-severe ARDS, there was only one CPG (NIH) which had opposing recommendations to those of the WHO. The NIH CPG allowed routine continuous infusion of neuromuscular blocking agents (NMBAs) to facilitate protective lung ventilation. This could be due to the mixed results reported thus far pertaining to this issue. A 2013 systematic review and meta-analysis,19 which included three randomised controlled trials with 431 patients, demonstrated a significant reduction in 90 day mortality with continuous NMBA infusion as compared with no NMBA infusion (RR 0.72, 95% CI 0.58 to 0.91). However, the 2019 ROSE trial19 reported no significant difference in 90 day mortality rate with and without routine continuous NMBA infusion (between-group difference, −0.3 percentage points; 95% CI −6.4 to 5.9; p=0.93). In addition, patients with routine continuous NMBA infusion were less physically active and had more adverse cardiovascular events than patients without routine continuous NMBA infusion. This is an area of controversy which may require consensus among professional societies to avoid confusion for clinicians, pending more randomised trials addressing this issue.
With observed inconsistencies in some recommendations of CPGs compared with those of the WHO, it is of the utmost importance that these inconsistencies are addressed by consensus among the relevant bodies, since there may be clinical implications for the care of critically ill COVID-19 patients. A consensus is important to avoid confusion in clinical practice where clinicians managing COVID-19 patients might hesitate in their clinical decision-making, potentially affecting the quality of care. Nevertheless, solving through consensus is only desired while awaiting clinical trials specific for COVID-19 patients. Results of clinical trials specific for COVID-19 patients should be adopted by CPGs to guide clinical practice if there is no ambiguity.
EAHP Statement 5: Patient Safety and Quality Assurance.
Contributors CSK conceived the content, retrieved the data, wrote the manuscript, and approved the final version. TC reviewed the data, wrote the manuscript, and approved the final version. STRZ reviewed the data, wrote the manuscript, and approved the final version. SSH conceived the content, helped in data extraction, revised the manuscript critically, and approved the final version.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests None declared.
Patient consent for publication Not required.
Provenance and peer review Not commissioned; externally peer reviewed.
Data availability statement Data sharing not applicable as no datasets generated and/or analysed for this study.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.