ARCHE - Systematic Reviews and Meta-Analyses

Introduction

A systematic review is a structured summary of all relevant studies on a topic, using explicit methods to identify, select, and critically appraise the evidence. A meta-analysis goes further by statistically combining results from similar studies to get a pooled estimate (e.g. overall odds ratio). These are at the top of the evidence hierarchy because they synthesize data from multiple sources, reducing bias and increasing precision. However, their quality depends on the included studies and the review methods. Appraisal focuses on whether the review was comprehensive, transparent, and free of bias (using tools like AMSTAR or PRISMA).

Key Elements to Appraise

Research Question & Protocol: Check if the review has a clear, focused question (e.g. PICO: Population, Intervention, Comparison, Outcome). Look for a pre-registered protocol (e.g. on PROSPERO) to prevent selective reporting. The AMSTAR tool asks if a protocol existed before the review started.

Search Strategy: Verify that the search was comprehensive: multiple databases (e.g. PubMed, Embase), grey literature, and hand-searching. Key terms and filters should be detailed. Check for language or date restrictions – unjustified ones can bias results. AMSTAR requires duplicate study selection and data extraction.

Inclusion/Exclusion Criteria: Criteria should be explicit and applied consistently. Appraise if they match the question (e.g. only RCTs for treatment effects). Duplicate reviewers for selection reduce errors.

Risk of Bias Assessment: Good reviews assess the quality of each included study (e.g. using Cochrane Risk of Bias tool for RCTs). Check if bias was considered in the synthesis – low-quality studies should not be weighted equally.

Data Synthesis: For meta-analysis, check if studies were similar enough to pool (clinical and methodological homogeneity). Look for use of appropriate models (fixed vs random effects) and tests for heterogeneity (e.g. I² statistic). Sensitivity analyses (e.g. excluding low-quality studies) add robustness.

Reporting of Results: Results should include forest plots, confidence intervals, and discussion of limitations. Check for publication bias assessment (e.g. funnel plot or Egger's test). The PRISMA checklist ensures transparent reporting.

Funding & Conflicts: Note any funding sources or author conflicts that could bias the review.

Common Mistakes / Red Flags

Incomplete Search: If only one database was used or grey literature ignored, important studies might be missed, leading to biased conclusions.

No Bias Assessment: Failing to appraise included studies' quality means the synthesis might include flawed data, overestimating effects.

Inappropriate Pooling: Combining heterogeneous studies without justification (high I²) can produce misleading pooled estimates.

Publication Bias Ignored: Not checking for or discussing publication bias (e.g. no funnel plot) is a red flag, as negative studies are less likely published.

Selective Reporting: If outcomes or subgroups are cherry-picked, or no protocol was registered, results may be biased toward positive findings.

Overinterpretation: Watch for conclusions that overstate certainty, especially if based on low-quality evidence or small numbers.

Example

Consider a systematic review on the efficacy of a new diabetes drug, pooling data from 10 RCTs. To appraise: Check the search – did they use multiple databases and include unpublished trials? If yes, it's comprehensive. Verify bias assessment: each RCT should have been rated for randomization, blinding, etc. In the meta-analysis, look for a forest plot showing a pooled RR of 0.8 (95% CI 0.7–0.9) with low heterogeneity (I²=20%). A red flag would be if they pooled studies with high bias without sensitivity analysis, or ignored publication bias despite an asymmetric funnel plot.

Quick Checklist

Clear question and pre-registered protocol.
Comprehensive search across multiple sources.
Explicit inclusion criteria and duplicate processes.
Risk of bias assessed for each study.
Appropriate synthesis with heterogeneity checks.
Publication bias evaluated; transparent reporting (PRISMA).

Take-Home Points