How good it the evidence?
It’s a systematic review with meta-analyses, so it’s at the top of the quality of evidence pyramid:
This review scores 10/11 using the AMSTAR tool to assess the quality of the systematic review. The AMSTAR tool identifies good practices that reduce the introduction of bias – all 11 areas are evidenced to be important sources of bias in systematic reviews. 10 is good. The other reviews I’ve appraised recently have scored 2-4. It gets a 10 because:
- The review provides an a priori design
- Duplicate study identification and data extraction were carried out
- It undertook a comprehensive literature search
- The review didn’t exclude unpublished data
- A list of included and excluded studies was provided
- Characteristics of the included studies were provided
- Scientific quality of the included studies was assessed and documented
- Quality of the included studies was used appropriately in formulating conclusions
- Methods used to combine the findings of studies were appropriate
- Conflict of interest was stated
- The likelihood of publication bias was not assessed
Overall, we can be pretty sure that the systematic review can be trusted to give us the true answer based on the available (crappy, better than nothing, see figure 2 for their risk of bias assessment) data.
What are the results?
The summary of results is that “cranberry products do not significantly reduce the risk of repeat symptomatic UTI compared to placebo or no treatment in groups of people at risk of repeat UTI (overall RR 0.86, 95% CI 0.71 to 1.04) or for any of the subgroups analysed.”
RR (relative risk or risk ratio) of 0.86 means that if you compared people who didn’t have cranberry with people who did have cranberry, for every 100 cases in people who didn’t get cranberry, 86 cases occurred in people who did have cranberry. However, this doesn’t mean cranberry works – take a look at the confidence intervals (CI). They span 1.00, which means that the difference seen between these groups may well have arisen by chance.
If this study was repeated, each time it was done, there would be a 95% chance that the 95% CI included the true risk. These 95% CI (0.71 to 1.04) do not show a statistically significant difference and do not exclude the possibility that cranberry products might be making things worse (104 people get cystitis instead of 100) rather than better.
The absolute risk (AR) and number needed to treat (NNT) would be much more informative than RR, but the review doesn’t provide them. Given that the reviewers used a random effects model, it would take me a bloody age to calculate them. And, the results show no difference between people taking cranberry products and those not, so calculating AR and NNT would be an academic exercise and clinically useless. And, the risk for different people in the general population varies hugely, so even if the results were statistically significant, and I could be arsed to calculate the AR and NNT, it still wouldn’t provide a useful statistic for the general population to use to understand our risk.
An aside, I think it’s worth thinking about why we aren’t more skeptical of magical fruit. Consider not buying expensive urine until there’s evidence that for you there are useful benefits (less cystitis) that outweigh harms (expending a finite financial resource, having to drink vast quantities of bitter super-staining juice every day for ever, or eat tablets every day for ever (not fun, as anyone who actually has to knows) and propping up quack shops and drug companies that prey on the trusting, vulnerable and desperate).