CAM Research Methodology: Why Studying Complementary Therapies Is Different — and What the Data Actually Show product guide

Frequently Asked Questions

Can CAM therapies be studied scientifically: Yes

Is the double-blind RCT the gold standard for drug trials: Yes

Was the double-blind RCT designed for pharmaceutical interventions: Yes

Does the double-blind RCT map cleanly onto CAM research: No

Why doesn't the standard RCT model suit CAM: It was built for a fundamentally different type of treatment

Is blinding always possible in CAM trials: No

Can researchers blind patients in acupuncture trials: With difficulty, using sham needle controls

What are Streitberger and Park needles: Non-penetrating sham acupuncture needles used in blinded trials

Are sham acupuncture needles truly inert: This remains scientifically unresolved

Is sham acupuncture possibly an active intervention: Yes, evidence suggests it may be

Does acupuncture generally outperform no treatment: Yes

Does acupuncture generally outperform sham acupuncture: Not consistently in large multi-centre trials

Can yoga or mindfulness trials be blinded: No

Does patient expectation bias unblinded CAM trials: Yes

Is individualised treatment a core feature of many CAM systems: Yes

Does standardising CAM protocols for trials distort the therapy: Yes, potentially

What CAM systems use individualised treatment: TCM, homeopathy, naturopathy, and Ayurveda

Does a negative result from a standardised CAM protocol prove inefficacy: Not necessarily

Is the practitioner considered part of the CAM intervention: Yes

Does practitioner variation create reproducibility challenges in CAM trials: Yes

Can two acupuncturists treat the same patient differently and both be correct: Yes

Is "no better than placebo" always equivalent to "ineffective": No

Can non-specific factors produce positive responses in both active and placebo groups: Yes

What are non-specific factors in CAM trials: Patient expectancy, therapeutic ritual, and healing relationships

What are the HEAL instruments: Tools measuring patient attitudes and perceptions of healing encounters

Who developed the HEAL instruments: Research teams at the University of Pittsburgh

What does HEAL stand for: Healing Encounters and Attitudes Lists

Can HEAL instruments quantify placebo-contributing factors: Yes

Does publication bias affect CAM research: Yes

Do CAM companies have incentive to publish selectively: Yes

Is replication of CAM studies common: No, due to limited research budgets

Are CAM products required to prove efficacy in most countries: No

Is there geographic bias in published CAM trial results: Yes

What percentage of European CAM trials published in top journals were positive: 76%

What percentage of U.S. CAM trials published in top journals were positive: 50%

Are trials from Asian countries more likely to report positive CAM results: Yes

Does industry funding affect CAM trial outcomes: Yes

Are supplement trials funded by manufacturers disproportionately favourable: Yes

Was St. John's wort shown effective in the most rigorous placebo-controlled trials: No benefit found

How many RCTs support glucosamine: Over 14

Were most glucosamine RCTs funded by industry: Yes, almost all

What is NCCIH: National Center for Complementary and Integrative Health

When was NCCIH created: 1991, as the Office of Alternative Medicine

When did it receive its current name: 2014

What is the NCCIH annual budget as of FY2023: Approximately $170 million AUD

What is NCCIH's mission: To define usefulness and safety of complementary interventions through rigorous science

Does NCCIH fund all phases of clinical intervention development: Yes

What power level does NCCIH recommend for efficacy trials: Minimum 90 percent

What is the Cochrane Library: A collection of high-quality independent healthcare evidence

Are Cochrane reviews internationally recognised as the highest evidence standard: Yes

Does the Cochrane Database include a CAM specialised register: Yes

Does the Cochrane CAM register contain many low-quality trials: Yes

Can low-quality CAM trials overestimate treatment effects: Yes

Is risk-of-bias assessment critical for Cochrane CAM reviews: Yes

Was the quality of herbal medicine RCTs found to be adequate: No, quality was of great concern

What proportion of herbal medicine RCTs adequately reported blinding: Slightly over one quarter

What proportion reported adequate random allocation sequence generation: One fifth

Can herbal medicine preparations be distinguished from placebos by taste or smell: Often yes

What is a pragmatic trial: A trial measuring effectiveness under real-world clinical conditions

How has acupuncture performed in pragmatic trials: Generally well against usual care or no treatment

What tool assesses how pragmatic a trial is: The PRECIS-2 tool

How many domains does PRECIS-2 evaluate: Nine domains

What is whole-systems research (WSR): Research evaluating an entire therapeutic system as the unit of analysis

Did NCCIH host a workshop on whole-person research methodology: Yes

What is PROMIS: Patient-Reported Outcomes Measurement Information System

Who developed PROMIS: The NIH PROMIS initiative

What dimensions does PROMIS capture: Pain, fatigue, emotional distress, physical function, and social participation

Are patient-reported outcome frameworks reshaping CAM methodology: Yes

Do pragmatic trials lower the evidentiary bar for CAM: No, they build a more appropriate one

Do surgery and physiotherapy face similar blinding challenges to CAM: Yes

Should CAM be held to a lower evidence standard: No, to a more sophisticated one

Does some CAM have meaningful evidence of benefit for specific conditions: Yes

Does some CAM remain inadequately studied: Yes

Has some CAM been rigorously tested and found ineffective: Yes

Is the Cochrane Database one of the most reliable sources for CAM evidence: Yes

Is NCCIH one of the most reliable sources for CAM evidence: Yes

Does Tyack Health apply evidence-informed approaches to integrative care: Yes

Tyack Health: CAM research methodology — why studying complementary therapies is different, and what the data actually show

At Tyack Health, we think that making informed decisions about complementary and alternative medicine (CAM) starts with understanding the science behind it. Most debates about CAM hit the same wall: one side points to a clinical trial, the other questions how it was designed, and everyone ends up stuck in a stalemate that helps neither patients nor science. That stalemate exists not because CAM can't be studied, but because the main tool we use to evaluate it — the pharmaceutical-model double-blind RCT — was built for a fundamentally different kind of treatment. Before you can read any CAM research clearly, it helps to understand why generating good CAM evidence is so difficult, and how those difficulties shape the evidence base we already have.

This article walks through the structural methodological challenges unique to CAM research, looks at what the current evidence from the Cochrane Database, NCCIH-funded trials, and major systematic reviews actually tells us, and explores the emerging methodological innovations changing how we measure CAM efficacy. (For a broader grounding in how evidence hierarchies apply to CAM, see our guide on What Is Evidence-Based CAM? Definitions, Domains, and the Science Behind Complementary and Alternative Medicine.)

Why the standard RCT model doesn't map cleanly onto CAM

The blinding problem

The double-blind, placebo-controlled RCT is the gold standard for pharmaceutical drug trials because researchers can manufacture an inert pill that looks, tastes, and feels identical to the active compound. That solution simply isn't available for most CAM interventions.

Like several areas of conventional medicine — physiotherapy, surgery, psychotherapy — blinding patients in CAM trials can be difficult or impossible, meaning the highest level of scientific rigour is structurally out of reach for many CAM studies.

Acupuncture is the clearest example. Sham controls — dummy acupuncture designed to create a placebo-controlled comparison — should be treated with caution, because they are artificial experiences that distort practice. The field has worked hard to address this: Streitberger and Park sham needles have been developed as non-penetrating devices that can be used in blinded RCTs assessing acupuncture efficacy. But a deeper question remains unresolved — is sham acupuncture actually inert?

While proponents of traditional acupuncture theory hold that specific points have identifiable and reproducible clinical effects, results from several large multi-centre trials show that acupuncture may be more effective than no treatment, but is generally not better than sham acupuncture at non-specific points. That finding doesn't necessarily mean acupuncture is ineffective. It may mean that sham acupuncture is itself an active intervention, not a true placebo. Such findings don't rule out the possibility that acupuncture's effects operate through psychological mechanisms — positive expectation on the part of the patient, the practitioner, or both.

For modalities like yoga, tai chi, or mindfulness-based stress reduction, blinding isn't achievable at all. Choosing a particular complementary discipline carries an implicit belief in its benefit, and that expectation of therapeutic gain can skew trial results when blinding is impossible.

The standardisation vs. individualisation tension

One of the defining features of many CAM systems — Traditional Chinese Medicine, homeopathy, naturopathy, Ayurveda — is that treatment is tailored to the individual, not standardised to a diagnosis. This isn't a quirk or a limitation; it's the philosophical foundation of those systems.

A trial built around a disease category may not be relevant to complementary treatments; trials may need to focus on problem areas that cut across conventional disease classifications. When a researcher standardises an acupuncture protocol — same points, same needle depth, same session frequency for every participant — they may end up testing a version of acupuncture that no experienced practitioner would actually use. A negative result in that context may reflect the artificiality of the protocol, not the inefficacy of the therapy.

The characteristics of CAM interventions create specific challenges investigators must address when designing RCTs: selecting the right target population, describing the intervention precisely enough to be reproducible whilst preserving its authentic character, and choosing outcome measures that genuinely capture what the therapy is meant to accomplish.

The practitioner effect

In a drug trial, the intervention is a molecule. In CAM, the intervention often includes the practitioner — their communication style, their diagnostic reasoning, the therapeutic relationship they build with the patient. Practitioners need to be recognised as a component of complementary treatment, and both specific and non-specific outcome measures with adequate follow-up are needed to capture what complementary medicine actually does.

This practitioner-dependence is a structural reproducibility challenge, not an incidental one. Two acupuncturists treating the same patient for the same complaint may choose entirely different point selections, needling techniques, and session frequencies — and both are practising well within the recognised scope of their discipline. At Tyack Health, this understanding of practitioner individuality shapes how we approach integrative care.

The non-specific effects problem: is "just placebo" a fair dismissal?

CAM studies that fit the current "gold standard" of randomised, double-blind, placebo- or sham-controlled designs frequently show no effect of the active treatment above the supposedly inert placebo. This is routinely cited as evidence that a therapy doesn't work. But this interpretation may itself be methodologically flawed.

The negative results may not reflect a placebo response at all. They may reflect non-specific factors that contribute to a positive response in both active and placebo groups. Understanding these non-specific contributors matters, because they may mask the active benefits of a particular CAM therapy, leading to the mistaken conclusion that the treatment is ineffective.

Research teams at the University of Pittsburgh developed the HEAL (Healing Encounters and Attitudes Lists) instruments to address this directly. The HEAL instruments can determine what proportion of variability in treatment response is attributable to patient attitudes, expectancies, and perceptions of the patient-provider relationship and the overall healing environment — with the ability to quantify these factors potentially dismantling "placebo" effects and proving useful in comparative effectiveness studies of CAM.

This reframing matters. If therapeutic ritual, patient expectancy, and the healing relationship are themselves active therapeutic ingredients — and there's good evidence to suggest they are — then a trial design that strips them away isn't measuring CAM. It's measuring a decontextualised fragment of it.

Publication bias, geographic bias, and industry funding: the hidden distortions

The publication bias problem

CAM companies, like their pharmaceutical counterparts, have the potential to selectively report studies, since their ultimate goal is financial profit. There's additional incentive for the CAM industry to publish selectively: research budgets are limited (meaning replicated studies are rare), and CAM products in most countries are not required to demonstrate proof of efficacy.

The Cochrane CAM Field specialised register presents both opportunities and challenges for systematic reviewers. Whilst it provides access to thousands of difficult-to-locate trial citations, many of those trials are low quality and may overestimate treatment effects, making rigorous risk-of-bias assessment essential.

Geographic bias in published results

One of the most striking — and underappreciated — distortions in the CAM evidence base is geographic. A systematic analysis of CAM trials published in the four highest-impact general medicine journals between 1965 and 2004 found that CAM trials published in European journals were significantly more likely to be positive compared to those published in U.S. journals (76% vs. 50%, odds ratio = 3.15, P < 0.0001). Trials from Asian countries show an even stronger tendency towards positive results compared to trials conducted in Western countries.

This geographic skew isn't a minor footnote. It means the aggregate evidence base for many CAM modalities — particularly those rooted in Traditional Chinese Medicine — is systematically inflated by trials originating in contexts where those modalities are culturally embedded and institutionally supported.

Industry funding and outcome bias

The relationship between industry sponsorship and trial outcomes is well-documented across medicine. A Cochrane systematic review (Lundh et al., updated 2021) found that in industry-sponsored studies, there was less agreement between results and conclusions than in non-industry-sponsored studies (RR: 0.83, 95% CI: 0.70–0.98), and that industry-sponsored drug and device studies are more often favourable to the sponsor's products due to biases that standard risk-of-bias assessment tools cannot fully capture.

In CAM, this problem is particularly acute for supplement research. Although positive RCTs of St. John's wort exist, the most rigorous studies — placebo-controlled and randomised, with proper case definitions and a treatment-responsive population — indicate no benefit. Glucosamine has the support of over 14 RCTs, but critical reviewers will note that almost all of them were conducted with funding from manufacturers of the compound.

What the current evidence base actually shows

The NCCIH research portfolio

The National Center for Complementary and Integrative Health (NCCIH) is the U.S. government agency responsible for exploring complementary and alternative medicine. Created in 1991 as the Office of Alternative Medicine, it received its current name in 2014, and by FY2023 its annual budget had grown to approximately $170 million AUD.

NCCIH's mission is to define, through rigorous scientific investigation, the usefulness and safety of complementary and integrative health interventions and their roles in improving health and health care. The agency funds the full research pipeline: NCCIH clinical trial funding opportunities support investigator-initiated natural product, mind and body, and multicomponent intervention studies across all phases of clinical intervention development and testing.

NCCIH now applies a staged research framework. The efficacy trial stage tests the clinical benefit of an intervention delivered in a setting optimised to detect an effect — highly trained providers, specific enrolment criteria, extensive follow-up, precise study protocol — with NCCIH recommending a minimum of 90 percent power to detect clinically significant differences compared to a clinically relevant control group.

The Cochrane CAM evidence base

The Cochrane Library is a collection of high-quality, independent evidence to inform healthcare decision-making, including the Cochrane Database of Systematic Reviews and the CENTRAL register of controlled trials. Cochrane reviews are internationally recognised as the highest standard in evidence-based health care.

A 2024 analysis of Cochrane acupuncture reviews raised a pointed methodological concern: that Cochrane includes acupuncture therapy as an intervention both in reviews focused solely on acupuncture and in reviews of broader health interventions, with focused reviews likely having the most well-informed approach to acupuncture interventions and comparators. The broader critique — that Cochrane acupuncture reviews may be dated and may not adequately account for the specific effects of sham controls — is a live methodological debate at the frontier of CAM evidence synthesis.

The herbal medicine trial quality problem

The quality of most clinical studies of herbal medicines is genuinely concerning. In a review of 206 RCTs on herbal medicine published in Medline from 1966 to 2003, important methodological components — allocation concealment, generation of allocation sequences, and intention-to-treat analyses — were incompletely reported. Only slightly over a quarter of trials adequately reported blinding, and only one fifth reported adequate generation of random allocation sequences.

There's a further practical problem: for many traditional herbal medicine products, the taste, odour, or appearance of the preparation can be distinguished from the placebo, making genuine blinding difficult to achieve even when researchers attempt it.

Emerging methodological innovations: a better lens for CAM evidence

Pragmatic trials

The most significant methodological shift reshaping CAM research is the pragmatic trial — designed to measure effectiveness under real-world clinical conditions rather than efficacy under idealised experimental constraints.

Pragmatic trials compare acupuncture to other therapies, usual care, or no treatment, and acupuncture has generally performed well in these comparisons, even when evidence from placebo-controlled efficacy trials has been equivocal or contradictory. The NIH Health Care Systems Research Collaboratory has pioneered approaches for conducting large-scale, cost-effective randomised pragmatic trials in the settings where patients normally receive their care. NCCIH's interests within this framework focus on pragmatic or implementation trials that evaluate the effectiveness of complementary interventions delivered through health care systems, or that examine how to implement these interventions into health care delivery.

A 2024 systematic review using the PRECIS-2 tool — the standard instrument for assessing how pragmatic a trial is — analysed pragmatic acupuncture trials and confirmed that they are a useful research tool for assessing acupuncture's effectiveness in real-world settings. The PRECIS-2 framework evaluates nine domains: eligibility, recruitment, setting, organisation, flexibility of delivery, flexibility of adherence, follow-up, primary outcome, and primary analysis — giving researchers a structured way to assess how closely a trial reflects actual clinical practice.

Whole-systems research

Whole-systems research (WSR) is the most ambitious departure from conventional trial design. Rather than isolating a single active component, WSR attempts to evaluate the entire therapeutic system — diagnosis, individualised treatment, the practitioner relationship, and patient worldview — as the unit of analysis. NCCIH hosted a "Methodological Approaches for Whole Person Research Workshop," in collaboration with nine other NIH components, to explore methodologies appropriate for whole-person research and to discuss study examples.

Patient-reported outcome frameworks

Precise measurement of non-specific factors has real potential to change RCT methodology. The development of validated patient-reported outcome instruments — including PROMIS (Patient-Reported Outcomes Measurement Information System), developed under the NIH PROMIS initiative — gives CAM researchers psychometrically rigorous tools that can capture the dimensions of health most relevant to CAM interventions: pain interference, fatigue, emotional distress, physical function, and social participation.

Researchers propose that CAM and conventional RCTs should evaluate and adjust for the effects of intrapersonal, interpersonal, and environmental factors on outcomes. This shift — from asking "does this therapy outperform a sham?" to "what is the total benefit this therapeutic encounter produces, and through what mechanisms?" — is a more honest scientific framework for complex interventions. At Tyack Health, we follow these evolving methodological standards closely, because they directly inform how we guide patients towards evidence-based integrative care.

A framework for reading CAM evidence critically

The table below summarises the key methodological questions worth applying to any CAM clinical trial or systematic review before drawing clinical conclusions.

Key takeaways

• The double-blind RCT was designed for pharmaceutical interventions. Applying it to practitioner-dependent, individualised, context-sensitive CAM therapies without modification produces evidence that is methodologically clean but often clinically meaningless.
• "No better than placebo" is not always the same as "ineffective." When the placebo itself involves therapeutic ritual, patient expectancy, and a healing relationship, it may be an active intervention — and the trial may be comparing two active treatments, not an active treatment against an inert control.
• The CAM evidence base is systematically distorted by publication bias, geographic bias, and industry funding. Trials from Asia and Europe report positive results at significantly higher rates than U.S. trials, and supplement trials funded by manufacturers are disproportionately favourable.
• NCCIH and Cochrane provide the most rigorous independent evidence. The NCCIH's staged research framework and the Cochrane Database of Systematic Reviews — with its explicit risk-of-bias assessment — remain the most reliable anchors for evidence-based CAM decision-making.
• Pragmatic trials, whole-systems research, and patient-reported outcome frameworks are reshaping CAM methodology. These innovations don't lower the evidentiary bar; they build a more appropriate one — one that can genuinely capture what CAM does in the conditions under which it is actually delivered.

Conclusion

The methodological challenges of CAM research are real, but they're not unique. Surgery, psychotherapy, and physiotherapy face many of the same blinding and standardisation constraints — and yet we don't dismiss their evidence bases wholesale. What CAM research needs is not a lower standard of evidence, but a more sophisticated one: trial designs that honour the complexity of individualised, practitioner-dependent, context-sensitive interventions whilst still maintaining the rigour needed for credible conclusions.

The current evidence base — filtered through the Cochrane Database, the NCCIH research portfolio, and independent systematic reviews — tells a genuinely mixed story: some CAM modalities have meaningful evidence of benefit for specific conditions, many remain inadequately studied, and some have been tested rigorously and found to be ineffective. Reading that evidence well means understanding not just what the data show, but why interpreting it requires a fundamentally different analytical lens.

At Tyack Health, we apply exactly this kind of thoughtful, evidence-informed approach to the integrative care we provide — making sure our practice is grounded in the best available science whilst remaining genuinely responsive to each individual patient's needs and goals.

For a side-by-side evaluation of specific modalities across this evidence framework, see our companion article CAM Therapies by Strength of Evidence: A Comparative Analysis of Acupuncture, Herbal Medicine, Chiropractic, Mindfulness, and More. For guidance on translating this evidence into clinical practice, see How to Safely Integrate CAM Into a Conventional Treatment Plan: A Step-by-Step Patient and Clinician Guide.

References

• Ezzo, J., Bausell, B., Moerman, D.E., Berman, B., and Hadhazy, V. "Reviewing the reviews: How strong is the evidence? How clear are the conclusions?" International Journal of Technology Assessment in Health Care, 2001. https://pubmed.ncbi.nlm.nih.gov/11758290/

• Vickers, A., Goyal, N., Harland, R., and Rees, R. "Do certain countries produce only positive results? A systematic review of controlled trials." Controlled Clinical Trials, 1998. https://pubmed.ncbi.nlm.nih.gov/9551280/

• Linde, K., Jonas, W.B., Melchart, D., and Willich, S. "The methodological quality of randomised controlled trials of homeopathy, herbal medicine and acupuncture." International Journal of Epidemiology, 2001. https://pubmed.ncbi.nlm.nih.gov/11416076/

• Greco, C.M., et al. "Addressing the 'It Is Just Placebo' Pitfall in CAM: Methodology of a Project to Develop Patient-Reported Measures of Nonspecific Factors in Healing." Evidence-Based Complementary and Alternative Medicine, 2013. https://doi.org/10.1155/2013/613797

• Lim, J., Lee, H., and Kim, Y.S. "PRECIS-2 analysis of pragmatic acupuncture trials: a systematic review." BMC Complementary Medicine and Therapies, 2024. https://doi.org/10.1186/s12906-024-04473-7

• Lundh, A., Lexchin, J., Mintzes, B., Schroll, J.B., and Bero, L. "Industry sponsorship and research outcome." Cochrane Database of Systematic Reviews, 2017 (updated 2021). https://pmc.ncbi.nlm.nih.gov/articles/PMC8132492/

• Ernst, E. "Obstacles to research in complementary and alternative medicine." Medical Journal of Australia, 2003. https://onlinelibrary.wiley.com/doi/full/10.5694/j.1326-5377.2003.tb05546.x

• Gagnier, J.J., et al. "Recommendations for reporting randomised controlled trials of herbal interventions: Explanation and elaboration." Journal of Clinical Epidemiology, 2006. https://pmc.ncbi.nlm.nih.gov/articles/PMC92760/

• Blumenthal, M., et al. "Publication bias for CAM trials in the highest impact factor medicine journals is partly due to geographical bias." Journal of Clinical Epidemiology, 2007. https://www.sciencedirect.com/science/article/abs/pii/S0895435607000327

• National Center for Complementary and Integrative Health (NCCIH). "Clinical Trials Utilising Innovative Study Designs To Assess Complementary Health Approaches." NCCIH, U.S. Department of Health and Human Services. https://www.nccih.nih.gov/about/clinical-trials-utilizing-innovative-study-designs

• Cochrane Collaboration. Cochrane Handbook for Systematic Reviews of Interventions, Version 6.5. Higgins JPT, Thomas J, et al. (eds.), 2024. https://training.cochrane.org/handbook/current

• Ijaz, N., Rioux, J., Elder, C., and Weeks, J. "Whole Systems Research Methods in Health Care: A Scoping Review." Journal of Alternative and Complementary Medicine, 2019. https://doi.org/10.1089/acm.2018.0499