BMJ review finds trials of LLM tools in medicine meet 40% to 61% of reporting standards

A systematic review published in the BMJ found that clinical trials of large language model (LLM) interventions in medicine report only a modest share of the detail that established reporting guidelines require ¹. The gaps are widest on the model-specific facts and on harms, which are the details a buyer or regulator would need to judge whether a tool is safe ¹.

What the review found

The authors searched Embase, Medline and CENTRAL, plus ClinicalTrials.gov and the WHO International Clinical Trials Registry Platform, for material published between 1 January 2017 and 6 December 2025 ¹. They included trials that prospectively assigned people to an LLM intervention used for a healthcare function, such as patient education, clinical documentation, communication support, triage, care navigation, management recommendations or clinical decision support, and that measured a health outcome ¹.

The search returned 461 records: 68 trial reports, 15 protocols and 378 registrations ¹. Most trial reports described small, single-site randomised studies published in 2024 and 2025, and most of the tools tested were aimed at patients, chiefly for education and management recommendations ¹. Two reviewers screened and extracted the studies, and three calibrated raters scored each reporting item ¹.

Mean reporting completeness for trial reports was 40% against TRIPOD-LLM, 44% against CONSORT 2025, 59% against CONSORT-AI and 52% against CHART, the chatbot assessment tool ¹. Protocols reached 51% against SPIRIT 2025 and 57% against SPIRIT-AI, and registrations reached 61% against the WHO Trial Registration Dataset ¹.

The reviewers found that reports favoured narrative framing and commentary over the methods needed to replicate a study, assess its risk of bias or govern how a model performs over time ¹. Items that were inconsistently reported included where the data and the intervention came from, input preprocessing, model version and how the model was accessed, inference parameters, output postprocessing, and access to protocols, data and code ¹. Planning, monitoring and reporting of harms were often incomplete ¹.

What it means for the market

For health systems buying LLM tools, the finding cuts against relying on a published trial alone as proof of safety. A trial that does not state the model version or the inference settings cannot show that the product a trust deploys is the one that was tested, and the authors say that weak harms reporting limits critical appraisal of safety ¹.

For suppliers, the review sets out a checklist of what evaluators are likely to look for: provenance, versioning, access modality, inference parameters, postprocessing and open protocols, data and code ¹. The review does not report how completeness varied by sponsor, country or product type, so it cannot say which companies or markets perform better ¹.

Journal editors and trial funders are also implicated. AI-specific guidelines such as CONSORT-AI and SPIRIT-AI exist, yet the best mean score against any guideline was 61%, for registrations ¹.

Caveats

The review measures how completely trials are reported, not whether the tools work or cause harm ¹. Its evidence base is dominated by small, single-site studies from 2024 and 2025, so it reflects an early stage of LLM evaluation ¹. The search closed on 6 December 2025, and trials reported since then are not included ¹. The authors conclude that underreporting of LLM-specific methods, reproducibility artefacts and harms persists, and that these details are needed before such tools move into clinical use ¹.